Data Engineering Team LeadAIMLEAP • Republic Of India, IN
No longer accepting applications
Data Engineering Team Lead
AIMLEAP • Republic Of India, IN
1 day ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Team Lead • Republic Of India, IN
Related jobs
Senior Dell Boomi Integration Engineer
Maitsys • India, India
Job Description : Senior Boomi Integration Engineer.Atom migration (on-prem → cloud), integration development, and ongoing support.
Senior Dell Boomi Integration Engineer.Boomi Atom to a cloud-hosted...Show more
Last updated: 15 days ago • Promoted
Technical Team Lead
AiPrise • India, India, India
The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics.Your role will involve strategizing, project management, part staff managemen...Show more
Last updated: 3 hours ago • Promoted • New!
Engineering Team Lead
Viridium.AI • Pune, Republic Of India, IN
AI, we are building a Material Intelligence platform.We are an early stage startup driven by the dual opportunity to build an amazing company and make a positive impact on the world.Our mission is ...Show more
Last updated: 9 days ago • Promoted
Engineering Team Lead
Hyatt Regency Chennai • Chennai, Republic Of India, IN
Lead, mentor, and develop a team of hardware, electrical, and / or mechanical engineers.Support career growth through regular 1 : 1s, performance reviews, and skills development.Foster a collaborative,...Show more
Last updated: 2 days ago • Promoted
Engineering Team Lead
Americana Restaurants • Republic Of India, IN
Position Title – Technical Lead & delivery – Platform (Java / J2EE Architect).Js, Java, Spring Boot, Python, AKS, Azure Cloud Native, Azure DevOps).
About Americana Restaurants International PLC.Ameri...Show more
Last updated: 9 hours ago • Promoted • New!
AI / ML Engineering Team Lead
MillionLogics • Republic Of India, IN
MillionLogics, a trusted Oracle Partner, is a global IT solutions leader with headquarters in London, UK, and a dynamic development hub in Hyderabad, India.
The company specialises in delivering sma...Show more
Last updated: 17 hours ago • Promoted • New!
Data Science Team Lead
Perfios • Republic Of India, IN
What your average day would look like : .Collaborate with product and engineering teams to understand requirements and devise possible solutions.
Explore existing research papers, ideas and codebases ...Show more
Last updated: 2 days ago • Promoted
Technical Lead
Mphasis • India, India
Looking for Senior Ingenium Developer with 10+ years' experience and following skills.Experience in Mainframe O / S and Development using COBOL programming language & JCL.
Experience in development an...Show more
Last updated: 15 days ago • Promoted
Lead Data Engineer
Guidanz Inc • India, India
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 day ago • Promoted
Lead Data Engineer
Ironbook AI • India, India
We are seeking an experienced and driven Lead Data Engineer to spearhead the.AI use cases across the organization.Minimum 7 years of experience in data engineering, with at.Strong hands-on experien...Show more
Last updated: 11 hours ago • Promoted • New!
Project Manager – Data Engineering & Analytics
Brillio • India
We are looking for a skilled Technical Project Manager to lead and deliver projects in data engineering and analytics.You will manage cross-functional teams to execute data platform, pipeline, and ...Show more
Last updated: 30+ days ago • Promoted
Azure Data Lead / Architect
Tech Mahindra • India, India
Role : Azure Data Lead / architect.Expertise in CI-CD, Pyspark, SQL.Expertise in ADF, ADLS & ADB (Mandatory).Experience in "Azure Services / Key Vault".
Experience in "GIT / Jenkins / Azure DevOps".Show more
Last updated: 21 hours ago • Promoted • New!
Data Engineering Lead
HCLTech • Chennai, Republic Of India, IN
Lead and manage a team of technical professionals working on sql, data analysis, and python projects.Design, develop, and implement complex sql queries and scripts for data analysis.Analyze data se...Show more
Last updated: 2 days ago • Promoted
Lead Data Engineer
Confidential • India
Expertise in big data technologies such as Apache Spark and real-time streaming technologies like Apache Kafka.Strong programming skills in Python, Java, C++, SQL etc.
Advanced knowledge of a major ...Show more
Last updated: 11 hours ago • Promoted • New!
Technical Team Lead
Confidential • Republic Of India, IN
The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics.Your role will involve strategizing, project management, part staff managemen...Show more
Last updated: 2 days ago • Promoted
Technical Lead
RapidBrains • India, India
We are looking for an experienced Technical Lead who can architect scalable systems, mentor development teams, and guide complex projects from concept to deployment.
You’ll partner closely with Prod...Show more
Last updated: 1 day ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • India, India
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 21 hours ago • Promoted • New!
Technical Team Lead
Indexnine Technologies • Pune, Republic Of India, IN
We are seeking a highly experienced and driven Full Stack Technical Lead with strong development expertise, especially in Node.
The ideal candidate should be technically sound, cloud-savvy (preferab...Show more
Last updated: 19 days ago • Promoted