AI-Driven Data Engineering ManagerAIMLEAP • Republic Of India, IN
AI-Driven Data Engineering Manager
AIMLEAP • Republic Of India, IN
7 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Data Engineering Manager • Republic Of India, IN
Related jobs
Engineering Manager
HiLabs • Republic Of India, IN
We are looking for an experienced Data Engineering Manager to lead and scale our data engineering function.The ideal candidate will be responsible for designing, building, and maintaining high-qual...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineering Manager
EXL • Pune, Republic Of India, IN
We are seeking a highly skilled.The ideal candidate will bring deep technical expertise, strong leadership, and the ability to collaborate effectively with global stakeholders in a fast-paced envir...Show more
Last updated: 10 days ago • Promoted
Data Engineering Manager
Kissht • Republic Of India, IN
We are looking for a Tech Lead – Data Engineering to design, develop, and optimize data systems supporting our digital lending platforms.
The ideal candidate will have deep hands-on expertise in MyS...Show more
Last updated: 2 days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT.IT / Data / AI / E-commerce / FinTech / Healthcare.
Experience working with cloud platforms such as...Show more
Last updated: 4 hours ago • Promoted • New!
Data Engineering Lead
HiLabs • Republic Of India, IN
We are looking for an experienced Data Engineering Manager to lead and scale our data engineering function.The ideal candidate will be responsible for designing, building, and maintaining high-qual...Show more
Last updated: 30+ days ago • Promoted
AI Engineering Manager
Blend • Chennai, Republic Of India, IN
We are looking for a Manager / Lead AI Engineer with hands-on experience designing and deploying scalable AI solutions.In this role, you will be part of a cross-functional team working on cutting-edg...Show more
Last updated: 18 days ago • Promoted
Engineering Manager - Ii, Data Engineering Platform
Tamara • Republic Of India, IN
Data Engineering Manager II : Real-Time Data & Experimentation Platform (Remote, India).Saudi Arabia's first fintech unicorn.
GCC, with a mission to empower dreams through customer-centric financial ...Show more
Last updated: 25 days ago • Promoted
Data Engineering Manager
Matrimony.com Limited • Chennai, Republic Of India, IN
Work Location : Chennai (Work From Office).The Decision Support Systems (DSS) function enables data-driven decision-making across Matrimony.
Com by delivering high-quality data pipelines, analytical ...Show more
Last updated: 18 days ago • Promoted
Senior Data Engineering Manager
Kanerika Inc • Indore, Republic Of India, IN
Hyderabad, Indore and Ahmedabad (India).Develop and maintain data pipelines using Azure Data Factory (ADF) / Databricks for data integration and ETL processes.
Design, implement, and optimize Power B...Show more
Last updated: 25 days ago • Promoted
Project Manager – Data Engineering & Analytics
Brillio • India
We are looking for a skilled Technical Project Manager to lead and deliver projects in data engineering and analytics.You will manage cross-functional teams to execute data platform, pipeline, and ...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager - Azure Databricks
PwC India • Republic Of India, IN
Job Title : Azure Data Engineer.As a Manager in the Data Engineering practice, you will lead teams and client engagements to design, architect, and deliver enterprise-grade data platforms.You’ll dri...Show more
Last updated: 10 days ago • Promoted
Data Engineering Operations Manager
Saks Global India • Republic Of India, IN
Data Engineer and Operations Lead.Saks Global is the largest multi-brand luxury retailer in the world, comprising Saks Fifth Avenue, Neiman Marcus, Bergdorf Goodman, Saks OFF 5TH, Last Call and Hor...Show more
Last updated: 30+ days ago • Promoted
AI / ML Engineering Manager
ACL Digital • Chennai, Republic Of India, IN
Experience is a must in development and deployment of production-level AI models at enterprise scale.Drive innovation in AI / ML applications across various business domains and modalities (vision, l...Show more
Last updated: 30+ days ago • Promoted
Manager, Data Engineering
HiLabs • Republic Of India, IN
We are looking for an experienced Data Engineering Manager to lead and scale our data engineering function.The ideal candidate will be responsible for designing, building, and maintaining high-qual...Show more
Last updated: 7 days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • India, India
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 11 hours ago • Promoted • New!
Data and Analytics Engineering Manager
Orbia Pune • Pune, Republic Of India, IN
Data and Analytics Engineering Lead.If you are someone who can develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azu...Show more
Last updated: 2 days ago • Promoted
Data Engineering Manager
Blend • Chennai, Republic Of India, IN
We are looking for an experienced.In this role, you will build and maintain scalable data pipelines and architecture to support analytics, data science, and business intelligence initiatives.You’ll...Show more
Last updated: 4 days ago • Promoted
Engineering Manager - II, Data Engineering Platform
Tamara • India, India
Data Engineering Manager II : Real-Time Data & Experimentation Platform (Remote, India).Saudi Arabia's first fintech unicorn.
GCC, with a mission to empower dreams through customer-centric financial ...Show more
Last updated: 25 days ago • Promoted