Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)AIMLEAP • faridabad, haryana, in
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • faridabad, haryana, in
2 days ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • faridabad, haryana, in
Related jobs
%20Pvt.%20Ltd.)
![Technology Manager I - Data Engineering [T500-21640]](https://cdn-logos.talent.com/v1/logo/image?feedcode=jobrapido-pms&company_name=lululemon)
![Senior Manager Engineering - Fullstack [T500-21642]](https://cdn-logos.talent.com/v1/logo/image?feedcode=jobrapido-pms&company_name=Albertsons%20Companies%20India)
Product Manager
Melhor Group • South Delhi, Delhi, India
FYORA is an innovative AI-driven platform revolutionizing workforce automation through specialized AI agents.We empower businesses by streamlining processes, boosting efficiency, and optimizing ope...Show more
Last updated: 30+ days ago • Promoted
Engineer
PHOENIX CONTACT (I) Pvt. Ltd. • Faridabad, Haryana, India
Job Title : Senior Engineer – OT, IIoT and ISO Compliance.Location : Prithla, Palwal, Haryana.This role is pivotal in ensuring robust security, compliance, and operational excellence across the entir...Show more
Last updated: 13 days ago • Promoted
Lead Generation Manager
lOOM SOLAR PRIVATE LIMITED • Faridabad, Haryana, India
Job Title : - Lead Generation Manager.Rewards : On Time Pay, Certificates, New Skills, Opportunity to become Marketing Head.
The Lead Generation Manager is responsible for building and managing a str...Show more
Last updated: 4 days ago • Promoted
Engineering Manager
Tuebora Inc • faridabad, haryana, in
Santa Clara, CA, USA, and Bangalore, India, is a pioneer in access governance.The company offers a cloud-based Identity and Access Management (IAM) solution, covering the entire IAM lifecycle.With ...Show more
Last updated: less than 1 hour ago • Promoted • New!
Senior Project Manager
Yamaha Motor Solutions India • Faridabad, Haryana, India
We are seeking a highly skilled and experienced Senior Project Manager to lead Microsoft-focused projects involving multiple Customer Relationship Management (CRM) platforms.The role involves manag...Show more
Last updated: 6 days ago • Promoted
Technology Manager I - Data Engineering [T500-21640]
lululemon • faridabad, haryana, in
Lululemon is an innovative performance apparel company for yoga, running, training, and other athletic pursuits.Setting the bar in technical fabrics and functional design, we create transformationa...Show more
Last updated: less than 1 hour ago • Promoted • New!
SIEM Engineering Manager
SpeedMart • faridabad, haryana, in
Our client is a global IT services company that helps businesses with digital transformation with offices in India and the United States.
It helps businesses with digital transformation, provide IT ...Show more
Last updated: less than 1 hour ago • Promoted • New!
Engineering Manager
Confidential • faridabad, haryana, in
The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics.Your role will involve strategizing, project management, part staff managemen...Show more
Last updated: less than 1 hour ago • Promoted • New!
Engineering Manager
IDfy • faridabad, haryana, in
IDfy is an Integrated Identity Platform offering products and solutions for KYC, KYB, Background Verifications, Risk Assessment, and Digital Onboarding.
We establish trust while delivering a frictio...Show more
Last updated: less than 1 hour ago • Promoted • New!
Engineering Manager
Talentiser • faridabad, haryana, in
SOFTWARE ENGINEERING MANAGER- JAVA & SAAS.We’re seeking an Engineering Manager to oversee and accelerate the development.
This role involves strategic planning, team leadership, and technical oversi...Show more
Last updated: less than 1 hour ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Faridabad, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT.IT / Data / AI / E-commerce / FinTech / Healthcare.
Experience working with cloud platforms such as...Show more
Last updated: 2 days ago • Promoted
Databricks Lead
Mastech Digital • faridabad, haryana, in
Immediate joiner / Serving notice with less than 60 days / Notice is less than 60 days.We are looking for a highly skilled.
The ideal candidate will lead teams in designing, building, and optimizing sc...Show more
Last updated: less than 1 hour ago • Promoted • New!
Senior Software Engineering Manager
The Glove • faridabad, haryana, in
Must have minimum 13 to 15 years of Experience, & 2+ years in handling large teams.Must have handled large Cloud Software programs and application Development & Deployment as an Engineering owner a...Show more
Last updated: less than 1 hour ago • Promoted • New!
Senior Manager Engineering - Fullstack [T500-21642]
Albertsons Companies India • faridabad, haryana, in
About Albertsons Companies Inc.As a leading food and drug retailer in the United States, Albertsons Companies, Inc.Our well-known banners across the United States, including Albertsons, Safeway, Vo...Show more
Last updated: less than 1 hour ago • Promoted • New!
Project Manager – Smart Metering (NB-IoT / cellular IoT)
ZENNER India • Faridabad, Haryana, India
ZENNER Aquamet India Private Limited (ZAIPL) is a joint venture with ZENNER International GmbH Germany specializing in water metering, submetering, and digital solutions for the Indian market.With ...Show more
Last updated: 6 days ago • Promoted
Data Science Manager
ConsultMeIn • faridabad, haryana, in
PLEASE READ THE JOB DESCRIPTIONBEFORE YOU APPLY • • •.We’re Hiring : Manager – Data Science | Ahmedabad (Onsite).Are you ready to influence business-critical decisions at scale? My client — a global po...Show more
Last updated: less than 1 hour ago • Promoted • New!
Engineering Manager
Reliance Industries Limited • faridabad, haryana, in
Engineering Manager (Experience 25+ years in Design and Engineering of EPC project in respective discipline).Experience of working in hyperscalers, with ability to Lead Large team of inter-discipli...Show more
Last updated: less than 1 hour ago • Promoted • New!
Deputy Manager - Industrial Engineering
Havells India Ltd • Faridabad, Haryana, India
Conduct Time study, Line Balancing, VSM studies, Lean Layout Designing, manufacturing lead time reduction, etc.Conduct time study, line balancing studies & prepare norms.
Prepare Layouts based on le...Show more
Last updated: 17 days ago • Promoted