Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)AIMLEAP • Gurgaon, Republic Of India, IN

Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)

AIMLEAP • Gurgaon, Republic Of India, IN

12 hours ago

Job description

Data Engineering Manager – Web Crawling & Pipeline Architecture

Experience : 7 to 12 Years

Location : Remote / Bangalore

Engagement : Full-time

Positions : 2

Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT

Industry : IT / Data / AI / E-commerce / FinTech / Healthcare

Notice Period : Immediate

What We Are Looking For

Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..

Responsibilities

Lead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices.

Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.

Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.

Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies.

Define and enforce data quality, validation, and security measures across all data flows and pipelines.

Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.

Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.

Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure.

Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling

Qualifications

Bachelor's or master's degree in engineering, Computer Science, or related field.

7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems .

Strong expertise in Python, SQL , and modern data processing practices.

Experience working with Airflow, Celery , or similar workflow automation tools.

Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture.

Hands-on experience with cloud data platforms (AWS / GCP / Azure).

Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).

Strong analytical, architectural, and leadership skills.

Create a job alert for this search

Engineering Manager • Gurgaon, Republic Of India, IN

Related jobs

Manager – Data Analytics

TP • Gurgaon, India

Hiring : Manager – Data Analytics.Location : Gurugram | Experience : 5 + years.We’re looking for a Manager – Analytics (Technical) to lead strategic programs, drive data-backed decision-making, and el...Show more

Last updated: 7 days ago • Promoted

Manager - data science and ai

Sirius AI • Gurgaon, Haryana, India

Engage with clients to understand their business objectives and challenges, providing data-driven recommendations and AI / ML solutions that enhance decision-making and deliver tangible value.Transla...Show more

Last updated: 2 hours ago • Promoted • New!

Engineering manager

AiSensy • Gurgaon, Haryana, India

Ai Sensy is a Whats App-based Marketing & Engagement platform helping businesses drive revenue through powerful customer engagement on Whats App. We’re trusted by 150,000+ businesses and work with t...Show more

Last updated: 2 days ago • Promoted

Engineering Manager

Cvent • Gurugram, Haryana, India

Cvent is a leading meetings, events, and hospitality technology provider with more than 4,800 employees and ~22,000 customers worldwide, including 53% of the Fortune 500. Founded in 1999, Cvent deli...Show more

Last updated: 30+ days ago • Promoted

Engineering manager

Parentune.com • Gurgaon, Haryana, India

Relevant experience : 5-10 years.Shortlisting criterion : Excellence in Backend Engineering, Proven leadership for a team of 4-15 Software Engineers. Should have solved real world problems with techno...Show more

Last updated: 2 days ago • Promoted

Engineering Manager - Full stack

Taggd • Gurugram, Haryana, India

We’re building an enterprise-grade AI-powered recruitment platform with scalable services, polished UIs, and reliable integrations. You’ll lead a small, high-leverage team that ships fast, measures ...Show more

Last updated: 13 days ago • Promoted

Analytics manager

Noon • Gurgaon, Haryana, India

We’re building an ecosystem of digital products and services that power everyday life across the Middle East—fast, scalable, and deeply customer-centric. Our mission is to deliver to every door ever...Show more

Last updated: 2 hours ago • Promoted • New!

Data science / ai leader

Intelliserve Konsulting Pvt Ltd • Gurgaon, Haryana, India

Organization- Leading Global Management Consulting Organization( One of the BIG 3 Consulting firms).Role : - Leadership Data Science / AI Consulting role. Expertise req – AI / Generative AI / Data Sci...Show more

Last updated: 2 hours ago • Promoted • New!

Manager, Site Reliability Engineering

Cvent • Gurugram, Haryana, India

Cvent is looking for a Manager, Site Reliability Engineering to help us scale our systems and ensure stability, reliability and performance and rapid deployments of our platform.We build teams that...Show more

Last updated: 30+ days ago • Promoted

Technical project manager

Lepton Software • Gurgaon, Haryana, India

We are seeking a highly skilled Technical Project Manager (TPM) to lead process optimization and automation efforts within organization. We’re looking for a seasoned Technical Project Manager with a...Show more

Last updated: 2 hours ago • Promoted • New!

Senior manager analytics

Snapdeal • Gurgaon, Haryana, India

Job Title : Senior Manager / Manager - Data Science & Analytics.We are looking for a Senior Manager / Manager - Data Science & Analytics to lead the business analytics and insights charter and partner...Show more

Last updated: 2 hours ago • Promoted • New!

Engineering manager

SynapOne • Gurgaon, Haryana, India

Work Mode : 5 Days WFO mode only.Prior experience managing cross-functional teams (Tech + Analytics) preferred.Experience in e-commerce or high-traffic consumer platforms is a plus.Strong technical ...Show more

Last updated: 2 hours ago • Promoted • New!

Engineering manager-ecommerce startup

Taglynk • Gurgaon, Haryana, India

We are seeking a dynamic and highly skilled Engineering Manager with a solid foundation in full-stack development to lead our engineering team. The ideal candidate will combine hands-on technical ex...Show more

Last updated: 2 hours ago • Promoted • New!

Engineering Manager - Data Backend [T500-21479]

REA Cyber City • Gurgaon, India

In 1995, in a garage in Melbourne, Australia, REA Group was born from a simple question : “Can we change the way the world experiences property?”. Fast forward 30 years, REA Group is a market leader ...Show more

Last updated: 6 days ago • Promoted

Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)

AIMLEAP • gurgaon, haryana, in

Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare . Experience working with cloud platforms such ...Show more

Last updated: 17 hours ago • Promoted • New!

Senior data engineer

KOGTA FINANCIAL • Gurgaon, Haryana, India

ETL & Data Warehouse Developer with strong expertise in AWS data services.As a key member of our data engineering team, you will be responsible for designing, developing, and optimizing ETL pipelin...Show more

Last updated: 2 days ago • Promoted

Product engineering manager

Walker Digital Table Systems, LLC • Gurgaon, Haryana, India

Walker Digital table Systems (WDTS) is a US based MNC, headquartered in Las Vegas & the leader in networked table game solutions. WDTS develops innovations for Asian Pacific gaming markets including...Show more

Last updated: 2 hours ago • Promoted • New!

Data Product Manager

Mastercard • Gurugram, HR, India

Mastercard powers economies and empowers people in 200+ countries and territories worldwide.Together with our customers, we’re helping build a sustainable economy where everyone can prosper.We supp...Show more

Last updated: 15 hours ago • Promoted • New!