Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)AIMLEAP • Gurgaon, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Gurgaon, Republic Of India, IN
1 day ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • Gurgaon, Republic Of India, IN
Related jobs![Engineering Manager - Data Backend [T500-21479]](https://cdn-logos.talent.com/v1/logo/image?feedcode=attb-in&company_name=REA%20Cyber%20City)
Engineering Manager - Data Backend [T500-21479]
REA Cyber City • Gurgaon, Haryana, India
About REA Group : In 1995, in a garage in Melbourne, Australia, REA Group was born from a simple question : “Can we change the way the world experiences property?” Could we? Yes.Fast forward 30 yea...Show more
Last updated: 9 days ago • Promoted
Manager – Data Analytics
TP • Gurgaon, Haryana, India
Hiring : Manager – Data Analytics Location : Gurugram | Experience : 5 + years We’re looking for a Manager – Analytics (Technical) to lead strategic programs, drive data-backed decision-making, and ...Show more
Last updated: 9 days ago • Promoted
Engineering Manager
Cvent • Gurugram, Haryana, India
Cvent is a leading meetings, events, and hospitality technology provider with more than 4,800 employees and ~22,000 customers worldwide, including 53% of the Fortune 500.
Founded in 1999, Cvent deli...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • gurugram, uttar pradesh, in
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 2 days ago • Promoted
Data Engineer
TerraGiG • gurgaon, haryana, in
Lead the design, development, and implementation of data solutions using AWS and Snowflake.Collaborate with cross-functional teams to understand business requirements and translate them into techni...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Grantify • gurgaon, haryana, in
Grantify is an innovative education platform that connects students and universities through a transparent admissions and tuition-matching ecosystem.
By aligning student budgets and academic aspirat...Show more
Last updated: 2 days ago • Promoted
Engineering Manager - Data
REA • Gurgaon, India
In 1995, in a garage in Melbourne, Australia, REA Group was born from a simple question : Can we change the way the world experiences property? Could we? Yes.
Fast forward 30 years, REA Group is a ma...Show more
Last updated: 30+ days ago • Promoted
Manager, Site Reliability Engineering
Cvent • Gurugram, Haryana, India
Cvent is looking for a Manager, Site Reliability Engineering to help us scale our systems and ensure stability, reliability and performance and rapid deployments of our platform.We build teams that...Show more
Last updated: 30+ days ago • Promoted
Staff Data Engineer
ANSR Summit Consulting • Gurgaon, India
Participate in the entire data pipeline lifecycle, focusing on building data lakes, data warehouses and related processes.
Contribute to the design and documentation of our data lake platform.Partne...Show more
Last updated: 30+ days ago • Promoted
Platform Engineering Manager
Axslogic Pte Ltd • Gurgaon, Haryana, India
Job Title : Platform Engineering Manager Location : Gurgaon Employment Type : Full-Time Department : Analytics About Axslogic Axslogic is a fintech analytics enterprise software company that helps...Show more
Last updated: 30+ days ago • Promoted
Senior Manager, Software Engineering
Palo Alto Networks • Gurugram, Haryana, India
At Palo Alto Networks® everything starts and ends with our mission : .Being the cybersecurity partner of choice, protecting our digital way of life.
Our vision is a world where each day is safer and m...Show more
Last updated: 1 day ago • Promoted
Engineering Manager - Full stack
Taggd • Gurugram, Haryana, India
We’re building an enterprise-grade AI-powered recruitment platform with scalable services, polished UIs, and reliable integrations.
You’ll lead a small, high-leverage team that ships fast, measures ...Show more
Last updated: 14 days ago • Promoted
Manager AI
Birdeye • Gurgaon, Haryana, India
Manager- AI Why Birdeye? Birdeye is the highest-rated reputation, social media, and customer experience platform for local businesses and brands.
Over 150,000 businesses use Birdeye’s AI-powered p...Show more
Last updated: 9 days ago • Promoted
Engineering Manager Full Stack
REA • Gurgaon, India
In 1995, in a garage in Melbourne, Australia, REA Group was born from a simple question : Can we change the way the world experiences property?Could we? Yes.
Fast forward 30 years, REA Group is a mar...Show more
Last updated: 30+ days ago • Promoted
E-commerce Technical Project Manager( Bigcommerce / Shopify)
Upbott Consulting, Inc • gurugram, uttar pradesh, in
E-commerce Technical Project Manager.BigCommerce or Shopify projects.Candidates must have led end-to-end e-commerce implementations specifically on.
This role requires someone who understands the Bi...Show more
Last updated: 2 days ago • Promoted
Engineering Manager (Loans)
Bachatt • gurugram, uttar pradesh, in
Bachatt has transformed how India saves.Now as we foray into solving the credit needs of our customers, we are looking to hire a talented.
Lead and mentor a team of high-functioning engineers who ta...Show more
Last updated: 6 hours ago • Promoted • New!
Senior Data Engineer
Primesoft Inc • gurgaon, haryana, in
Primesoft Enterprise IT Services Pvt.As a Software Engineer II - Data, you will contribute to the design and development of data systems including pipelines, APIs, analytics, AI and machine learnin...Show more
Last updated: 30+ days ago • Promoted
Data Science Manager
RenewBuy • Gurgaon, Haryana, India
Data Science Manager Description of the role : We are looking for a Data Science Manager to lead and fully own the development and delivery of multiple Artificial Intelligence teams.To be success...Show more
Last updated: 9 days ago • Promoted