Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • Gurgaon, Republic Of India, IN
No longer accepting applications
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • Gurgaon, Republic Of India, IN
23 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • Gurgaon, Republic Of India, IN
Related jobs
About the company Fast-track your career with the Marlee Talent Pool.We're not just matching you with your ideal roles but unlocking your long-term career potential.
Marlee goes above and beyond by ...Show more
We have more than 560 consultants who bring together vast industry experience and diverse insight, operating globally through 63 offices in 36 countries spanning across Europe, the Americas, Asia P...Show more
Aws Tech Lead - Contract
Gravity Infosolutions, Inc. • Gurgaon, Republic Of India, IN
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 7 hours ago • Promoted • New!
Lead Applied AI Engineer
Taggd • Gurugram, Haryana, India
Applied AI / LLMs; solid traditional ML).We’re building agentic AI for recruitment workflows—sourcing, screening, interview assistance, and offer orchestration.
You’ll own LLM / agent design, retrieval,...Show more
Last updated: 12 days ago • Promoted
Technical Lead - Finance Solutions / FinTech
Talent Pro • Delhi NCR, India
Key Responsibilities : - Lead end-to-end implementation projects for enterprise fintech clients - Translate client requirements into detailed implementation...Show more
Last updated: 30+ days ago • Promoted
Tech Lead
Marlee • Gurugram, Haryana, IN
Quick Apply
Last updated: 30+ days ago
NetSuite Technical - Lead
Egon Zehnder • Gurugram, HR, IN
Quick Apply
Last updated: 30+ days ago
Full-Stack / Python Developer (Web Scraping & Automation Specialist)
Youngun • Gurugram, Haryana, India
Flexible, results-driven environment.MeldIt develops large-scale social media automation and data intelligence solutions.
Our platform manages campaigns across multiple brands and collects high-volu...Show more
Last updated: 3 days ago • Promoted
Mamaearth - Team Lead - Website Operations
HONASA CONSUMER LIMITED • Gurugram, India
Role Overview - We are seeking a proactive and skilled Team Lead - Web Operations to oversee the end-to-end management of our brand websites across Shopify and Magent...Show more
Last updated: 4 days ago • Promoted
Product Tech Lead
Sun Life • Gurgaon, Haryana, India
You are as unique as your background experience and point of view.Here youll be encouraged empowered and challenged to be your best self.
Youll work with dynamic colleagues - experts in their fields...Show more
Last updated: 23 days ago • Promoted
Lead Business Analyst – PropTech Workflow Automation
BigStep Technologies • Gurugram, Haryana, India
We are looking for a Lead Business Analyst with strong experience in PropTech or real-estate technology solutions.This role acts as the bridge between the client and our workflow engineering team, ...Show more
Last updated: 2 days ago • Promoted
Technology Leader(10-18 years)
Airtel Digital • Gurugram, Haryana, India
You’ll play a key role in shaping the future of teacher capability development by leading our technical strategy, modernizing systems, and ensuring scalability and reliability at every level.This i...Show more
Last updated: 2 days ago • Promoted
Technical Lead
GradRight Inc. • Gurugram, Haryana, India
Our vision is to be the world’s leading Ed-Fin Tech company dedicated to making higher education accessible and affordable to all.
Our mission is to drive transparency and accountability in the glob...Show more
Last updated: 30+ days ago • Promoted
Senior-Technical Lead
Incedo • Gurgaon, Haryana, India
Incedo is a US-based consulting data science and technology services firm with over 3000 people helping clients.We help our clients achieve competitive advantage through.
Our uniqueness lies in brin...Show more
Last updated: 6 days ago • Promoted
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • gurgaon, haryana, in
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 10 hours ago • Promoted • New!
Full-Stack+AI Automation Tech Lead - Proptech
BigStep Technologies • Gurugram, Haryana, India
Full-Stack+AI Automation Tech Lead - Proptech.AI-first, cloud-native product engineering company.We are dedicated to leveraging cutting-edge AI and cloud technologies to transform business processe...Show more
Last updated: 20 hours ago • Promoted • New!
Technical Lead
Sodexo • Gurgaon, Haryana, India
As the Technical Lead for mobile and digital applications, you will play a crucial role in leading the design, development, and deployment of cutting-edge digital solutions.You will collaborate clo...Show more
Last updated: 24 days ago • Promoted
Team Lead II – Fullstack Developer
Real Time Data Services • Gurugram, Haryana, India
Lead a team of developers to design, develop, and maintain scalable and secure RESTful APIs and web services.Drive technical discussions, provide guidance, and ensure adherence to clean code practi...Show more
Last updated: 20 hours ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • Gurgaon, Haryana, India
Data Engineering Manager – Web Crawling & Pipeline Architecture Experience : 7 to 12 Years Location : Remote / Bangalore Engagement : Full-time Positions : 2 Qualification : B.Tech / MCA / Compute...Show more
Last updated: 11 hours ago • Promoted • New!
Tech Lead (Full Stack | React + Python)
Aumne AI • Gurgaon, Haryana, India
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.What You’ll Work On •...Show more
Last updated: 7 hours ago • Promoted • New!