Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • indore, madhya pradesh, in
No longer accepting applications
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • indore, madhya pradesh, in
1 day ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • indore, madhya pradesh, in
Related jobs
Technical Lead - Full Stack to join our innovative team at DigiValet.In this pivotal role, you will spearhead our product development team, contributing to the creation of cutting-edge software app...Show more
Who We Are Ccube is a fast-growing technology company transforming how businesses harness the power of Apps, Data, and AI.
We help organizations thrive by delivering integrated digital solutions tha...Show more
Fortune 500 brands and trailblazing startups across 17+ industries.We rank among the leaders in areas like application development and AI / ML, and our people-first culture has earned us multiple Bes...Show more
Fortune 500 brands and trailblazing startups across 17+ industries.We rank among the leaders in areas like application development and AI / ML, and our people-first culture has earned us multiple Bes...Show more
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • indore, madhya pradesh, in
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 7 hours ago • Promoted • New!
Salesforce Tech Lead- Contract
Gravity Infosolutions, Inc. • Indore, Madhya Pradesh, India
Role : Salesforce Tech Lead Experience : 5+ Years Contract Duration : 6 months Location : Remote Shift : 11am- 8pm IST Participate in refining and scoping upcoming sprint work.Assist solution arch...Show more
Last updated: 8 hours ago • Promoted • New!
Project Leader - IT
Sequifi • Indore, Madhya Pradesh, India
Position Title : Project Leader.HR Tech company focused on transforming workforce and talent operations through next-generation automation, intelligent workflows, and scalable digital solutions.Our ...Show more
Last updated: 7 days ago • Promoted
Site Reliability Engineer (SRE)
HRhelpdesk • Indore, Madhya Pradesh, India
Company is a rapidly growing, private equity backed SaaS product company and provides cloud-based solutions.As a Site Reliability Engineer (SRE), you will be responsible for building and maintainin...Show more
Last updated: 9 days ago • Promoted
AWS Data Engineer
Tata Consultancy Services • Indore, Madhya Pradesh, India
TCS is Hiring AWS Data Engineer For Indore location.Please apply only if you are available for a In-person interview at TCS Indore on 6th December 2025 (Saturday).
AWS Snowflake, Python, Pyspark.Sup...Show more
Last updated: 30+ days ago • Promoted
Technology Partner
ClearTrail Technologies • Indore, Madhya Pradesh, India
At ClearTrail, work is more than ‘just a job’.Our calling is to develop solutions that empower those dedicated to keep their people, places and communities safe.
For over 24 years, law enforcement &...Show more
Last updated: 23 days ago • Promoted
Tech Lead –.Net / Python & AI
Skillvera • Indore, Madhya Pradesh, India
Technical Skills & Stack Requirements : Programming Expertise : Advanced proficiency in.Net and Python for backend architecture, API development, and service orchestration.Generative AI Framework...Show more
Last updated: less than 1 hour ago • Promoted • New!
Technical Manager
INFOCRATS • Indore, Madhya Pradesh, India
Technical Project / Delivery Manager (Microsoft.We are seeking a highly skilled and experienced.NET technologies to lead end-to-end delivery of enterprise-grade software solutions.You will act as the...Show more
Last updated: 30+ days ago • Promoted
Technical Lead - Full Stack
DigiValet • Indore, Madhya Pradesh, India, India, India
Quick Apply
Last updated: 30+ days ago
Team Leader-3
TaskUs • Indore, Madhya Pradesh, India
TaskUs is a provider of outsourced digital services and next-generation customer experience to fast-growing technology companies helping its clients represent protect and grow their brands.Leveragi...Show more
Last updated: 16 days ago • Promoted
Webflow Developer (Finsweet Client-First + CMS-Driven Build)
RB Law • indore, madhya pradesh, in
We need a Webflow developer who can.Webflow using best practices, correct naming conventions, and scalable CMS structures.
The goal is to ensure the marketing team can easily maintain and expand the...Show more
Last updated: 3 hours ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • Indore, Madhya Pradesh, India
Data Engineering Manager – Web Crawling & Pipeline Architecture Experience : 7 to 12 Years Location : Remote / Bangalore Engagement : Full-time Positions : 2 Qualification : B.Tech / MCA / Compute...Show more
Last updated: 8 hours ago • Promoted • New!
Tech Lead (Full Stack | React + Python)
Aumne AI • indore, madhya pradesh, in
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.Lead UI architecture ...Show more
Last updated: 7 hours ago • Promoted • New!
Full Stack Lead
Ccube • Indore, Madhya Pradesh, IN
Quick Apply
Last updated: 18 days ago
AI Team Lead ID44464
AgileEngine • Indore, MP, in
Quick Apply
Last updated: 26 days ago
Senior Technical Recruiter
EngineerBabu • Indore, Madhya Pradesh, India
Senior Technical Recruiter Tech Hiring.The ideal candidate will come from a.Software Engineers DevOps QA Cloud and Product Engineers.
Execute proactive sourcing strategies using platforms like.Link...Show more
Last updated: 30+ days ago • Promoted
ClearTrail Technologies - Technology Partner
ClearTrail Technologies • Indore, India
At ClearTrail, work is more than 'just a job'.Our calling is to develop solutions that empower those dedicated to keep their people, places and communities safe.For over 24 years, law enf...Show more
Last updated: 14 days ago • Promoted
AI Full Stack Team Lead ID44464
AgileEngine • Indore, MP, in
Quick Apply
Last updated: 26 days ago