Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • bhopal, India
No longer accepting applications
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • bhopal, India
18 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • bhopal, India
Related jobs
Turing is looking for experienced Full Stack Developers to build modern solutions that power AI products and evaluation workflows.
React / Angular / Vue) to implement features, improve code quality and ...Show more
Job Title : Tech Lead – Full Stack.The ideal candidate brings deep expertise in.You’ll lead technical strategy, ensure system reliability, and drive engineering excellence across teams.Architect and...Show more We’re looking for experienced Python developers to collaborate with one of the world’s top Large Language Model (LLM) companies.
Your work will directly help improve how AI models think, reason, and...Show more 
Tech Lead (Full Stack | React + Python)
Aumne AI • bhopal, madhya pradesh, in
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.Lead UI architecture ...Show more
Last updated: 9 hours ago • Promoted • New!
Lead Data Engineer
Ironbook AI • bhopal, madhya pradesh, in
We are seeking an experienced and driven Lead Data Engineer to spearhead the.AI use cases across the organization.Minimum 7 years of experience in data engineering, with at.Strong hands-on experien...Show more
Last updated: 5 hours ago • Promoted • New!
Lead Data Engineer
Confidential • bhopal, madhya pradesh, in
Expertise in big data technologies such as Apache Spark and real-time streaming technologies like Apache Kafka.Strong programming skills in Python, Java, C++, SQL etc.
Advanced knowledge of a major ...Show more
Last updated: 5 hours ago • Promoted • New!
Lead DevOps Engineer
Unified Infotech • bhopal, madhya pradesh, in
Unified Infotech is a 14-year-old, multi-award-winning digital transformation partner accelerating business growth for.Fortune 500s, MNCs, SMEs, and high-potential Startups.We empower organizations...Show more
Last updated: 30+ days ago • Promoted
Webflow Developer (Finsweet Client-First + CMS-Driven Build)
RB Law • bhopal, madhya pradesh, in
We need a Webflow developer who can.Webflow using best practices, correct naming conventions, and scalable CMS structures.
The goal is to ensure the marketing team can easily maintain and expand the...Show more
Last updated: 5 hours ago • Promoted • New!
Tech Lead –.Net / Python & AI
Skillvera • Bhopal, Madhya Pradesh, India
Technical Skills & Stack Requirements : Programming Expertise : Advanced proficiency in.Net and Python for backend architecture, API development, and service orchestration.Generative AI Framework...Show more
Last updated: 3 hours ago • Promoted • New!
Full Stack Engineer
Programmers.io • bhopal, madhya pradesh, in
We are seeking highly skilled Senior.Laravel and modern frontend frameworks (Vue.The candidate should have deep technical expertise, leadership ability, and experience architecting scalable web sol...Show more
Last updated: 20 days ago • Promoted
Senior Full Stack Developer (DocuSign CLM)
Gravity Infosolutions, Inc. • bhopal, madhya pradesh, in
Job Title : Senior Full Stack Developer (DocuSign CLM).We are looking for a Senior Full Stack Developer with strong.DocuSign CLM workflows, templates, and document automation.Develop full-stack solu...Show more
Last updated: 9 hours ago • Promoted • New!
Remote Full-Stack Developer
Turing • bhopal, madhya pradesh, in
Remote
Last updated: 5 hours ago • Promoted • New!
Salesforce Tech Lead- Contract
Gravity Infosolutions, Inc. • Bhopal, Madhya Pradesh, India
Role : Salesforce Tech Lead Experience : 5+ Years Contract Duration : 6 months Location : Remote Shift : 11am- 8pm IST Participate in refining and scoping upcoming sprint work.Assist solution arch...Show more
Last updated: 11 hours ago • Promoted • New!
Technical Lead (Node+React+Golang) Remote
BigStep Technologies • bhopal, madhya pradesh, in
Remote
Last updated: 1 hour ago • Promoted • New!
Remote Sr. Python Developer
Turing • bhopal, madhya pradesh, in
Remote
Last updated: 5 hours ago • Promoted • New!
Lead Test Automation Architect – DevOps & CI / CD
Applicantz • bhopal, madhya pradesh, in
THIS IS A CONTRACT POSITION WITH ONE OF THE LARGEST, GLOBAL, TECHNOLOGY LEADER.The ideal candidate will create highly scalable automation frameworks, streamline build and deployment processes, impr...Show more
Last updated: 3 days ago • Promoted
Lead Data Engineer
Guidanz Inc • bhopal, madhya pradesh, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 day ago • Promoted
Senior Full Stack Engineer
Black Piano • bhopal, madhya pradesh, in
Lead development of software applications across client portfolio with a focus on MERN or MEAN frameworks within an Azure, AWS or GCP environment.
Lead the continuous development of bespoke web appl...Show more
Last updated: 30+ days ago • Promoted
Senior Implementation Specialist (PowerScale / Isilon)
Norwin Technologies • bhopal, madhya pradesh, in
Senior Implementation Specialist.Interested candidates can share profile on anjalihb@norwintechnologies.Administer and optimize PowerScale clusters across Multi-potocols (NFS / SMB / HDFS / S3) environme...Show more
Last updated: 1 day ago • Promoted
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • bhopal, madhya pradesh, in
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 9 hours ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • Bhopal, Madhya Pradesh, India
Data Engineering Manager – Web Crawling & Pipeline Architecture Experience : 7 to 12 Years Location : Remote / Bangalore Engagement : Full-time Positions : 2 Qualification : B.Tech / MCA / Compute...Show more
Last updated: 11 hours ago • Promoted • New!