Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • vijayapura, India
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • vijayapura, India
17 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • vijayapura, India
Related jobs
Medical Students for AI Training (Remote).Are you a medical student with a passion for advancing AI? We are seeking medical experts to contribute to the training and refinement of cutting-edge arti...Show more
We're Hiring "Front End Developer (Remote)" | Earn up to $2500 per month.Contribute to training and refining cutting-edge AI systems.
Adopt a “user mindset” to produce natural and realistic data for...Show more 
Data Analysts or Data Scientists.In this role, you will work with datasets, apply analytical methods, and provide insights that improve AI performance.
Identify and source datasets relevant to AI tr...Show more 
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • vijayapura, India
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 5 hours ago • Promoted • New!
Global SC Solutions Product Owner
Olympus Corporation • Doddaballapura, Karnataka, India
Objective of the Job Global Supply Chains digitalize with pace and innovation.Your role at Olympus is to guide us through this revolution, by acting as a product owner and solution expert in the g...Show more
Last updated: 7 days ago • Promoted
Azure AI Foundry Developer
Undocked • vijayapura, India
At Undocked, we help companies excel in e-commerce by delivering bespoke optimizations and cutting-edge analytics.Our experiences in retail and supply chain product strategy, technology, and operat...Show more
Last updated: 17 hours ago • Promoted • New!
AI / ML Engineer – LLM & Agentic AI Systems (3 to 9 yrs)
AIMLEAP • vijayapura, India
AI / ML Engineer – LLM & Agentic AI Systems.Tech in Computer Science, AI / ML, or related field.LLM and agentic AI development.
AI pipelines, APIs, and integrations.LangChain, LlamaIndex, AutoGen.AI sys...Show more
Last updated: 5 hours ago • Promoted • New!
Frontend Developer
Jobs Ai • vijayapura, India
Hiring Remote Frontend Developers | Earn.This is an ideal role for early-career developers seeking to gain.Whether you're a recent graduate, self-taught programmer, or switching into tech, this rol...Show more
Last updated: 4 hours ago • Promoted • New!
Guidewire Policycenter dveloper
PwC Acceleration Center India • Doddaballapura, Karnataka, India
Job Summary : Minimum of 4 to 10 years of experience Role : Policy Center Configuration developer Minimum Degree Required Bachelor’s Degree Willingness to work Second Shift (2 pm IST to 11 pm IST) t...Show more
Last updated: 30+ days ago • Promoted
RPA Developer
hyprtask • Vijayapura, Karnataka, India
We are seeking a proactive and experienced RPA Developer to design build and optimize automation solutions for real-time business processes within the healthcare and technology domains.The role foc...Show more
Last updated: 30+ days ago • Promoted
Medical AI Training Intern (Remote | $2K–$3K / Month)
Get Jobs • vijayapura, India
Remote
Last updated: 4 hours ago • Promoted • New!
Faculty for Computer Science Engineering - GITAM School of Technology - GITAM ( Deemed to be University )
GITAM Deemed University • Dodda Ballapur, Karnataka, India
Faculty for Computer Science Engineering - GITAM School of Technology - GITAM ( Deemed to be University ).The Department of Computer Science Engineering at GITAM invites bright and young faculty fo...Show more
Last updated: 30+ days ago • Promoted
Freelance Writer
Prime Jobs • vijayapura, India
We're Hiring "Content Specialist (Freelance / Remote)" | Earn up to $2500 per month.Outsmart Artificial Intelligence with Your Human Brilliance.
Join a global community of talented writers to shape th...Show more
Last updated: 4 hours ago • Promoted • New!
Freelance Machine Learning Engineer
Leading MNC • vijayapura, India
Freelance Machine Learning Engineer.The candidate should have a minimum of 10+ yrs.If you're looking for freelance / part time opportunity (along with your day job) & a chance to work with the top 0...Show more
Last updated: 11 days ago • Promoted
Linfox - Site Security Manager
Linfox • Chikkaballapura, India
Description : Job Title : Site Security Manager Location : Chikkaballapura, Karnataka Department : Security & SafetyShow more
Last updated: 30+ days ago • Promoted
Lead AI Engineer
APPIT Software Inc • Doddaballapura, Karnataka, India
Role : Lead AI Engineer Experience : 8+ yrs Location : MG Road, Bangalore Work Mode : Hybrid (3 days from office) About the Role We are seeking a highly skilled Lead AI Engineer to architect, build ...Show more
Last updated: 5 hours ago • Promoted • New!
Data Analyst
Tata Consultancy Services • Doddaballapura, Karnataka, India
Greetings from TCS! Job Title : Data Analyst Required Skillset : Visualization and Reporting Tools (e.PowerBI, Tableau), good data analytics and better communication skills.Location : Bangalore Exper...Show more
Last updated: 10 days ago • Promoted
Front End Developer (Remote)
Get Jobs • vijayapura, India
Remote
Last updated: 4 hours ago • Promoted • New!
Civil Estimation & Tendering Engineer
NEO HEIGHTS BUILDERS & PROMOTERS PRIVATE LIMITED • Doddaballapura, Karnataka, India
Job Description : Civil Estimation & Tendering Engineer Position : Estimation & Tendering Engineer / Manager Department : Tendering / Estimation / Contracts Location : Bangalore Experience Required : 6–10...Show more
Last updated: 7 days ago • Promoted
Data Scientist (Remote)
Jobs Ai • vijayapura, India
Remote
Last updated: 4 hours ago • Promoted • New!
Medicine Researcher
Turing • vijayapura, India
Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems.
Turing helps customers in two ways : Working with the world’s leading AI...Show more
Last updated: 4 hours ago • Promoted • New!