Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)AIMLEAP • vijayapura, India
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • vijayapura, India
16 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • vijayapura, India
Related jobs
![Engineer - Databases [Returnship Program]](https://cdn-logos.talent.com/v1/logo/image?feedcode=joveo-kenvue&company_name=Kenvue)
These fully remote, long-term freelance roles are ideal for engineers who can build scalable data pipelines, work with modern cloud-native data stacks, and support large-scale enterprise data initi...Show more
Business System Analyst
HCLTech • Doddaballapura, Karnataka, India
We’re Hiring : Business Analyst – Manufacturing & Logistics Location : Bangalore Experience : 7–10 years Are you passionate about Manufacturing, Logistics, and Digital Transformation ? Join us and ...Show more
Last updated: 4 days ago • Promoted
Key account manager
Exide Energy Solutions Ltd • Doddaballapura, Karnataka, India
Exide Energy Solutions Limited is a wholly owned Subsidiary of Exide Industries Limited, leading battery manufacturing and distribution company in India with annual revenues of $1.Exide Energy Solu...Show more
Last updated: 10 hours ago • Promoted • New!
Full-Stack Developer - 20414
Turing • vijayapura, India
Turing is looking for experienced Full Stack Developers to build modern solutions that power AI products and evaluation workflows.
LLM behavior with real-world user needs.This is a remote, flexible ...Show more
Last updated: 30+ days ago • Promoted
Engineer - Databases [Returnship Program]
Kenvue • Devanhalli, Karnataka, India
Kenvue is currently recruiting for a : .Engineer - Databases [Returnship Program].At Kenvue , we realize the extraordinary power of everyday care.
Built on over a century of heritage and rooted in sc...Show more
Last updated: 11 days ago • Promoted
Faculty for Computer Science Engineering - GITAM School of Technology - GITAM ( Deemed to be University )
GITAM Deemed University • Dodda Ballapur, Karnataka, India
Faculty for Computer Science Engineering - GITAM School of Technology - GITAM ( Deemed to be University ).The Department of Computer Science Engineering at GITAM invites bright and young faculty fo...Show more
Last updated: 26 days ago • Promoted
Linfox - Site Security Manager
Linfox • Chikkaballapura, India
Description : Job Title : Site Security Manager Location : Chikkaballapura, Karnataka Department : Security & SafetyShow more
Last updated: 30+ days ago • Promoted
Civil estimation & tendering engineer
NEO HEIGHTS BUILDERS & PROMOTERS PRIVATE LIMITED • Doddaballapura, Karnataka, India
Job Description : Civil Estimation & Tendering Engineer.Position : Estimation & Tendering Engineer / Manager.Department : Tendering / Estimation / Contracts.
Experience Required : 6–10years (Civil contrac...Show more
Last updated: 10 hours ago • Promoted • New!
Client-Focused Interior Designer
HomeLane • Devanhalli, Republic Of India, IN
Design Consultant (FOFO) Bangalore, Design.Be the Lead Designer for a set of HomeLane customers, from the first meeting to order booking to design finalization Collaborate with the customer, unders...Show more
Last updated: 2 days ago • Promoted
Data Engineer - Fully Remote (Global Data Platform & Analytics Projects)
SkillsCapital • vijayapura, India
Remote
Last updated: 5 hours ago • Promoted • New!
Data centre infra manager - server
Exide Energy Solutions Ltd • Doddaballapura, Karnataka, India
Exide Energy Solutions Limited is a wholly owned Subsidiary of Exide Industries Limited, leading battery manufacturing and distribution company in India with annual revenues of $1.Exide Energy Solu...Show more
Last updated: 30+ days ago • Promoted
Civil Estimation & Tendering Engineer
NEO HEIGHTS BUILDERS & PROMOTERS PRIVATE LIMITED • Doddaballapura, Karnataka, India
Job Description : Civil Estimation & Tendering Engineer Position : Estimation & Tendering Engineer / Manager Department : Tendering / Estimation / Contracts Location : Bangalore Experience Required : 6–10...Show more
Last updated: 2 days ago • Promoted
Innovation Ecosystem Manager
GITAM Deemed University • Dod Ballāpur, Republic Of India, IN
Venture Coach (VC) – Venture Development Centre.Reporting to the Site Head – VDC Bengaluru.Coach ventures(Faculty / Alumni / Students) assigned and fulfil all requirements for completing RSG (NU-IDEA R...Show more
Last updated: 17 days ago • Promoted
Residential Design Specialist
HomeLane • Devanhalli, Republic Of India, IN
Design Consultant (FOFO) Bangalore, Design.Be the Lead Designer for a set of HomeLane customers, from the first meeting to order booking to design finalization Collaborate with the customer, unders...Show more
Last updated: 2 days ago • Promoted
Startup Incubation Specialist
GITAM Deemed University • Dod Ballāpur, Republic Of India, IN
Venture Coach (VC) – Venture Development Centre.Reporting to the Site Head – VDC Bengaluru.Coach ventures(Faculty / Alumni / Students) assigned and fulfil all requirements for completing RSG (NU-IDEA R...Show more
Last updated: 17 days ago • Promoted
Global sc solutions product owner
Olympus Corporation • Doddaballapura, Karnataka, India
Global Supply Chains digitalize with pace and innovation.Your role at Olympus is to guide us through this revolution, by acting as a product owner and solution expert in the global Supply Chain.You...Show more
Last updated: 10 hours ago • Promoted • New!
Interior design consultant
HomeLane • Devanahalli, Karnataka, India
Design Consultant (FOFO) Bangalore, Design.Be the Lead Designer for a set of Home Lane customers, from the first meeting to order booking to design finalization Collaborate with the customer, under...Show more
Last updated: 10 hours ago • Promoted • New!
Home Design Solutions Expert
HomeLane • Devanhalli, Republic Of India, IN
Design Consultant (FOFO) Bangalore, Design.Be the Lead Designer for a set of HomeLane customers, from the first meeting to order booking to design finalization Collaborate with the customer, unders...Show more
Last updated: 2 days ago • Promoted
Interior Design Consultant
HomeLane • Devanahalli, Karnataka, India
Design Consultant (FOFO) Bangalore, Design.Be the Lead Designer for a set of HomeLane customers, from the first meeting to order booking to design finalization Collaborate with the customer, unders...Show more
Last updated: 2 days ago • Promoted