Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • Salem,Tamil Nadu, IN
No longer accepting applications
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • Salem,Tamil Nadu, IN
1 day ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • Salem,Tamil Nadu, IN
Related jobs
Tech Lead –.Net / Python & AI
Skillvera • Salem,Tamil Nadu, IN
Technical Skills & Stack Requirements : .API development, and service orchestration.AWS or Azure cloud architecture.Bedrock, Lambda, ECS / EKS, Step Functions, S3, and SageMaker and Azure equivalents.U...Show more
Last updated: 4 hours ago • Promoted • New!
Cybersecurity Lead(6 months contract)
Sekuro Asia • Salem,Tamil Nadu, IN
Our client oversees and operates digital asset-related businesses.Our client aims to transform the financial industry by building a tech-enabled institutional grade ecosystem for issuance, distribu...Show more
Last updated: 9 days ago • Promoted
Salesforce Senior Tech Lead (Noida)
Connect Tech+Talent • Salem,Tamil Nadu, IN
Remote / Noida (as applicable).Shift Timing : 10 : 00 PM to 7 : 00 AM IST (US Time Zone Coverage).We are looking for a highly skilled Salesforce Senior Technical Lead to lead a team of 3–5 Salesforce de...Show more
Last updated: 10 days ago • Promoted
Technical Product Manager
Pearson • Salem, Tamil Nadu, India
This job is with Pearson, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community.
Please do not contact the recruiter directly.Technical Produc...Show more
Last updated: 1 day ago • Promoted
Webflow Developer (Finsweet Client-First + CMS-Driven Build)
RB Law • Salem,Tamil Nadu, IN
We need a Webflow developer who can.Webflow using best practices, correct naming conventions, and scalable CMS structures.
The goal is to ensure the marketing team can easily maintain and expand the...Show more
Last updated: 4 hours ago • Promoted • New!
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • Salem,Tamil Nadu, IN
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 14 hours ago • Promoted • New!
Lead Engineer
Hyqoo • Salem,Tamil Nadu, IN
Design, deploy, and manage AWS cloud infrastructure, including EC2 instances, S3 buckets, VPCs, RDS databases, and Lambda functions.
Assist in the design, implementation, and maintenance of backup, ...Show more
Last updated: 25 days ago • Promoted
Technical Lead
Mphasis • Salem,Tamil Nadu, IN
Looking for Senior Ingenium Developer with 10+ years' experience and following skills.Experience in Mainframe O / S and Development using COBOL programming language & JCL.
Experience in development an...Show more
Last updated: 15 days ago • Promoted
Salesforce Tech Lead- Contract
Gravity Infosolutions, Inc. • Salem, Tamil Nadu, India
Role : Salesforce Tech Lead Experience : 5+ Years Contract Duration : 6 months Location : Remote Shift : 11am- 8pm IST Participate in refining and scoping upcoming sprint work.Assist solution arch...Show more
Last updated: 5 hours ago • Promoted • New!
Site Reliability Engineer
Capgemini • Salem,Tamil Nadu, IN
Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues...Show more
Last updated: 30+ days ago • Promoted
Tech Lead (Full Stack | React + Python)
Aumne AI • Salem,Tamil Nadu, IN
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.Lead UI architecture ...Show more
Last updated: 14 hours ago • Promoted • New!
Technical Operations Lead
ClearTrail Technologies • Salem, Tamil Nadu, India
Job Title : Technical Operations Lead Location : Indore, MP (work-from-office) Experience : 8–12 Years Shift : 24x7 Rotational Shifts Qualification : B.
Computer Science, Information Technology, or a...Show more
Last updated: 30+ days ago • Promoted
Technical Lead
RapidBrains • Salem,Tamil Nadu, IN
We are looking for an experienced Technical Lead who can architect scalable systems, mentor development teams, and guide complex projects from concept to deployment.
You’ll partner closely with Prod...Show more
Last updated: 3 days ago • Promoted
QA Team Leader
Huxley • Salem,Tamil Nadu, IN
QA Team Leader – E-Commerce Platform.Join a dynamic development team responsible for building and maintaining large-scale online commerce services.
This role focuses on a popular cashback and reward...Show more
Last updated: 9 days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • salem, India
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 1 hour ago • Promoted • New!
E-commerce Technical Project Manager( Bigcommerce / Shopify)
Upbott Consulting, Inc • Salem, Tamil Nadu, India
Job Description Location : Remote Job Type : Contract About the Role : We are seeking an E-commerce Technical Project Manager with mandatory hands-on experience in managing BigCommerce or Shopify ...Show more
Last updated: 5 days ago • Promoted
Senior Data Architect- Snowflake
USEReady • Salem,Tamil Nadu, IN
USEReady is a data and analytics firm that provides the strategies, tools, capability, and capacity that businesses need to turn their data into a competitive advantage.
USEReady partners with cloud...Show more
Last updated: 14 days ago • Promoted
Quality Assurance Lead (Ruby on Rails)
Hireologist • Salem, Tamil Nadu, India
Job Title - QA Lead Location - Remote Experience - 7-10 Years About the Role You’ll lead automation, performance, and data integrity initiatives — ensuring our product can handle thousands of c...Show more
Last updated: 30+ days ago • Promoted