Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • Anantapur, IN
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • Anantapur, IN
10 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • Anantapur, IN
Related jobs
Web Crawling Engineer
Forage AI • Anantapur, IN
The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.
Forage AI is a pioneering A...Show more
Last updated: 22 days ago • Promoted
Global coupa Technical / functional Lead
APPIT Software Inc • Anantapur, IN
Job Title : Global COUPA Technical / Functional Lead.Mandatory Skills : • Coupa, configuration, Procurement, integration testing, sap, solution design, Ariba, Python, Java, Spark, Kafka, SQL, AWS.Desira...Show more
Last updated: 10 hours ago • Promoted • New!
AWS Data Architect
ACL Digital • Anantapur, IN
AWS (S3, Redshift, Glue, Lake Formation, IAM).Proficient in data modeling, performance tuning, and security best practices.
.AWS Certified Solutions Architect preferred.Show more
Last updated: 20 days ago • Promoted
Lead Data Engineer
Guidanz Inc • Anantapur, IN
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 10 hours ago • Promoted • New!
Salesforce Senior Tech Lead (Noida)
Connect Tech+Talent • Anantapur, IN
Remote / Noida (as applicable).Shift Timing : 10 : 00 PM to 7 : 00 AM IST (US Time Zone Coverage).We are looking for a highly skilled Salesforce Senior Technical Lead to lead a team of 3–5 Salesforce de...Show more
Last updated: 9 days ago • Promoted
APAC AWS Alliance Lead
SoftwareOne • Anantapur, IN
SoftwareOne focuses on developing and strengthening strategic alliances with AWS and other relevant ISV partners.This position requires establishing and maintaining strong relationships with key st...Show more
Last updated: 26 days ago • Promoted
Team Lead
ALTISOURCE BUSINESS SOLUTIONS PRIVATE LIMITED • Anantapur, IN
Willing to work in night shift.Lead the property inspection operations in a multi-client environment ensuring adherence to service level agreements and quality standards.
Track team perfoJob Descrip...Show more
Last updated: 20 days ago • Promoted
Technical Operations Lead
ClearTrail Technologies • Anantapur, IN
Computer Science, Information Technology, or a related field.We are seeking a highly skilled and experienced.The ideal candidate will have a strong background in Linux system administration, incide...Show more
Last updated: 30+ days ago • Promoted
Data Engineer - Web Scraping
Alternative Path • Anantapur, IN
Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm.In this role, you will collaborate with individuals across various company de...Show more
Last updated: 30+ days ago • Promoted
Lead Expert - Information Systems (SAP PP / QM) Business
Suzlon Group • Anantapur, IN
Seeking an experienced S / 4HANA PP / QM Consultant with 5-6 years of hands-on experience in SAP Production Planning (PP) and Quality Management (QM) modules within the S / 4HANA environment.The ideal ca...Show more
Last updated: 17 days ago • Promoted
Technical Lead
Mphasis • Anantapur, IN
Looking for Senior Ingenium Developer with 10+ years' experience and following skills.Experience in Mainframe O / S and Development using COBOL programming language & JCL.
Experience in development an...Show more
Last updated: 14 days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • Anantapur, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 4 days ago • Promoted
Lead Engineer
Hyqoo • Anantapur, IN
Design, deploy, and manage AWS cloud infrastructure, including EC2 instances, S3 buckets, VPCs, RDS databases, and Lambda functions.
Assist in the design, implementation, and maintenance of backup, ...Show more
Last updated: 24 days ago • Promoted
Team Lead
Zensar Technologies • Anantapur, IN
ZENSAR -TEAM LEAD | PROJECT MANAGER OPPORTUNITY FOR GEN AI PROJECT.Dear Aspirant, Greetings from Zensar!!.We are a technology consulting and services company with over 11,500 associates in 33 globa...Show more
Last updated: 26 days ago • Promoted
E-commerce Technical Project Manager( Bigcommerce / Shopify)
Upbott Consulting, Inc • Anantapur, IN
E-commerce Technical Project Manager.BigCommerce or Shopify projects.Candidates must have led end-to-end e-commerce implementations specifically on.
This role requires someone who understands the Bi...Show more
Last updated: 5 days ago • Promoted
Senior / Staff Full‑Stack Engineer — CEO’s Build Partner (AI‑Augmented)
Truey • Anantapur, IN
Senior / Staff Full‑Stack Engineer — CEO’s Build Partner (AI‑Augmented) 🚀.C2C with your own LLC considered; NO staffing vendors — direct to Truey.
You’ll turn ambiguous ideas into working software : d...Show more
Last updated: 19 days ago • Promoted
Azure Databricks
Kumaran Systems • Anantapur, IN
We are seeking a skilled Azure Databricks with strong hands-on experience in Microsoft Azure services such as Data Factory, Storage, Synapse, and Key Vault.
The candidate should be proficient in Dat...Show more
Last updated: 15 days ago • Promoted
Site Reliability Engineer
Capgemini • Anantapur, IN
Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues...Show more
Last updated: 30+ days ago • Promoted