Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • secunderabad, telangana, in
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • secunderabad, telangana, in
14 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • secunderabad, telangana, in
Related jobs
Tech Lead (Full Stack | React + Python)
Aumne AI • Hyderabad, IN
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.Lead UI architecture ...Show more
Last updated: less than 1 hour ago • Promoted • New!
Tech Lead
CherryTechSolutions • Hyderabad, Telangana, India
We are looking for an experienced, skilled, enthusiastic, and energetic.He / she will analyze and improve upon technology standards across the organization to maintain a technological and competitive...Show more
Last updated: 30+ days ago • Promoted
Liferay Tech Lead
ValueMomentum • Hyderabad, Telangana, India
We are seeking a highly skilled and hands-on.The ideal candidate will be responsible for designing optimal solutions, leading development efforts, and providing technical guidance to the team.Stron...Show more
Last updated: 6 days ago • Promoted
Lead Engineer
Hyqoo • Hyderabad, IN
Design, deploy, and manage AWS cloud infrastructure, including EC2 instances, S3 buckets, VPCs, RDS databases, and Lambda functions.
Assist in the design, implementation, and maintenance of backup, ...Show more
Last updated: 24 days ago • Promoted
Web Analytics Specialist
vidaXL • Hyderabad, Telangana, India
A web analyst is responsible for collecting and analyzing data on user interaction with the platform and on the overall operation of the platform itself.
You evaluate the conformity of all products ...Show more
Last updated: 30+ days ago • Promoted
Technical Lead
Narwal • Hyderabad, Telangana, India
Narwal, with its Global Delivery Model, strategically expands its reach across North America, the United Kingdom, and an offshore development centre in India.
Delivery cutting edge AI, Data and Qual...Show more
Last updated: 22 days ago • Promoted
Technical Lead
Mphasis • Hyderabad, IN
Looking for Senior Ingenium Developer with 10+ years' experience and following skills.Experience in Mainframe O / S and Development using COBOL programming language & JCL.
Experience in development an...Show more
Last updated: 14 days ago • Promoted
QA Team Leader
Huxley • Hyderabad, IN
QA Team Leader – E-Commerce Platform.Join a dynamic development team responsible for building and maintaining large-scale online commerce services.
This role focuses on a popular cashback and reward...Show more
Last updated: 8 days ago • Promoted
Cybersecurity Lead(6 months contract)
Sekuro Asia • Hyderabad, IN
Our client oversees and operates digital asset-related businesses.Our client aims to transform the financial industry by building a tech-enabled institutional grade ecosystem for issuance, distribu...Show more
Last updated: 8 days ago • Promoted
Technical Lead
BeeHyv Software • Hyderabad, Telangana, India
Full-time, Can join immediately.BeeHyv is a premium software engineering partner for firms building innovative world class products and solutions to make an impact.
Whether it be a hot Bay Area tech...Show more
Last updated: 22 days ago • Promoted
E-commerce Technical Project Manager( Bigcommerce / Shopify)
Upbott Consulting, Inc • Hyderabad, IN
E-commerce Technical Project Manager.BigCommerce or Shopify projects.Candidates must have led end-to-end e-commerce implementations specifically on.
This role requires someone who understands the Bi...Show more
Last updated: 5 days ago • Promoted
Technical Lead
AutoRABIT • Hyderabad, Telangana, India
AutoRABIT is the leader in DevSecOps for SaaS platforms such as Salesforce.Its unique metadata-aware capability makes Release Management, Version Control, and Backup & Recovery complete, reliable, ...Show more
Last updated: 29 days ago • Promoted
Liferay 7.4
ValueMomentum • Hyderabad, Telangana, India
Hyderabad Preferred ( Open to Remote also).We are seeking a highly skilled and hands-on.The ideal candidate will be responsible for designing optimal solutions, leading development efforts, and pro...Show more
Last updated: 16 days ago • Promoted
Site Reliability Engineer
Capgemini • Hyderabad, IN
Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues...Show more
Last updated: 30+ days ago • Promoted
AWS Technical Lead
Tata Consultancy Services • Hyderabad, Telangana, India
AWS (Quick Sight, Step Functions, Athena, S3, Glue), SQL, Documentation, Reporting, MS Office.Hands-on experience with Amazon Quick Sight, Step Functions, Athena, and S3.
Strong SQL skills for data ...Show more
Last updated: 16 days ago • Promoted
Technical Lead
vidaXL • Hyderabad, Telangana, India
Job Title : Tech Lead Salesforce (E-Commerce).Work Mode : Semi Hybrid Mode (4days work from office) and (1 day work from home(either Monday or Wednesday or Friday)) in a week.We are looking for an ex...Show more
Last updated: 30+ days ago • Promoted
Content Team Lead
NxtWave • Hyderabad, Telangana, India
As a Content Lead at NxtWave, you will shape the strategic vision and operational excellence behind our technical curriculum, including core areas like MERN Full Stack, Python, and AI.You will lead...Show more
Last updated: 9 days ago • Promoted
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • Hyderabad, IN
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: less than 1 hour ago • Promoted • New!