Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • vizianagaram, andhra pradesh, in
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • vizianagaram, andhra pradesh, in
1 hour ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • vizianagaram, andhra pradesh, in
Related jobs
The ideal candidate will have a strong foundation in.Job Title : Azure Data Engineer.Work Timings : 5 : 00 PM to 2 : 00 AM IST.
If your expertise is primarily in.Lead the migration of large-scale SQL work...Show more
Technical Specialist
Confidential • vizianagaram, andhra pradesh, in
Do you love being a powerful positive force in the success of others? Are you a Team player who effectively builds relationships with cross-functional team members? If so, we might have the role fo...Show more
Last updated: 1 hour ago • Promoted • New!
Sr Azure Data Engineer - Remote work
Techolution • vizianagaram, andhra pradesh, in
Remote
Last updated: 30+ days ago • Promoted
Global coupa Technical / functional Lead
APPIT Software Inc • vizianagaram, andhra pradesh, in
Job Title : Global COUPA Technical / Functional Lead.Mandatory Skills : • Coupa, configuration, Procurement, integration testing, sap, solution design, Ariba, Python, Java, Spark, Kafka, SQL, AWS.Desira...Show more
Last updated: 1 hour ago • Promoted • New!
Lead Data Engineer
Guidanz Inc • vizianagaram, andhra pradesh, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Implementation Specialist (PowerScale / Isilon)
Norwin Technologies • vizianagaram, andhra pradesh, in
Senior Implementation Specialist.Interested candidates can share profile on anjalihb@norwintechnologies.Administer and optimize PowerScale clusters across Multi-potocols (NFS / SMB / HDFS / S3) environme...Show more
Last updated: 1 hour ago • Promoted • New!
Full Stack Developer
Turing • vizianagaram, andhra pradesh, in
Turing is seeking experienced Full Stack Developers to help build end-to-end AI-driven applications for US customers — spanning backend services, web frontends, and evaluation tooling.In this role,...Show more
Last updated: 1 hour ago • Promoted • New!
Azure AI Foundry Developer
Undocked • vizianagaram, andhra pradesh, in
At Undocked, we help companies excel in e-commerce by delivering bespoke optimizations and cutting-edge analytics.Our experiences in retail and supply chain product strategy, technology, and operat...Show more
Last updated: 30+ days ago • Promoted
Senior Full Stack Engineer
OWOW • vizianagaram, andhra pradesh, in
Front-end technologies like ReactJS, Redux, TypeScript, Tailwind CSS.Exposure to other cloud platforms beyond AWS.Experience with microservices or event-driven architectures.Familiarity with AWS La...Show more
Last updated: 25 days ago • Promoted
EDI Leader
Sol-Millennium Medical Group • vizianagaram, andhra pradesh, in
The EDI Implementer is a crucial role within our organization, aimed at growing and maintaining our eSolutions function and EDI capabilities.
This position involves a blend of technical proficiency,...Show more
Last updated: 30+ days ago • Promoted
Need extra earnings? Drive your bike with the Uber app
Uber • Denkada, IN, IN
Driving with the Uber app is a flexible way to earn using your own bike.It gives you the freedom to choose when, where, and how often you drive — while earning on your own terms.Driving with the Ub...Show more
Last updated: 23 hours ago • Promoted
Full Stack Engineer
Programmers.io • vizianagaram, andhra pradesh, in
We are seeking highly skilled Senior.Laravel and modern frontend frameworks (Vue.The candidate should have deep technical expertise, leadership ability, and experience architecting scalable web sol...Show more
Last updated: 18 days ago • Promoted
AWS Solution Architect
Saxon AI • vizianagaram, andhra pradesh, in
We are looking for a highly skilled.Architecture & System Modernization.Own the redesign of the Truscan backend architecture : .
Multi-stage compute pipeline (Stage 1–4).High-performance Python worker...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Tech Lead CRM Developer with AI Builder Experience
GTRTeK • vizianagaram, andhra pradesh, in
Microsoft Dynamics CRM 365 Senior Developer with minimum 5 years of experience in D 365 CRM along with .Looking for competent candidate in the relevant module.
Minimum 3 years of work experience .Bu...Show more
Last updated: 1 hour ago • Promoted • New!
D365 F&O Technical Consultant (7+ Years Experience)
Proso.ai • vizianagaram, andhra pradesh, in
Immediate joiners or candidates who can join within 30 days.Proso AI is looking for a highly skilled.Develop, customize, and enhance solutions in.
Design and implement integrations between D365 F&O ...Show more
Last updated: 1 hour ago • Promoted • New!
Technical Analyst – Mendix
Yoda Tech • vizianagaram, andhra pradesh, in
Singapore, focuses on dividing aspects of digitalization into small logical Micro initiatives using ready-to-use Micro-bots, instead of large transformational initiatives.This approach aims to redu...Show more
Last updated: 1 hour ago • Promoted • New!
Full Stack Tech Lead- Contract
Gravity Infosolutions, Inc. • vizianagaram, India
Job description for Tech Lead Full Stack : .Conducts code review, contributes writing code, proficient in Java + at least one language between Typescript and Python.
Basic understanding of Infrastruct...Show more
Last updated: 2 days ago • Promoted
Technical Lead (Dotnet)
Closeloop Technologies • vizianagaram, andhra pradesh, in
Experience Required : 12 to 18 years.We are seeking a highly experienced Technical Lead (.NET) with 12 - 18 years of hands-on development and leadership experience to drive end-to-end technical deli...Show more
Last updated: 1 hour ago • Promoted • New!
Snowflake Developer
Yoda Tech • vizianagaram, andhra pradesh, in
Singapore-based company that focuses on dividing digitalization into small logical Micro initiatives with ready-to-use Micro-bots.
The company aims to reduce IT operations spend by emphasizing Autom...Show more
Last updated: 1 hour ago • Promoted • New!