Technical Lead – Web Crawling Systems, Data PipelinesAIMLEAP • Ludhiāna, Republic Of India, IN
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • Ludhiāna, Republic Of India, IN
14 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Technical Lead • Ludhiāna, Republic Of India, IN
Related jobs
Tech Lead (Full Stack | React + Python)
Aumne AI • ludhiana, punjab, in
Aumne AI is building next-gen AI systems for customer experience.We’re looking for a frontend-strong Tech Lead who values clean design, simple architecture, and fast execution.Lead UI architecture ...Show more
Last updated: 3 hours ago • Promoted • New!
Salesforce Tech Lead- Contract
Gravity Infosolutions, Inc. • ludhiana, punjab, in
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 7 hours ago • Promoted • New!
Lead Data Engineer
Guidanz Inc • ludhiana, punjab, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 day ago • Promoted
Senior Technical Consultant
MyData Insights Pvt Ltd • ludhiana, punjab, in
We are seeking an experienced Senior Technical Consultant to join our Dynamics 365 & Power.You will play a key role in designing, extending, and delivering modern business applications built on Mic...Show more
Last updated: 3 hours ago • Promoted • New!
Technical Lead (Dotnet)
Closeloop Technologies • ludhiana, punjab, in
Experience Required : 12 to 18 years.We are seeking a highly experienced Technical Lead (.NET) with 12 - 18 years of hands-on development and leadership experience to drive end-to-end technical deli...Show more
Last updated: 1 day ago • Promoted
EDI Leader
Sol-Millennium Medical Group • ludhiana, punjab, in
The EDI Implementer is a crucial role within our organization, aimed at growing and maintaining our eSolutions function and EDI capabilities.
This position involves a blend of technical proficiency,...Show more
Last updated: 30+ days ago • Promoted
Azure Developer
XFactor Talent • ludhiana, punjab, in
APIs, Python, Azure Functions, Azure Data Services, and Jitterbit Integration Platform.The ideal candidate should have solid knowledge of cloud environments, integration tools, data systems, and AI...Show more
Last updated: 3 hours ago • Promoted • New!
Senior Implementation Specialist (PowerScale / Isilon)
Norwin Technologies • ludhiana, punjab, in
Senior Implementation Specialist.Interested candidates can share profile on anjalihb@norwintechnologies.Administer and optimize PowerScale clusters across Multi-potocols (NFS / SMB / HDFS / S3) environme...Show more
Last updated: 1 day ago • Promoted
Senior Full Stack Developer (DocuSign CLM)
Gravity Infosolutions, Inc. • ludhiana, punjab, in
Job Title : Senior Full Stack Developer (DocuSign CLM).We are looking for a Senior Full Stack Developer with strong.DocuSign CLM workflows, templates, and document automation.Develop full-stack solu...Show more
Last updated: 3 hours ago • Promoted • New!
Web Developer
Best NanoTech • ludhiana, punjab, in
The ideal candidate is a creative problem solver who will work in coordination with cross-functional teams to design, develop, and maintain our next generation websites and web tools.You must be co...Show more
Last updated: 7 hours ago • Promoted • New!
Full-Stack PHP Developer (Laravel / OpenCart)
TMD Software Pvt. Ltd. • Ludhiana, Punjab, India
We are looking for a passionate.The ideal candidate should have strong knowledge of PHP, MySQL, OpenCart custom development, API integration, and Laravel-based applications.SaaS modules, custom ext...Show more
Last updated: 12 hours ago • Promoted • New!
Senior Full Stack Engineer
Black Piano • ludhiana, punjab, in
Lead development of software applications across client portfolio with a focus on MERN or MEAN frameworks within an Azure, AWS or GCP environment.
Lead the continuous development of bespoke web appl...Show more
Last updated: 30+ days ago • Promoted
Full Stack Engineer
Programmers.io • ludhiana, punjab, in
We are seeking highly skilled Senior.Laravel and modern frontend frameworks (Vue.The candidate should have deep technical expertise, leadership ability, and experience architecting scalable web sol...Show more
Last updated: 19 days ago • Promoted
Freelance Azure Data Engineer - Vectr & Cribl Expert
ThreatXIntel • ludhiana, punjab, in
ThreatXIntel is a startup cyber security company focused on empowering businesses and organizations to defend against evolving cyber threats.
Offering a comprehensive suite of services, including cl...Show more
Last updated: 3 hours ago • Promoted • New!
AWS Tech Lead - Contract
Gravity Infosolutions, Inc. • ludhiana, punjab, in
Participate in refining and scoping upcoming sprint work.Assist solution architects with technical design and breaking down complex tasks.
Accountable for timely delivery of assigned tickets, meetin...Show more
Last updated: 3 hours ago • Promoted • New!
Partnership Sales Lead / Supply Lead
MacroHire • Ludhiana, Punjab, India
We are hiring Partnership Sales Lead for one of our client who is Global leader in Co-Working Spaces.Are you ready to lead the charge and be at the forefront of our ambitious global expansion? Is h...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • ludhiana, punjab, in
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 7 hours ago • Promoted • New!
D365 F&O Technical Consultant (7+ Years Experience)
Proso.ai • ludhiana, punjab, in
Immediate joiners or candidates who can join within 30 days.Proso AI is looking for a highly skilled.Develop, customize, and enhance solutions in.
Design and implement integrations between D365 F&O ...Show more
Last updated: 1 day ago • Promoted