Data Engineering Manager – Web Crawling & Pipeline Architecture (2 To 7yrs)AIMLEAP • Baddi, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 To 7yrs)
AIMLEAP • Baddi, Republic Of India, IN
15 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • Baddi, Republic Of India, IN
Related jobs
Sr Manager Analytics
Live Connections • baddi, himachal pradesh, in
Required Notice Period - Immediate Joiners or Serving Notice Period.Should have a technical background.Should be working on production projects.
Required Skills and Qualifications.Proven experience ...Show more
Last updated: 30+ days ago • Promoted
Lead Data Engineer
Confidential • baddi, himachal pradesh, in
Expertise in big data technologies such as Apache Spark and real-time streaming technologies like Apache Kafka.Strong programming skills in Python, Java, C++, SQL etc.
Advanced knowledge of a major ...Show more
Last updated: 10 hours ago • Promoted • New!
Performance Analytics Manager
American Giant Global Careers • baddi, himachal pradesh, in
A Luxury Oracle of Intelligence Narrative.There are roles that track performance—.It is the central oracle of the organization—.
Someone who reads patterns like a language,.This role demands a rare ...Show more
Last updated: 10 hours ago • Promoted • New!
Data Architect (12+years)
MindBrain • baddi, himachal pradesh, in
Define and execute the enterprise data architecture strategy in alignment with business goals and future growth needs.Architect, design, and implement Data Lakes, Data Warehouses, and Data Lakehous...Show more
Last updated: 10 hours ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • baddi, himachal pradesh, in
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 18 hours ago • Promoted • New!
Energy Assessment Engineer
KISEM IIT Ropar • Rupnagar, Punjab, India
Applications are invited for the position of Energy Assessment Engineers for the Kotak-IITM Save Energy Mission (KISEM) at IIT Ropar.
The primary objective of this project is to conduct an energy as...Show more
Last updated: 14 hours ago • Promoted • New!
Engineering Manager
Cargoz.com • baddi, himachal pradesh, in
This role is perfect for leaders who excel in dynamic, high-velocity environments, enjoy developing both people and systems, and want to help shape our product and engineering culture from the grou...Show more
Last updated: 14 hours ago • Promoted • New!
Lead Data Engineer
Ironbook AI • baddi, himachal pradesh, in
We are seeking an experienced and driven Lead Data Engineer to spearhead the.AI use cases across the organization.Minimum 7 years of experience in data engineering, with at.Strong hands-on experien...Show more
Last updated: 10 hours ago • Promoted • New!
Engineering Director - Python / AI / ML
Aarav Solutions • baddi, himachal pradesh, in
Candidates are encouraged to only apply through LinkedIn.IT consulting, committed to driving digital transformation for industries such as Telecom, Finance, Government, and Utilities.We specialize ...Show more
Last updated: 10 hours ago • Promoted • New!
Data Engineer
MyRemoteTeam Inc • baddi, himachal pradesh, in
MyRemoteTeam, Inc is a fast-growing distributed workforce enabler, helping companies scale with top global talent.We empower businesses by providing world-class software engineers, operations suppo...Show more
Last updated: 10 hours ago • Promoted • New!
Contract- Snowflake Data Engineer with Data Vault (Snowpark or CortexAI)
KPG99 INC • baddi, himachal pradesh, in
Role Snowflake-focused Data Engineer.Location : Offshore Remote (India).Project involves implementing Data Vault 2.Work includes specific pipeline development in Snowflake with Streamlit.Not a tradi...Show more
Last updated: 14 hours ago • Promoted • New!
Senior Data Engineer
Ironbook AI • baddi, himachal pradesh, in
The ideal candidate will have strong experience with cloud platforms, modern ETL / ELT tools, and deep technical skills in Python, SQL, and distributed data frameworks.
Design, develop, and maintain s...Show more
Last updated: 10 hours ago • Promoted • New!
Lead Data Engineer
Guidanz Inc • baddi, himachal pradesh, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 day ago • Promoted
Enterprise Application Developer (FP&A and Data Integration)
DRISHTICON Inc • baddi, himachal pradesh, in
Job Title : Experienced Anaplan Model Builder (FP&A & Data Integration).The position is long term contract and Remote.Nice to have skills : FP&A knowledge or certification, Anaplan Data Integration u...Show more
Last updated: 10 hours ago • Promoted • New!
Data Scientist
People Prime Worldwide • baddi, himachal pradesh, in
Job Description : Data Scientist.We are seeking a highly skilled.Java, Python, Generative AI, and Google Vertex AI.The ideal candidate will design, build, and deploy data-driven solutions leveraging...Show more
Last updated: 30+ days ago • Promoted
Data Scientist
Enterprise Minds, Inc • baddi, himachal pradesh, in
Hiring : Senior Data Scientist – Generative AI (3.Generative AI, LLM, and agentic systems.In this role, you will transform complex business problems into.
As a key individual contributor, you will : .L...Show more
Last updated: 30+ days ago • Promoted
Data Engineer (Snowflake + Databricks)
MyRemoteTeam Inc • baddi, himachal pradesh, in
MyRemoteTeam, Inc is a fast-growing distributed workforce enabler, helping companies scale with top global talent.We empower businesses by providing world-class software engineers, operations suppo...Show more
Last updated: 10 hours ago • Promoted • New!
Technical Lead & delivery
Americana Restaurants • mohali district, punjab, in
Position Title – Technical Lead & delivery – Platform (Java / J2EE Architect).Java, Spring Boot, Python, AKS, Azure Cloud Native, Azure DevOps).
About Americana Restaurants International PLC.Americana...Show more
Last updated: 10 hours ago • Promoted • New!