Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)AIMLEAP • Erode, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Erode, Republic Of India, IN
1 day ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • Erode, Republic Of India, IN
Related jobs
For an international project in Chennai, we are urgently looking for a Full Remote Senior Azure Data Engineer, who will build data pipeline for enterprise search applications using ADF and Databric...Show more
)
)
These fully remote, long-term freelance roles are ideal for engineers who can build scalable data pipelines, work with modern cloud-native data stacks, and support large-scale enterprise data initi...Show more
Engineering Manager - II, Data Engineering Platform
Tamara • Erode, IN
Data Engineering Manager II : Real-Time Data & Experimentation Platform (Remote, India).Saudi Arabia's first fintech unicorn.
GCC, with a mission to empower dreams through customer-centric financial ...Show more
Last updated: 26 days ago • Promoted
Data Engineer
TerraGiG • Erode, IN
Lead the design, development, and implementation of data solutions using AWS and Snowflake.Collaborate with cross-functional teams to understand business requirements and translate them into techni...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • Erode, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 1 day ago • Promoted
Engineering Manager
Branch International • Erode, IN
Branch delivers world-class financial services to the mobile generation.With offices in the United States, Nigeria, Kenya, and India, Branch is a for-profit socially conscious company that uses the...Show more
Last updated: 30+ days ago • Promoted
Engineering Manager
AiPrise • Erode, IN
The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics.Your role will involve strategizing, project management, part staff managemen...Show more
Last updated: 30+ days ago • Promoted
Business Insights Manager - AI & Data Science (Black Belt)
Three Across • Erode, IN
Role Overview : Business Insight Manager – AI & Data Science (Black Belt • • • • •).Remote / Indore / Mumbai / Chennai / Gurugram.
Days (45 Days for Notice Serving).Must be from a BPO / KPO / Shared Services or...Show more
Last updated: 8 days ago • Promoted
(Senior) Azure Data Engineer - Full remote - contractor in USD
All European Careers • Erode, IN
Remote
Last updated: 12 days ago • Promoted
Project Manager – Data Engineering & Analytics
Brillio • Erode, IN
We are looking for a skilled Technical Project Manager to lead and deliver projects in data engineering and analytics.You will manage cross-functional teams to execute data platform, pipeline, and ...Show more
Last updated: 30+ days ago • Promoted
Associate Director, Data Architecture (Snowflake • Databricks • AWS • CPG / FMCG • Enterprise Data)
Sky Systems, Inc. (SkySys) • Erode, IN
Associate Director, Data Architecture.Full-Time Contract (40hrs / week).Months+ (with a possibility of Contract-to-Hire).
Marketing, Product, Finance, Supply Chain, and global business teams.This role...Show more
Last updated: 11 days ago • Promoted
Full Stack AI engineer
AnswerThis (YC F25) • Erode, IN
Remote (Applications open worldwide).Semantic Search, Vector Databases, Prompt Engineering, GenAI Frameworks, React Agents, Graph Agents, Document Parsing, Python, Scalable APIs.AnswerThis is an AI...Show more
Last updated: 30+ days ago • Promoted
Data Engineer - Web Scraping
Alternative Path • Erode, IN
Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm.In this role, you will collaborate with individuals across various company de...Show more
Last updated: 30+ days ago • Promoted
Databricks Data Engineer Lead – Sustainability Project
Blue Cloud Softech Solutions Limited • Erode, IN
BCSS is seeking a Databricks Data Engineer to support its enterprise-wide Sustainability initiative.The engineer will be responsible for building data pipelines and models to support product-level ...Show more
Last updated: 12 days ago • Promoted
Data Engineer - Fully Remote (Global Data Platform & Analytics Projects)
SkillsCapital • Erode, IN
Remote
Last updated: 1 day ago • Promoted
Data Engineer
Mastek • Erode, IN
Deep hands-on experience with Unity Catalog — creating and managing catalogs, schemas, and tables.Experience automating data onboarding and metadata registration via Unity Catalog APIs or Databrick...Show more
Last updated: 20 days ago • Promoted
AI Implementation Manager
Sutra.AI • Erode, IN
Role : Senior Implementation Manager.As we expand our delivery footprint, we’re seeking a.Sutra’s implementation workflows into a.
The Implementation Leader will own the.AI solution deployments - fro...Show more
Last updated: 7 days ago • Promoted
AWS Data Architect
ACL Digital • Erode, IN
AWS (S3, Redshift, Glue, Lake Formation, IAM).Proficient in data modeling, performance tuning, and security best practices.
.AWS Certified Solutions Architect preferred.Show more
Last updated: 17 days ago • Promoted
AI Lead Engineer
TekGenio • Erode, IN
Experience : 5+ Years | Type : Full-Time | Location : WFH.Minimum of 5+ years of experience in AI / ML engineering, data science, or algorithm development.
Strong experience in machine learning, deep lea...Show more
Last updated: 5 days ago • Promoted
Data Engineer
Aceolution • Erode, IN
Data Engineer – Python Expert(Freelance Role).We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) developmen...Show more
Last updated: 30+ days ago • Promoted