Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)AIMLEAP • Thoothukudi, Republic Of India, IN
No longer accepting applications
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Thoothukudi, Republic Of India, IN
4 days ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • Thoothukudi, Republic Of India, IN
Related jobs
)
Data Solutions Director
Amethyst Partners • Thoothukudi, IN
This role carries both strategic and operational leadership responsibilities, managing Principal Analysts across APAC and working closely with data engineering, development, and global teams.You wi...Show more
Last updated: 14 days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • Thoothukudi, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 14 hours ago • Promoted • New!
Full Stack AI engineer
AnswerThis (YC F25) • Thoothukudi, IN
Remote (Applications open worldwide).Semantic Search, Vector Databases, Prompt Engineering, GenAI Frameworks, React Agents, Graph Agents, Document Parsing, Python, Scalable APIs.AnswerThis is an AI...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 To 7yrs)
AIMLEAP • Thoothukudi, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT.IT / Data / AI / E-commerce / FinTech / Healthcare.
Experience working with cloud platforms such as...Show more
Last updated: 6 hours ago • Promoted • New!
E-commerce Technical Project Manager( Bigcommerce / Shopify)
Upbott Consulting, Inc • Thoothukudi, IN
E-commerce Technical Project Manager.BigCommerce or Shopify projects.Candidates must have led end-to-end e-commerce implementations specifically on.
This role requires someone who understands the Bi...Show more
Last updated: 6 days ago • Promoted
Data Engineer
TerraGiG • Thoothukudi, IN
Lead the design, development, and implementation of data solutions using AWS and Snowflake.Collaborate with cross-functional teams to understand business requirements and translate them into techni...Show more
Last updated: 30+ days ago • Promoted
Sr Manager Analytics
Live Connections • Thoothukudi, IN
Required Notice Period - Immediate Joiners or Serving Notice Period.Should have a technical background.Should be working on production projects.
Required Skills and Qualifications.Proven experience ...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Mastek • Thoothukudi, IN
Deep hands-on experience with Unity Catalog — creating and managing catalogs, schemas, and tables.Experience automating data onboarding and metadata registration via Unity Catalog APIs or Databrick...Show more
Last updated: 24 days ago • Promoted
Lead Data Engineer
VXI Global Solutions • Thoothukudi, IN
We are seeking talented and motivated Data Engineers to join our dynamic team and contribute to our mission of harnessing the power of data to drive growth and success.
As a Data Engineer at VXI Glo...Show more
Last updated: 30+ days ago • Promoted
Lead Data Engineer
Ironbook AI • Thoothukudi, IN
We are seeking an experienced and driven Lead Data Engineer to spearhead the.AI use cases across the organization.Minimum 7 years of experience in data engineering, with at.Strong hands-on experien...Show more
Last updated: 5 hours ago • Promoted • New!
Lead Data Engineer
Confidential • Thoothukudi, IN
Expertise in big data technologies such as Apache Spark and real-time streaming technologies like Apache Kafka.Strong programming skills in Python, Java, C++, SQL etc.
Advanced knowledge of a major ...Show more
Last updated: 5 hours ago • Promoted • New!
Senior Data Architect- Snowflake
USEReady • Thoothukudi, IN
USEReady is a data and analytics firm that provides the strategies, tools, capability, and capacity that businesses need to turn their data into a competitive advantage.
USEReady partners with cloud...Show more
Last updated: 14 days ago • Promoted
Senior / Staff Full‑Stack Engineer — CEO’s Build Partner (AI‑Augmented)
Truey • Thoothukudi, IN
Senior / Staff Full‑Stack Engineer — CEO’s Build Partner (AI‑Augmented) 🚀.C2C with your own LLC considered; NO staffing vendors — direct to Truey.
You’ll turn ambiguous ideas into working software : d...Show more
Last updated: 20 days ago • Promoted
Lead Data Engineer
Guidanz Inc • Thoothukudi, IN
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineer
Ironbook AI • Thoothukudi, IN
The ideal candidate will have strong experience with cloud platforms, modern ETL / ELT tools, and deep technical skills in Python, SQL, and distributed data frameworks.
Design, develop, and maintain s...Show more
Last updated: 5 hours ago • Promoted • New!
Senior Data Engineer - Data Acquisition
InfoBeans • Thoothukudi, IN
We are seeking a highly skilled.Senior Data Engineer – Data Acquisition (ODS).The ideal candidate will have extensive hands-on experience in building and optimizing data ingestion and transformatio...Show more
Last updated: 23 days ago • Promoted
Engineering Manager
Cargoz.com • Thoothukudi, IN
This role is perfect for leaders who excel in dynamic, high-velocity environments, enjoy developing both people and systems, and want to help shape our product and engineering culture from the grou...Show more
Last updated: 14 hours ago • Promoted • New!
Engineering Manager
Confidential • Thoothukudi, IN
The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics.Your role will involve strategizing, project management, part staff managemen...Show more
Last updated: 3 days ago • Promoted