Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)AIMLEAP • srikakulam, andhra pradesh, in
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • srikakulam, andhra pradesh, in
2 hours ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • srikakulam, andhra pradesh, in
Related jobs
The ideal candidate will have a strong foundation in.Job Title : Azure Data Engineer.Work Timings : 5 : 00 PM to 2 : 00 AM IST.
If your expertise is primarily in.Lead the migration of large-scale SQL work...Show more
Data Analysts and Data Scientists.In this role, you will work with datasets, apply analytical methods, and provide insights that improve AI performance.
Identify and source datasets relevant to AI t...Show more
Lead Data Engineer
Guidanz Inc • srikakulam, andhra pradesh, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 22 hours ago • Promoted • New!
Master of AI tools
Vaomi • srikakulam, andhra pradesh, in
Vaomi is Hiring : 𝗠𝗮𝘀𝘁𝗲𝗿 𝗼𝗳 𝗔𝗜 𝘁𝗼𝗼𝗹𝘀.Vaomi AI is looking for a resourceful, lightning-fast Master of AI Tools to join the team.
In this role, you will become the operational backbone o...Show more
Last updated: 22 hours ago • Promoted • New!
Data Engineer
ShimentoX Technologies • srikakulam, andhra pradesh, in
Data Engineer (Strong with Building data connectors).Key Skills : Python, Data Connectors, Metadata, API Integration-Rest / GraphQL.
Must have proven background in building data connectors.Experience s...Show more
Last updated: 22 hours ago • Promoted • New!
Full Stack Engineer
Programmers.io • srikakulam, andhra pradesh, in
We are seeking highly skilled Senior.Laravel and modern frontend frameworks (Vue.The candidate should have deep technical expertise, leadership ability, and experience architecting scalable web sol...Show more
Last updated: 19 days ago • Promoted
BigCommerce Developer
Upbott Consulting, Inc • srikakulam, andhra pradesh, in
Job Title : BigCommerce Developer.We are seeking a BigCommerce Developer to join our dynamic team.The ideal candidate will have extensive experience in BigCommerce development, including customizing...Show more
Last updated: 30+ days ago • Promoted
AI Data Engineer
Turing • srikakulam, andhra pradesh, in
We’re looking for experienced AI data engineers skilled in Python to collaborate with one of the world’s top Large Language Model (LLM) companies.
Your work will directly help improve how AI models ...Show more
Last updated: 22 hours ago • Promoted • New!
Senior Software Engineer
Programmers.io • srikakulam, andhra pradesh, in
Senior AI-Integrated Software Engineer (.Remote until office reopens, Work from Home.We are looking for a dynamic and innovative.
The ideal candidate will bring hands-on experience in AI-assisted de...Show more
Last updated: 30+ days ago • Promoted
Azure AI Foundry Developer
Undocked • srikakulam, andhra pradesh, in
At Undocked, we help companies excel in e-commerce by delivering bespoke optimizations and cutting-edge analytics.Our experiences in retail and supply chain product strategy, technology, and operat...Show more
Last updated: 30+ days ago • Promoted
Freelance Data Quality Engineer
Leading MNC • srikakulam, andhra pradesh, in
Freelance Data Quality Engineer.The candidate should have a minimum of 8+ yrs.If you're looking for freelance / part time opportunity (along with your day job) & a chance to work with the top 0.You ...Show more
Last updated: 29 days ago • Promoted
Data Engineer
Sicame GBS • srikakulam, andhra pradesh, in
At Sicame Global Business Support (GBS), we are the dynamic support platform of the Sicame Group, operating across 5 continents and supporting over 50 companies in 26 countries.From business analys...Show more
Last updated: 2 hours ago • Promoted • New!
Sr Azure Data Engineer - Remote work
techolution • srikakulam, andhra pradesh, in
Remote
Last updated: 30+ days ago • Promoted
Data Analyst - Remote
Jobs Ai • srikakulam, andhra pradesh, in
Remote
Last updated: 2 hours ago • Promoted • New!
Lead of AI
TIGI HR • srikakulam, andhra pradesh, in
This position is central to shaping the AI strategy and data capabilities of a rapidly growing technology startup focused on transforming how manufacturers understand and address complex Scope 3 su...Show more
Last updated: 22 hours ago • Promoted • New!
Technical Lead – Web Crawling Systems, Data Pipelines
AIMLEAP • Srikakulam, Andhra Pradesh, India
Experience : 7 to 12 Years Location : Remote / Bangalore Engagement : Full-time Positions : 2 Qualification : B.Tech / MCA / Computer Science / IT Industry : IT / Data / AI / E-commerce / FinTech / ...Show more
Last updated: 19 hours ago • Promoted • New!
Azure DevOps Data Engineer
Paritas Recruitment • srikakulam, andhra pradesh, in
Azure DevOps (Data) Engineer - 6+ Month Rolling Contract.Remote - (Sunday to Thursday working days).Paritas is working with a global IT Consultancy & leading Energy client who are seeking a skilled...Show more
Last updated: 22 hours ago • Promoted • New!
Azure Data Engineer
Paritas Recruitment • srikakulam, andhra pradesh, in
Azure Data Engineer - 6 Month+ Rolling Contract.Remote - (Sunday to Thursday working days).Paritas is working with a global IT Consultancy & leading Energy client who are seeking an experienced and...Show more
Last updated: 22 hours ago • Promoted • New!
Sr. Azure Data Architect & Presales Solution
Programmers.io • srikakulam, andhra pradesh, in
Job Title : Azure Data Architect.Location : Hyderabad, Pune, Jaipur.Experience required : 12+ years.We are seeking a highly experienced.
The ideal candidate should bring strong expertise in SQL, ETL / EL...Show more
Last updated: 23 days ago • Promoted
Senior Full Stack Engineer
Black Piano • srikakulam, andhra pradesh, in
Lead development of software applications across client portfolio with a focus on MERN or MEAN frameworks within an Azure, AWS or GCP environment.
Lead the continuous development of bespoke web appl...Show more
Last updated: 30+ days ago • Promoted