Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)AIMLEAP • raipur, chattisgarh, in
No longer accepting applications
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • raipur, chattisgarh, in
5 days ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • raipur, chattisgarh, in
Related jobs
Engineering Manager
Cargoz.com • raipur, chattisgarh, in
This role is perfect for leaders who excel in dynamic, high-velocity environments, enjoy developing both people and systems, and want to help shape our product and engineering culture from the grou...Show more
Last updated: 14 hours ago • Promoted • New!
Lead Data Engineer
Guidanz Inc • raipur, chattisgarh, in
BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show more
Last updated: 1 day ago • Promoted
Data Analyst
BHEL • Raipur, Chhattisgarh, India
We suggest you enter details here.This is a full-time, on-site Data Analyst role located in Raipur.The Data Analyst will be responsible for collecting, organizing, and analyzing data to support dec...Show more
Last updated: 5 days ago • Promoted
Data Engineer (Snowflake + Databricks)
MyRemoteTeam Inc • raipur, chattisgarh, in
MyRemoteTeam, Inc is a fast-growing distributed workforce enabler, helping companies scale with top global talent.We empower businesses by providing world-class software engineers, operations suppo...Show more
Last updated: 10 hours ago • Promoted • New!
Data Scientist
Enterprise Minds, Inc • raipur, chattisgarh, in
Hiring : Senior Data Scientist – Generative AI (3.Generative AI, LLM, and agentic systems.In this role, you will transform complex business problems into.
As a key individual contributor, you will : .L...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineer
Ironbook AI • raipur, chattisgarh, in
The ideal candidate will have strong experience with cloud platforms, modern ETL / ELT tools, and deep technical skills in Python, SQL, and distributed data frameworks.
Design, develop, and maintain s...Show more
Last updated: 10 hours ago • Promoted • New!
Sr Manager Analytics
Live Connections • raipur, chattisgarh, in
Required Notice Period - Immediate Joiners or Serving Notice Period.Should have a technical background.Should be working on production projects.
Required Skills and Qualifications.Proven experience ...Show more
Last updated: 30+ days ago • Promoted
Performance Analytics Manager
American Giant Global Careers • raipur, chattisgarh, in
A Luxury Oracle of Intelligence Narrative.There are roles that track performance—.It is the central oracle of the organization—.
Someone who reads patterns like a language,.This role demands a rare ...Show more
Last updated: 10 hours ago • Promoted • New!
Data Engineer
MyRemoteTeam Inc • raipur, chattisgarh, in
MyRemoteTeam, Inc is a fast-growing distributed workforce enabler, helping companies scale with top global talent.We empower businesses by providing world-class software engineers, operations suppo...Show more
Last updated: 10 hours ago • Promoted • New!
Engineering Director - Python / AI / ML
Aarav Solutions • raipur, chattisgarh, in
Candidates are encouraged to only apply through LinkedIn.IT consulting, committed to driving digital transformation for industries such as Telecom, Finance, Government, and Utilities.We specialize ...Show more
Last updated: 10 hours ago • Promoted • New!
Design Lead
Livspace • Raipur, Chhattisgarh, India
A Business Manager - Design will be responsible for managing the designing for 12 to 15 projects month-on-month through a team of 6 to 10 designers.
The output of which would be achieved through man...Show more
Last updated: 14 hours ago • Promoted • New!
Lead Data Engineer
Ironbook AI • raipur, chattisgarh, in
We are seeking an experienced and driven Lead Data Engineer to spearhead the.AI use cases across the organization.Minimum 7 years of experience in data engineering, with at.Strong hands-on experien...Show more
Last updated: 10 hours ago • Promoted • New!
Enterprise Application Developer (FP&A and Data Integration)
DRISHTICON Inc • raipur, chattisgarh, in
Job Title : Experienced Anaplan Model Builder (FP&A & Data Integration).The position is long term contract and Remote.Nice to have skills : FP&A knowledge or certification, Anaplan Data Integration u...Show more
Last updated: 10 hours ago • Promoted • New!
Data Scientist
People Prime Worldwide • raipur, chattisgarh, in
Job Description : Data Scientist.We are seeking a highly skilled.Java, Python, Generative AI, and Google Vertex AI.The ideal candidate will design, build, and deploy data-driven solutions leveraging...Show more
Last updated: 30+ days ago • Promoted
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 to 7yrs)
AIMLEAP • raipur, chattisgarh, in
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT .IT / Data / AI / E-commerce / FinTech / Healthcare .
Experience working with cloud platforms such ...Show more
Last updated: 18 hours ago • Promoted • New!
Lead Data Engineer
Confidential • raipur, chattisgarh, in
Expertise in big data technologies such as Apache Spark and real-time streaming technologies like Apache Kafka.Strong programming skills in Python, Java, C++, SQL etc.
Advanced knowledge of a major ...Show more
Last updated: 10 hours ago • Promoted • New!
Data Engineering Manager – Web Crawling & Pipeline Architecture (2 To 7yrs)
AIMLEAP • Raipur, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT.IT / Data / AI / E-commerce / FinTech / Healthcare.
Experience working with cloud platforms such as...Show more
Last updated: 15 hours ago • Promoted • New!
Data Architect (12+years)
MindBrain • raipur, chattisgarh, in
Define and execute the enterprise data architecture strategy in alignment with business goals and future growth needs.Architect, design, and implement Data Lakes, Data Warehouses, and Data Lakehous...Show more
Last updated: 10 hours ago • Promoted • New!