Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)AIMLEAP • raipur, chattisgarh, in
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 to 2 yrs)
AIMLEAP • raipur, chattisgarh, in
2 days ago
Job descriptionLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS / GCP / Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems . Strong expertise in Python, SQL , and modern data processing practices. Experience working with Airflow, Celery , or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS / GCP / Azure). Experience with AI / LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.
Data Engineering Manager – Web Crawling & Pipeline Architecture
Experience : 7 to 12 Years
Location : Remote / Bangalore
Engagement : Full-time
Positions : 2
Qualification : B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry : IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period : Immediate
What We Are Looking For
- Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
- Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .
- Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
- Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
- Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR / CCPA-safe crawling).
- Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..
Responsibilities
Qualifications
Create a job alert for this search
Engineering Manager • raipur, chattisgarh, in
Related jobs
)
Data Engineering Manager – Web Crawling & Pipeline Architecture ( 7 To 2 Yrs)
AIMLEAP • Raipur, Republic Of India, IN
Data Engineering Manager – Web Crawling & Pipeline Architecture.Tech / MCA / Computer Science / IT.IT / Data / AI / E-commerce / FinTech / Healthcare.
Experience working with cloud platforms such as...Show more
Last updated: 2 days ago • Promoted
Data analyst
BHEL • Raipur, Chhattisgarh, India
We suggest you enter details here.This is a full-time, on-site Data Analyst role located in Raipur.The Data Analyst will be responsible for collecting, organizing, and analyzing data to support dec...Show more
Last updated: 1 day ago • Promoted
Data Analyst
BHEL • Raipur, Chhattisgarh, India
We suggest you enter details here.This is a full-time, on-site Data Analyst role located in Raipur.The Data Analyst will be responsible for collecting, organizing, and analyzing data to support dec...Show more
Last updated: 2 days ago • Promoted
Data Engineer
TerraGiG • raipur, chattisgarh, in
Lead the design, development, and implementation of data solutions using AWS and Snowflake.Collaborate with cross-functional teams to understand business requirements and translate them into techni...Show more
Last updated: 30+ days ago • Promoted
Lead Geology
Mining Tech Consultancy Services (MTCS) • Raipur, Chhattisgarh, India
Mining Tech Consultancy Services Ltd.MTCS brings deep domain expertise across a wide range of commodities including.With a focus on innovation, compliance, and operational excellence, MTCS supports...Show more
Last updated: 3 days ago • Promoted
Senior Data Engineer
CXC • raipur, chattisgarh, in
Please apply only if you are available to work in Australian time zone and comfortable with 6 months contract duration • •.
We’re seeking a highly skilled and autonomous.Power BI implementations to jo...Show more
Last updated: 17 hours ago • Promoted • New!
Shopify Project Manager
Upbott Consulting, Inc • raipur, chattisgarh, in
E-commerce Project Manager- Shopify / Big commerce.BigCommerce or Shopify projects.Candidates must have led end-to-end e-commerce implementations specifically on.
This role requires someone who under...Show more
Last updated: 3 days ago • Promoted
AI Analyst
Aventis Solutions • raipur, chattisgarh, in
Aventis Solutions is igniting the AI revolution : .They have just launched The AI Executive podcast, which can be found here : .
Now, our tech partner is establishing a new AI Innovation Hub in Pune, In...Show more
Last updated: 30+ days ago • Promoted
Lead Full-Stack + AI Engineer (Founding Team)
Grovio AI • raipur, chattisgarh, in
We’re building an autonomous, multi-agent AI OS that plans, executes, and optimizes marketing across modern digital ecosystems.
Think : an AI that acts like a virtual CMO — planning, writing, analyz...Show more
Last updated: 2 days ago • Promoted
Senior Data Engineer
Primesoft Inc • raipur, chattisgarh, in
Primesoft Enterprise IT Services Pvt.As a Software Engineer II - Data, you will contribute to the design and development of data systems including pipelines, APIs, analytics, AI and machine learnin...Show more
Last updated: 30+ days ago • Promoted
Data Scientist
Recro • raipur, chattisgarh, in
We’re seeking a highly skilled, hands-on Data Scientist with 4–10 years of experience in applied AI / ML to join our fast-paced team.
This role requires deep expertise in transformer architectures and...Show more
Last updated: 30+ days ago • Promoted
Data Product Owner
Integrated Wireless Solutions • raipur, chattisgarh, in
We're looking for an experienced.Candidate & Recruiting Data Product Portfolio.In this role, you’ll define the vision, roadmap, and delivery of data products that power hiring insights, analytics, ...Show more
Last updated: 17 hours ago • Promoted • New!
Full Stack Engineer
Plumloom • raipur, chattisgarh, in
We’re building an AI evaluation platform that’s becoming indispensable for AI teams.As a Full Stack Developer (backend focused), you’ll craft intuitive, high-performance user experiences and mainta...Show more
Last updated: 30+ days ago • Promoted
Project Manager
Hello Energy • raipur, chattisgarh, in
Client & Utility data connections onboarding.We are looking for a Project Manager who can coordinate and deliver multiple small-to-medium projects simultaneously, each representing a client’s build...Show more
Last updated: 3 days ago • Promoted
Full Stack Engineer
Programmers.io • raipur, chattisgarh, in
Job Title : Senior Full Stack Developer (Laravel + Vue).We are seeking highly skilled Senior Full Stack Developers with 7–10 years of experience in Laravel and modern frontend frameworks (Vue.The ca...Show more
Last updated: 17 days ago • Promoted
Co-Founder / Head of Engineering
Cropsphere • raipur, chattisgarh, in
Cropsphere is building India’s first.Our ecosystem consists of four layers : .Agri + Climate OS for soil, crop, land & supply intelligence (IoT, AI, computer vision, satellite, disease prediction).We...Show more
Last updated: 10 hours ago • Promoted • New!
Senior Technical Project Manager
VBeyond Corporation • raipur, chattisgarh, in
We are seeking a seasoned Senior Technical Project Manager to lead the planning, execution, and delivery of complex technology initiatives.
The ideal candidate is a strategic thinker with a strong t...Show more
Last updated: 17 hours ago • Promoted • New!
Interior Design Project Manager
Livspace • Raipur, Chhattisgarh, India
The ideal candidate will be responsible for planning, coordinating, and implementing projects within the decided-upon budget, timeline, and scope.
They will also effectively monitor and present proj...Show more
Last updated: 9 days ago • Promoted