Freelance Deep Web Crawler Engineer (AI-Integrated Data Pipeline)
Sixteen Alpha AIlucknow, uttar pradesh, in7 days ago
Job descriptionProven experience building deep or dark web crawlers (Playwright, Scrapy, Puppeteer, or custom async frameworks). Strong understanding of browser automation, session management, and anti-detection mechanisms . Experience integrating AI / ML / NLP pipelines — e.g., text classification, entity recognition, or embedding-based similarity. Skilled in asynchronous Python (asyncio, aiohttp, Playwright async API). Familiar with database and pipeline systems — PostgreSQL, MongoDB, Elasticsearch, or similar. Ability to design robust data flows that connect crawling → AI inference → storage / visualization. Knowledge of LLMs (OpenAI, Hugging Face, LangChain, or custom fine-tuned models) . Experience with data cleaning, deduplication, and normalization pipelines . Familiarity with distributed crawling frameworks (Ray, Celery, Kafka) . Prior experience integrating real-time analytics dashboards or monitoring tools. Competitive freelance pay based on expertise and delivery. Flexible, async-first remote collaboration. Opportunity to shape an AI-first data platform from the ground up. Potential for long-term partnership if the collaboration is successful.
About the Project
We’re developing a next-generation intelligent web crawling system capable of exploring deep and dynamic web data sources — including sites behind authentication, infinite scrolls, and JavaScript-heavy pages.
The crawler will be integrated with an AI-driven pipeline for automated data understanding, classification, and transformation.
We’re looking for a highly experienced engineer who has previously built large-scale, distributed crawling frameworks and integrated AI or NLP / LLM-based components for contextual data extraction.
Key Responsibilities
- Design, develop, and deploy scalable deep web crawlers capable of bypassing common anti-bot mechanisms.
- Implement AI-integrated pipelines for data processing, entity extraction, and semantic categorization.
- Develop dynamic scraping systems for sites that rely on JavaScript, infinite scrolling, or APIs.
- Integrate with vector databases , LLM-based data labeling, or automated content enrichment modules.
- Optimize crawling logic for speed, reliability, and stealth across distributed environments.
- Collaborate on data pipeline orchestration using tools like Airflow, Prefect, or custom async architectures.
Required Expertise
Nice to Have
What We Offer
Create a job alert for this search
Engineer • lucknow, uttar pradesh, in
Related jobs
)
)
- Promoted
Web Developer
GOODFAITH BRANDING & DIGITAL MARKETINGLucknow, IN GOODFAITH Branding & Digital Marketing specializes in creating deeply human, optimally structured and impactful digital branding and marketing strategies rooted in logic.
Combining advanced AI tools...Show moreLast updated: 5 days ago
- Promoted
Data Engineer
RecroLucknow, IN Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake.
AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago
- Promoted
AI Platform Engineer
BayOne SolutionsLucknow, IN We are seeking a highly skilled.In this role, you will work on advanced AI systems including.Retrieval-Augmented Generation (RAG).
Model Context Protocol (MCP) tools.OpenWebUI or custom-built soluti...Show moreLast updated: 6 days ago
- Promoted
- New!
Deep Learning Engineer
Tomorrow World Technology (TWT)Lucknow, IN Position : Deep Learning Engineer – Computer Vision & Autonomy.An experienced Deep Learning Engineer specializing in Computer Vision, Sensor Fusion, and Multimodal AI.
R&D; in autonomous aerial syste...Show moreLast updated: 4 hours ago
- Promoted
Data Engineer - Web Scraping
Alternative PathLucknow, IN Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm.In this role, you will collaborate with individuals across various company de...Show moreLast updated: 30+ days ago
- Promoted
Ai Web Scraping Engineer
Acuity Knowledge PartnersLucknow, Republic Of India, IN Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm.In this role, you will collaborate with individuals across various company de...Show moreLast updated: 2 days ago
- Promoted
Google Looker Data Engineer
Insight GlobalLucknow, IN We’re seeking a skilled Data Engineer to support a major reporting migration from Tableau to Google Looker.This role focuses on designing and building scalable data models, data marts, and ETL / ELT ...Show moreLast updated: 7 days ago
- Promoted
- New!
Web Scraping Engineer
noonlucknow, India Job title : Web Scraping Engineer.The ideal candidate will design and implement robust scrapers to collect, clean, and normalize product data (pricing, availability, reviews, images, etc.Develop and...Show moreLast updated: 21 hours ago
- Promoted
Lead AI Engineer
BlendLucknow, IN We are looking for an AI Engineer with hands-on experience designing and deploying scalable AI solutions.In this role, you will be part of a cross-functional team working on cutting-edge projects i...Show moreLast updated: 11 days ago
- Promoted
Forward Deployed Engineer
Searchability®Lucknow, IN Forward Deployed Engineer - AI💻.Remote-based - relocation to Dubai📍.Searchability MENA is working with an innovative AI startup looking for a.
This is a rare chance to get involved with a company ...Show moreLast updated: 2 days ago
- Promoted
- New!
Senior Web Scraping Engineer
Zomunklucknow, India We're building a product that relies heavily on collecting structured data from a number of known websites.We need someone experienced who can own this part of the system end-to-end; from writing s...Show moreLast updated: 21 hours ago
- Promoted
Freelance AI / ML Engineer (ETL + MLOps + AWS)
ThreatXIntelLucknow, IN ThreatXIntel is a dedicated cybersecurity startup specializing in protecting businesses and organizations from cyber threats.
With services such as cloud security, web and mobile security testing, c...Show moreLast updated: 1 day ago
- Promoted
Full Stack Web Developer
DecodesLucknow, IN Familiarity with containerization tools (.A maker mindset! you care about both.Build and ship features end-to-end, fast.Work closely with design, product, and other devs to make magic happen.Keep t...Show moreLast updated: 2 days ago
- Promoted
AI Ops Engineer + Rag + AIML+ GitHub Copilot(Hybrid : Bangalore)
DigiHelic Solutions Pvt. Ltd.Lucknow, IN Champion the pragmatic use of AI tools (e.GitHub Copilot) across various development teams.Design and simulate realistic use cases to demonstrate the value of AI tools in different tech stacks.Act ...Show moreLast updated: 1 day ago
- Promoted
Full Stack AI engineer
AnswerThis (YC F25)Lucknow, IN Remote (Applications open worldwide).Semantic Search, Vector Databases, Prompt Engineering, GenAI Frameworks, React Agents, Graph Agents, Document Parsing, Python, Scalable APIs.AnswerThis is an AI...Show moreLast updated: 30+ days ago
- Promoted
Webflow Developer
Summit Tech.auLucknow, IN Summit Tech is an Australian technology agency dedicated to providing affordable and innovative online solutions for enterprise companies.
Based in Australia, Summit Tech combines cutting-edge techn...Show moreLast updated: 1 day ago
- Promoted
Web Analytics & Tracking Lead
The Conqueror ChallengesLucknow, IN We are a growing team of passionate, performance-driven individuals on a mission to be the best at growing multiple international e-commerce businesses with great products.Over the past 8 years, we...Show moreLast updated: 26 days ago
- Promoted
Machine Learning Engineer-Agentic AI
Innodata Inc.Lucknow, IN Design and implement multi-agent systems using LangChain, LangGraph, CrewAI, AutoGen or similar frameworks.Build A2A (agent-to-agent) orchestration and implement MCP (multi-context protocol) for co...Show moreLast updated: 15 days ago