AI-Powered Web Crawling Specialist (Freelance)
Sixteen Alpha AINew Delhi, Republic Of India, IN9 days ago
Job descriptionProven experience building deep or dark web crawlers (Playwright, Scrapy, Puppeteer, or custom async frameworks). Strong understanding of browser automation, session management, and anti-detection mechanisms . Experience integrating AI / ML / NLP pipelines — e.G., text classification, entity recognition, or embedding-based similarity. Skilled in asynchronous Python (asyncio, aiohttp, Playwright async API). Familiar with database and pipeline systems — PostgreSQL, MongoDB, Elasticsearch, or similar. Ability to design robust data flows that connect crawling → AI inference → storage / visualization. Knowledge of LLMs (OpenAI, Hugging Face, LangChain, or custom fine-tuned models) . Experience with data cleaning, deduplication, and normalization pipelines . Familiarity with distributed crawling frameworks (Ray, Celery, Kafka) . Prior experience integrating real-time analytics dashboards or monitoring tools. Competitive freelance pay based on expertise and delivery. Flexible, async-first remote collaboration. Opportunity to shape an AI-first data platform from the ground up. Potential for long-term partnership if the collaboration is successful.
About the Project
We’re developing a next-generation intelligent web crawling system capable of exploring deep and dynamic web data sources — including sites behind authentication, infinite scrolls, and JavaScript-heavy pages.
The crawler will be integrated with an AI-driven pipeline for automated data understanding, classification, and transformation.
We’re looking for a highly experienced engineer who has previously built large-scale, distributed crawling frameworks and integrated AI or NLP / LLM-based components for contextual data extraction.
Key Responsibilities
- Design, develop, and deploy scalable deep web crawlers capable of bypassing common anti-bot mechanisms.
- Implement AI-integrated pipelines for data processing, entity extraction, and semantic categorization.
- Develop dynamic scraping systems for sites that rely on JavaScript, infinite scrolling, or APIs.
- Integrate with vector databases , LLM-based data labeling, or automated content enrichment modules.
- Optimize crawling logic for speed, reliability, and stealth across distributed environments.
- Collaborate on data pipeline orchestration using tools like Airflow, Prefect, or custom async architectures.
Required Expertise
Nice to Have
What We Offer
Create a job alert for this search
Freelance Specialist • New Delhi, Republic Of India, IN
Related jobs
- Promoted
- New!
Webflow Developer
goatdevelopers.comNagpur, IN From concept to launch, we cover the full spectrum - UI / UX design, backend engineering, AI integration, and frontend development.
Our team has successfully partnered with clients across the globe, d...Show moreLast updated: 21 hours ago
- Promoted
Web Crawling Engineer
Forage AINagpur, IN The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.
Forage AI is a pioneering A...Show moreLast updated: 11 days ago
- Promoted
- New!
Webflow CMS Specialist
goatdevelopers.comRepublic Of India, IN From concept to launch, we cover the full spectrum - UI / UX design, backend engineering, AI integration, and frontend development.
Our team has successfully partnered with clients across the globe, d...Show moreLast updated: 12 hours ago
- Promoted
Web Developer - Webflow / Shopify
Snaphunt Pte LtdIndia Description : - Opportunity within a company with a solid track record of performance Our client is a Google Partner and B Co...Show moreLast updated: 12 days ago
- Promoted
Ai Web Scraping Engineer
S2T AI - AI-Powered InvestigationsRepublic Of India, IN We're seeking a forward-thinking.AI tools to accelerate development and streamline data extraction processes.Join our India team and work at the intersection of traditional scraping expertise and c...Show moreLast updated: 30+ days ago
- Promoted
- New!
Full Stack Web Developer
GT TechnovationNagpur, IN Full Stack Web Developer – Internship (Remote).Duration : 1–6 Months (Based on performance & availability).GT Technovation is an emerging tech company focused on building secure, scalable, and moder...Show moreLast updated: 11 hours ago
- Promoted
Asana & AI Productivity – Freelance SME
LetsTuteIndia, India Let'sTute is actively looking for experienced .Asana Mastery & Workflow Optimization.Automation, Reporting, & Productivity Enhancements.
Integrations with Jira, Power BI & other tools.Data Quality, ...Show moreLast updated: 1 day ago
- Promoted
Technical SEO Specialist
QR TIGER - QR Code GeneratorIndia, India Technical SEO Specialist (Expert).QR TIGER, you will take a role in shaping and maintaining the technical foundation of our digital ecosystem.
Your mission is to optimize site infrastructure, enhanc...Show moreLast updated: 2 days ago
- Promoted
Freelance Webflow Developer
Black PianoPune, Republic Of India, IN We are looking for a talented and detail-oriented.In this role, you will be responsible for translating high-fidelity designs into responsive, pixel-perfect websites using Webflow.You’ll collaborat...Show moreLast updated: 17 days ago
- Promoted
Sr Web Developer
ConfidentialIndia Marks and Methods Branding is a leading branding agency dedicated to helping businesses establish a compelling digital presence.
With a focus on innovative design, strategic branding, and cutting-ed...Show moreLast updated: 22 days ago
- Promoted
Hypervisor Sr QA
Avikal SolutionsIndia, India Mode of work : Onsite (Bangalore).This is a full-time remote role for a Hypervisor Senior QA professional.In this role, you will support testing efforts for virtualization platforms and hypervisors,...Show moreLast updated: 1 day ago
- Promoted
Freelance Writer
Taskify AINagpur, IN We're Hiring "Content Specialist (Freelance / Remote)" | Earn up to $2500 per month.Outsmart Artificial Intelligence with Your Human Brilliance.
Join a global community of talented writers to shape th...Show moreLast updated: 3 days ago
- Promoted
AI Web Scraping Engineer
S2T AI - AI-Powered InvestigationsIndia, India We're seeking a forward-thinking.AI tools to accelerate development and streamline data extraction processes.Join our India team and work at the intersection of traditional scraping expertise and c...Show moreLast updated: 30+ days ago
- Promoted
Freelance Deep Web Crawler Engineer (Ai-Integrated Data Pipeline)
Sixteen Alpha AINew Delhi, Republic Of India, IN The crawler will be integrated with an.AI or NLP / LLM-based components.JavaScript, infinite scrolling, or APIs.LLM-based data labeling, or automated content enrichment modules.Airflow, Prefect, or c...Show moreLast updated: 10 days ago
- Promoted
Freelance Web AR Filter Developer
ConfidentialIndia We're looking for a freelance Web AR developer experienced in building browser-based AR filters.Proven experience with Zappar, 8th Wall, or similar WebAR platforms.
Skilled in designing, deploying, ...Show moreLast updated: 22 days ago
- Promoted
- New!
Freelance Web Development Specialist (WordPress)
Egg Preservation Institute of AsiaRepublic Of India, IN Position : Freelance WordPress Developer (Remote).Monthly Hours : ~30 hours per month.Perform corrections, bug-fixes and optimisations on our current WordPress site : themes, plugins, CSS / HTML, respon...Show moreLast updated: 10 hours ago
- Promoted
Freelance Web Developer
ConfidentialIndia At AloneClone, we create intelligent entertainment, scalable systems, and future-ready AI, blending fun, function, and innovation into every product we build.
We have over 6 years of experience in f...Show moreLast updated: 22 days ago
- Promoted
Website Developer (Webflow Specialist)
ConfidentialNagpur, India Confidential (based on experience).Part Time Contract for 6 Months(40 hrs a week / 160 hrs a month).Note : This is a requirement for one of Uplers' client - UK Mental Health Care).What do you need for...Show moreLast updated: 22 days ago