Web Crawling Engineer

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

Salary budget upto - Rs. 10 LPA

About Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

Key Responsibilities :

• Maintain and enhance existing web scraping and data crawling projects.
• Develop and refine crawlers using Python-based tools and frameworks.
• Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
• Clean, validate, and integrate extracted data into downstream storage systems.
• Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
• Optimize crawling efficiency and ensure compliance with web crawling best practices.
• Collaborate with cross-functional teams to improve data acquisition strategies.

Required Skills & Qualifications :

Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -

no other obstacles to interrupted, efficient work).