Web Crawling Engineer
Forage AIThoothukudi, IN9 days ago
Job descriptionProficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy). Strong knowledge of browser automation tools such as Playwright or Selenium. Solid understanding of HTML, CSS, and selector languages (XPath / CSS). Experience in handling anti-scraping challenges and ensuring robust data extraction. Familiarity with distributed scraping techniques and data pipelines. Ability to troubleshoot and optimize web crawlers for performance and reliability. Strong analytical and problem-solving skills with attention to detail. Excellent communication and inter-personal skills. High-speed internet connectivity for video calls and efficient work. Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and Headphones with clear audio quality. Stable power connection and backups in case of internet / power failure.
We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.
Salary budget upto - Rs. 10 LPA
About Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.
Key Responsibilities :
- Maintain and enhance existing web scraping and data crawling projects.
- Develop and refine crawlers using Python-based tools and frameworks.
- Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
- Clean, validate, and integrate extracted data into downstream storage systems.
- Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
- Optimize crawling efficiency and ensure compliance with web crawling best practices.
- Collaborate with cross-functional teams to improve data acquisition strategies.
Required Skills & Qualifications :
Other Infrastructure Requirements
Since this is a completely work-from-home position, you will also require the following -
no other obstacles to interrupted, efficient work).
Create a job alert for this search
Engineer • Thoothukudi, IN
Related jobs
- Promoted
Freelance Webflow Developer
Black Pianothoothukudi, tamil nadu, in We are looking for a talented and detail-oriented.In this role, you will be responsible for translating high-fidelity designs into responsive, pixel-perfect websites using Webflow.You’ll collaborat...Show moreLast updated: 15 days ago
- Promoted
SFRA Frontend Developer
Systems PlusThoothukudi, IN SystemsPlus is hiring for SFRA Front End Developer.SFCC / SFRA frontend experience with composable commerce).We are looking for an experienced.
Salesforce Commerce Cloud (SFCC).Storefront Reference Ar...Show moreLast updated: 7 days ago
- Promoted
- New!
Senior Web Developer
Innomax IT SolutionsThoothukudi, IN Job Title : Senior Developer (Web & Mobile App Development).Company : Innomax IT Solutions Pvt Ltd (8+years IT & Product Engineering expertise).
Innomax IT is looking for a Senior Developer who can le...Show moreLast updated: 15 hours ago
- Promoted
WebMethod Developer
MSH IndiaThoothukudi, IN Hiring : Senior webMethods Consultant | 8+ Years Experience | Offshore (India).Join our Integration team as a Senior webMethods Consultant!.
We’re looking for an experienced professional who can desi...Show moreLast updated: 27 days ago
- Promoted
Web Designer
SocialPilotThoothukudi, IN SocialPilot is seeking a talented Web Designer to join our dynamic team.In this role, you will contribute to building impactful SaaS products in the Martech space that resonate with our customers.Y...Show moreLast updated: 30+ days ago
- Promoted
- New!
Web Designer
EaseMyTrip.comThoothukudi, IN The Web Designer at EaseMyTrip.This job requires knowledge in web technologies like HTML5, CSS3, and JavaScript, as well as design tools such as Photoshop.
The designer will make websites that work ...Show moreLast updated: 15 hours ago
- Promoted
- New!
Back End Developer
DistronixThoothukudi, IN We are looking for an experienced.The ideal candidate should be proficient in building scalable and efficient server-side applications, integrating multiple data sources, and ensuring high performa...Show moreLast updated: 15 hours ago
- Promoted
Back End Developer
BayOne SolutionsThoothukudi, IN As a key member of our engineering team, you’ll contribute to the architecture, development, and operations of mission-critical services that support our platform.
You’ll be part of a collaborative ...Show moreLast updated: 30+ days ago
- Promoted
- New!
Drupal and React QA Engineer
Cloudely, IncThoothukudi, IN Hiring : Drupal and React QA Engineer for web application testing.Type : Fulltime permanent opportunity.We are seeking support for our in-house applications to Trust, Help, and Brand websites, which ...Show moreLast updated: 15 hours ago
- Promoted
- New!
Senior System Engineer
Creditsafe TechnologyThoothukudi, IN We are seeking a skilled and enthusiastic.In this position, you will guide fellow developers and collaborate within an Agile environment to deliver high-quality products.
Develop, test, and maintain...Show moreLast updated: 15 hours ago
- Promoted
- New!
Back-End-Entwickler
EXIDEUS LLCThoothukudi, IN We’re not building “just another platform.Sales, Medicine, and Psychology.Our MVP is in an advanced stage, the foundation is solid, and we already have a strong core team (Backend, Frontend, Design...Show moreLast updated: 15 hours ago
- Promoted
- New!
Web Designing
WS BakersThoothukudi, IN WS Bakers is a renowned pastry and confectionery chain based in Pune, bringing joy to customers for over 10 years.We specialize in offering freshly baked, pure vegetarian cakes, pastries, and desse...Show moreLast updated: 15 hours ago
- Promoted
- New!
Web Development Specialist
Marsh McLennanThoothukudi, IN We are seeking a talented individual to join our operations team (MMA) at Marsh.This role will be based in Mumbai.This is a hybrid role that has a requirement of working at least three days a week ...Show moreLast updated: 9 hours ago
- Promoted
- New!
Web Developer
Talentoindia - Hire Candidates | Find JobsThoothukudi, Republic Of India, IN This is a full-time remote Web Developer role.In this position, you will design, develop, and maintain high-quality websites and web applications with a focus on both back-end and front-end functio...Show moreLast updated: 12 hours ago
- Promoted
- New!
PHP Full-stack Developer
BeGigThoothukudi, IN Job Role- PHP Full-stack Developer.The ideal candidate must have excellent knowledge of both, along with experience in.Develop, enhance, and maintain web applications using.Collaborate with UI / UX d...Show moreLast updated: 13 hours ago
- Promoted
AI Web Scraping Engineer
S2T AI - AI-Powered InvestigationsThoothukudi, IN We're seeking a forward-thinking.AI tools to accelerate development and streamline data extraction processes.Join our India team and work at the intersection of traditional scraping expertise and c...Show moreLast updated: 30+ days ago
- Promoted
Senior Front-End Web Developer (HTML & Bootstrap)
KBM ResortsThoothukudi, IN Copy the following with answers and email to.Number of years and version experience in : .Notice period in current company (or) How soon can you join ? (in days).
Attach Resume along with the email.Ar...Show moreLast updated: 30+ days ago
- Promoted
- New!
React.js + PHP Developer
Discoveries QuintessentialThoothukudi, IN Software Development Engineer (SDE-II).We are a leading retail technology company building India’s most advanced omni-retail software suite, designed to streamline operations for both online and of...Show moreLast updated: 15 hours ago