Develop and maintain efficient web crawlers with a focus on speed, performance, and scalability .
Write clean, reusable, and optimized Python code for large-scale data extraction.
Analyze and understand website structures, implement logic to handle dynamic and JS-rendered pages .
Troubleshoot and debug crawl issues, optimize request handling, and improve crawler reliability.
Collaborate with backend and data teams to ensure smooth data flow and integration.
Monitor crawler performance, logs, and resolve runtime errors.
Participate in code reviews, documentation, and continuous process improvements.
Skills Required :
Strong coding skills in Python .
Good understanding of how web pages, HTTP, and crawlers work.
Hands-on experience with Docker , Postman , and PowerShell .
Basic understanding of HTML , AngularJS , RabbitMQ , and Celery .
Experience with Git and Bitbucket for version control and team collaboration.
Familiarity with XPath , CSS selectors , and regular expressions for content extraction.
Preferred :
Exposure to Scrapy , Puppeteer / Playwright , or other crawling frameworks.
Understanding of Elasticsearch , HAProxy , or Curator for large-scale data handling.
Create a job alert for this search
Web Developer • vadodara, India
Related jobs
AI Engineer (Backend) - Python
Edstem Technologies • Anand, IN
We are building the next generation of AI-powered applications that deliver actionable financial and marketing insights to our clients.
As an AI Engineer focused on backend development, you'll be in...Show more
Last updated: 7 days ago • Promoted
HYPERVISOR TEST ENGINEER (Foundation Level)
Piepeople Consulting Inc. • Nadiad, IN
Solid understanding of hypervisors, virtual machines (VMs), and core concepts like CPU, memory, and I / O allocation.Basic operating systems (especially Linux), hardware basics, and fundamental progr...Show more
Last updated: 2 days ago • Promoted
Python Automation & Web Scraping Engineer (2 to 4 yrs)
AIMLEAP • Anand, IN
Python Automation & Web Scraping Engineer (WFH).Bachelor’s degree in Computer Science / Information Technology.Selenium, BeautifulSoup, Requests.
Experience in backend / API development using.Strong s...Show more
Last updated: 2 days ago • Promoted
Senior Python Developer
HISH IT SERVICES • anand, gujarat, in
Role : Senior Python Developer.Location : Remote(Banglore,Chennai,Pune).Pay : 23LPA - 27 LPA(Based on Experience).Timings : A couple of hours overlap with EST, as the client is Canada-based (till 12...Show more
Last updated: 10 days ago • Promoted
Frontend Developer
GoQuant • Vadodara, IN
Front-End Software Engineer (Python / JavaScript).GoQuant Technologies develops innovative solutions that enhance profitability and ease of use for digital asset traders and researchers.Our key produ...Show more
Last updated: 30+ days ago • Promoted
SFRA Backend Developer
Systems Plus • Vadodara, IN
SystemsPlus is hiring for SFRA Backend Developer.Salesforce Commerce Cloud (SFCC).Storefront Reference Architecture (SFRA).
The ideal candidate will build and maintain backend solutions, create inte...Show more
Last updated: 12 days ago • Promoted
Back End Developer
Digitalzone • Vadodara, IN
We are building a new AI-native engineering team that leverages Vibe Coding, a cutting-edge development workflow infused with AI agent pair programming (e.
Claude Code, Cursor AI, GitHub Copilot, Co...Show more
Last updated: 21 days ago • Promoted
(Laravel / PHP) Web developer with React Native Experience
TellByte • Vadodara, IN
PHP / Laravel applications into a.The ideal candidate will have a solid background in backend development, database management, and API design, with hands-on experience enabling smooth integration wi...Show more
Last updated: 3 days ago • Promoted
AI Web Scraping Engineer
S2T AI - AI-Powered Investigations • Nadiad, IN
We're seeking a forward-thinking.AI tools to accelerate development and streamline data extraction processes.Join our India team and work at the intersection of traditional scraping expertise and c...Show more
Last updated: 30+ days ago • Promoted
Web Crawling Engineer
Forage AI • Nadiad, IN
The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.
Forage AI is a pioneering A...Show more
Last updated: 14 days ago • Promoted
Backend Developer
Assima • Anand, IN
Assima is a worldwide leader in training, assistance and software adoption for Fortune 500 companies.Our patented technology is trusted by 15 of the top 20 banks, major international organizations ...Show more
Last updated: 3 days ago • Promoted
Web Developer
Smart Moves Consultants • Nadiad, IN
Design and develop high-performance, responsive web portals using React.Build scalable backend services and APIs with Node.
Integrate and optimize Snowflake for secure data storage, querying, and an...Show more
Last updated: 4 hours ago • Promoted • New!
Python Developer
Tata Consultancy Services • vadodara, gujarat, in
Required Skillset : Python, OOP, Django, Fast API.Core Python Knowledge, which is most important for any framework.Libraries and Frameworks (eg : Django, FastAPI etc).
At least basic knowledge of depl...Show more
Last updated: 30+ days ago • Promoted
Web Developer (Freelance)
Sweet • Nadiad, IN
Sweet is the AI-native business platform built for creators — a business partner that clears the clutter, automates the back-office, and gives creators the freedom to focus on craft, while Sweet gr...Show more
Last updated: 1 day ago • Promoted
Forward Deployed Engineer
Searchability® • Vadodara, IN
Forward Deployed Engineer - AI💻.Remote-based - relocation to Dubai📍.Searchability MENA is working with an innovative AI startup looking for a.
This is a rare chance to get involved with a company ...Show more
Last updated: 8 days ago • Promoted
Python Developer
GoML • anand, gujarat, in
We are in search of a passionate and skilled Python Developer to join our dynamic team.This role is ideal for someone with a profound expertise in Python and its ecosystem, focused on backend devel...Show more
Last updated: 5 days ago • Promoted
Software Engineer
Tilda Research • anand, gujarat, in
A passion for building scalable AI agents.Build scalable back-end services using.Optimize graph database queries and models in Neo4j.
Collaborate cross-functionally with Product, Engineering, and Cl...Show more
Last updated: 3 days ago • Promoted
Senior Python Developer
Aarav Solutions • vadodara, gujarat, in
Aarav Solutions is a global Product Engineering and IT Consulting services provider.Our goal is to enable organizations in their digital transformation journey, leveraging our domain knowledge and ...Show more