Python Developer (Web Crawling Exp)

Python Developer (Web Crawling Exp) - 5+years - Kochi

Responsibilities :

Develop and maintain efficient

web crawlers

with a focus on

speed, performance, and scalability .

Write clean, reusable, and optimized

Python

code for large-scale data extraction.

Analyze and understand website structures, implement logic to handle

dynamic and JS-rendered pages .

Troubleshoot and debug

crawl issues, optimize request handling, and improve crawler reliability.

Collaborate with backend and data teams to ensure smooth data flow and integration.

Monitor crawler performance, logs, and resolve runtime errors.

Participate in code reviews, documentation, and continuous process improvements.

Skills Required :

Strong coding skills in

Python .

Good understanding of how

web pages, HTTP, and crawlers

work.

Hands-on experience with

Docker ,

Postman , and

PowerShell .

Basic understanding of

HTML ,

AngularJS ,

RabbitMQ , and

Celery .

Experience with

Git

and

Bitbucket

for version control and team collaboration.

Familiarity with

XPath ,

CSS selectors , and

regular expressions

for content extraction.

Preferred : Exposure to

Scrapy ,

Puppeteer / Playwright , or other crawling frameworks.

Understanding of

Elasticsearch ,

HAProxy , or

Curator

for large-scale data handling.