Python Developer (Web Crawling Exp)

Python Developer (Web Crawling Exp) - 5+years - Kochi

Responsibilities :

Develop and maintain efficient web crawlers with a focus on speed, performance, and scalability .

Write clean, reusable, and optimized Python code for large-scale data extraction.

Analyze and understand website structures, implement logic to handle dynamic and JS-rendered pages .

Troubleshoot and debug crawl issues, optimize request handling, and improve crawler reliability.

Collaborate with backend and data teams to ensure smooth data flow and integration.

Monitor crawler performance, logs, and resolve runtime errors.

Participate in code reviews, documentation, and continuous process improvements.

Skills Required :

Strong coding skills in Python .

Good understanding of how web pages, HTTP, and crawlers work.

Hands-on experience with Docker , Postman , and PowerShell .

Basic understanding of HTML , AngularJS , RabbitMQ , and Celery .

Experience with Git and Bitbucket for version control and team collaboration.

Familiarity with XPath , CSS selectors , and regular expressions for content extraction.

Preferred :

Exposure to Scrapy , Puppeteer / Playwright , or other crawling frameworks.

Understanding of Elasticsearch , HAProxy , or Curator for large-scale data handling.