Senior Python Developer

CJ KonsultantsMadurai, IN

1 day ago

Job description

Position : Python Developer

Location : Remote (India)

Salary : 80-90k pm

Experience : 4+ years

Job Description :

We’re looking for a Python Developer with solid hands-on experience in job and web data scraping,

AI-enabled data processing, and building scalable ETL pipelines. You’ll design, implement, and operate scraping and enrichment systems that feed recruitment and analytics products. Familiarity with Aspen / Webspidermount and Kadoa is a plus. QA-minded engineers who care about data accuracy and observability will thrive in this role.

Key Responsibilities :

Design, develop, and maintain AI-powered web and job-scraping pipelines.

Extract structured and unstructured data from job boards, company sites, APIs, and other sources.

Build and operate ETL processes to clean, normalize, enrich, and load data into analytics stores.

Ensure data accuracy, completeness, and consistency via automated validation and QA checks.

Implement and maintain monitoring, alerting, and observability for scraper health and data quality.

Optimize scraping and pipeline performance for reliability and scale (rate-limiting, retries, distributed workers).

Integrate scraped and enriched data with internal systems, analytics platforms, and downstream consumers.

Collaborate with product, data science, and QA teams to meet business and compliance requirements.

Stay current with AI, scraping techniques, legal / ethical scraping best practices, and anti-bot mitigations.

Required Skills :

Proficient in Python with production experience (3–5 years).
Strong SQL skills; experience with PostgreSQL and data modeling.
Hands-on with Python scraping and data libraries : BeautifulSoup, Scrapy, Selenium, Pandas .
Practical experience with unit and integration testing in Python (for example, pytest ).
Knowledge of concurrency in Python : multiprocessing and multithreading patterns.
Experience building robust data pipelines, working with APIs, rate limits, pagination, and proxying.
Proven experience building data scrapers and AI-assisted scraping / enrichment workflows.
Familiarity with data QA processes, validation frameworks, and creating automated data checks.
Excellent debugging skills and experience with logging, metrics, and alerting for production jobs.
Strong problem-solving, communication skills, and ability to work independently and in teams.

Customer- and user-oriented mindset.

Preferred Skills :

Experience with Aspen / Webspidermount and Kadoa platforms.

Hands-on experience with machine learning or AI-driven data processing and enrichment.

Familiarity with recruitment data, job-board ecosystems, and common job taxonomies (titles, skills, locations).

Experience with distributed task queues (e.g., Celery, RQ) or orchestration (Airflow, Prefect).

Experience with cloud platforms (AWS / GCP) and containerization (Docker, Kubernetes).

Knowledge of legal / compliance aspects of web scraping and data privacy.

What we offer :

Challenging problems at the intersection of data engineering, AI, and recruitment analytics.

Collaborative, fast-paced engineering culture that values quality and ownership.

Opportunity to influence data architecture and product direction.

Create a job alert for this search

Senior Python Developer • Madurai, IN