PySpark Developer - Azure Databricks

Job Title : PySpark Developer

Locations : Chennai, Hyderabad, Kolkata

Work Mode : MondayFriday (5 days WFO)

Experience : 5+ years in Backend / Data Engineering

Notice Period : Immediate 15 days

Must-Have : Python, PySpark, Amazon Redshift, PostgreSQL

About the Role :

We are seeking an experienced PySpark Developer with strong data engineering expertise to design, develop, and optimize scalable data pipelines for large-scale data processing. The role involves working across distributed systems, ETL / ELT frameworks, cloud data platforms, and analytics-driven architecture. You will collaborate closely with cross-functional teams to ensure efficient ingestion, transformation, and delivery of high-quality data.

Key Responsibilities :

• Design and develop robust, scalable ETL / ELT pipelines using PySpark to process data from databases, APIs, logs, and file-based sources.
• Convert raw data into analysis-ready datasets for data hubs and analytical data marts.
• Build reusable, parameterized Spark jobs for batch and micro-batch processing.
• Optimize PySpark performance to handle large and complex datasets.
• Ensure data quality, consistency, lineage, and maintain detailed documentation for all ingestion workflows.
• Collaborate with Data Architects, Data Modelers, and Data Scientists to implement data ingestion logic aligned with business requirements.
• Work with AWS services (S3, Glue, EMR, Redshift) for data ingestion, storage, and processing.
• Support version control, CI / CD practices, and infrastructure-as-code workflows as needed.

Must-Have Skills :

Good-to-Have Skills :

(ref : hirist.tech)