Senior Data Engineer - Python/PySpark

About the Company :

A leading technology organization delivering cloud-native data engineering and analytics solutions for global enterprises. The company specializes in modern data platforms, large-scale data processing, and digital transformation, enabling businesses to derive actionable insights through scalable, secure, and high-performance data solutions.

Job Summary :

We are seeking a Senior Data Engineer with strong expertise in Python, PySpark, Azure Data Services, and ETL development to build and optimize scalable data platforms. The ideal candidate will have experience designing enterprise-grade data pipelines, implementing cloud-based analytics solutions, and driving data engineering best practices.

Key Responsibilities :

- Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.

- Build and optimize data processing solutions on Azure Databricks.

- Develop and maintain data integration workflows using Azure Data Factory.

- Design and optimize SQL queries, schemas, and data models for high-performance analytics.

- Implement CI/CD pipelines using GitHub Actions for automated testing and deployments.

- Deploy and manage applications using OpenShift and HELM.

- Configure monitoring dashboards using Grafana to ensure data quality and pipeline health.

- Collaborate with cross-functional teams to deliver secure, scalable, and reliable data solutions.

- Implement best practices for performance optimization, data governance, and cloud architecture.

Required Skills :

- Python

- PySpark

- ETL/ELT Development

- SQL

- Data Warehousing

- Azure Databricks

- Azure Data Factory

- Azure SQL Server

- Azure Key Vault

- GitHub

- GitHub Actions

- CI/CD

- OpenShift

- Kubernetes

- HELM

- Grafana

- Data Pipeline Development

- Data Modeling

- Cloud Data Engineering.

Job Highlights :

- Build enterprise-scale data platforms using Azure cloud technologies.

- Design and optimize high-performance ETL/ELT pipelines.

- Work with modern DevOps, containerization, and cloud-native data engineering practices.

- Collaborate on large-scale analytics and digital transformation initiatives using Azure and Databricks.

(ref:hirist.tech)