Data Engineer - Python / Apache Spark

Description :

Key Responsibilities :

• Design, develop, and deploy end-to-end data engineering solutions using Databricks, Apache Spark, PySpark, Python, and SQL.
• Build scalable and efficient ETL / ELT pipelines for data ingestion, transformation, and integration from various sources.
• Work with data warehousing solutions and ensure high-performance data processing and storage.
• Engage directly with clients to gather requirements, propose solutions, and manage delivery expectations.
• Collaborate with data scientists, analysts, and other engineers to support ML model deployment, MLOps, and advanced analytics use cases.
• Implement data modeling, data architecture, and adhere to best practices in modern data platform design.
• Utilize CI / CD tools such as GitHub, Azure DevOps, and Pipelines for version control, testing, and production deployment.
• Manage technical project delivery, including scope, timelines, and technical documentation.
• Troubleshoot performance issues and optimize distributed computing processes using Apache Spark.
• Continuously learn and apply new features and tools to enhance existing solutions and architectures.

Required Skills & Experience :

Preferred Qualifications :

(ref : hirist.tech)