ZettaMine - Data Engineer - ETL / Azure

Role Overview :

ZettaMine is seeking a skilled and experienced Data Engineer to design, build, and maintain robust data pipelines and infrastructure on the Azure platform. The ideal candidate will have hands-on experience working with modern data technologies and tools, and a strong understanding of data modeling, ETL / ELT pipelines, and cloud-native services in Azure.

This role requires working closely with data architects, analysts, and business stakeholders to enable data-driven decision-making by ensuring the availability, quality, and reliability of enterprise data.

Key Responsibilities :

• Design, develop, and optimize scalable ETL / ELT pipelines for both batch and real-time / streaming data ingestion and processing.
• Develop robust, reusable, and high-performance data ingestion frameworks using Azure Data Factory (ADF) and Azure Databricks.
• Implement workflows to ingest structured and unstructured data from various on-prem and cloud sources.
• Build and maintain data solutions using Azure Data Services, including ADF, Azure Synapse Analytics, Data Lake, Blob Storage, and Databricks.
• Ensure seamless integration of data systems across different Azure services.
• Collaborate with platform and infrastructure teams to optimize data storage, access, and performance.
• Design and implement data models (star / snowflake, normalized / denormalized) suitable for analytical and operational workloads.
• Perform complex data transformations and aggregations to make data analytics-ready.
• Apply data quality and governance standards across pipelines.
• Optimize performance of data processes through indexing, partitioning, caching, and tuning of Spark SQL or T-SQL queries.
• Monitor pipeline performance, troubleshoot issues, and ensure reliability and scalability.
• Lead or support data migration initiatives from on-premises systems to Azure cloud environments.
• Integrate various data sources, including APIs, databases, flat files, and streaming services.
• Follow CI / CD practices using DevOps tools for automated deployment and testing.
• Maintain detailed documentation of workflows, data models, and system configurations.
• Advocate for and implement data security, privacy, and compliance best practices.

Required Skills & Experience

1. Python for data processing and transformation

2. SQL (T-SQL or Spark SQL) for querying and scripting

3. Azure Data Factory (ADF) Pipelines, triggers, linked services

4. Azure Synapse Analytics or SQL DW

5. Databricks Notebooks, Spark, data exploration & transformation

(ref : hirist.tech)