Azure Data Pipeline Architect

Key Responsibilities :

• Design and build scalable ETL / ELT pipelines using Azure Data Factory (ADF), Azure Databricks (Spark), and Azure Synapse Analytics.
• Develop and optimize PySpark / Spark SQL jobs for large-scale batch and streaming data transformations.
• Ingest data from various sources including Apache Kafka, REST APIs, and RDBMS, ensuring real-time or near-real-time availability.
• Implement data modeling strategies (star schema, snowflake schema) for analytics consumption layers in Synapse or ADLS.
• Collaborate with DevOps teams to automate deployment using CI / CD pipelines (Azure DevOps, GitHub Actions, etc.).
• Monitor, troubleshoot, and optimize data workflows for performance, cost-efficiency, and reliability.
• Follow coding standards, participate in peer reviews, and maintain version-controlled code in Git repositories.
• Support data quality checks, logging, alerting, and observability mechanisms for production workloads.
• Participate in sprint ceremonies and contribute to task estimation and delivery planning.

Must-Have Skills :

Azure Data Factory (ADF) – building pipelines, triggers, linked services.

Azure Databricks – building and managing Spark jobs in PySpark.

Azure Synapse Analytics – data warehousing, SQL queries, workspace orchestration.

Apache Kafka – consuming and processing real-time data streams.