Azure Data Engineer

Role : Data Engineer

Exp : 5 Years+

Location : Remote / Hybrid

Work Timing : 2 PM - 10 PM UK Time Zone

Immediate Joiners

Key Skills : Python, Pyspark, Azure Cloud, Azure Functions for Data Extraction from different APIs , Azure Synapse, Data Bricks , JSON, SQL

We are looking for an experienced

Data Engineer (5+ years)

who has strong hands-on expertise in

building scalable data pipelines, API integrations, cloud-native solutions, and big data processing

• . The ideal candidate must have deep experience with

Python

PySpark

Azure Data Platform (ADLS, ADF, Functions, Synapse, Databricks)

, and

SQL-based data modeling

Key Responsibilities

1. Data Pipeline Development

Design, develop, and maintain

end-to-end ETL / ELT pipelines

using Azure Data Factory, Databricks, and PySpark.

Build scalable data ingestion workflows from various sources including

REST APIs, databases, cloud storage, and streaming systems

Implement

incremental load, CDC (Change Data Capture)

, and high-volume data ingestion frameworks.

2. API Data Extraction & Azure Functions

Develop

Azure Functions (Python)

to extract data from

REST / SOAP APIs

, handle pagination, throttling, and authentication (OAuth2, JWT, API keys).

Implement retry logic, monitoring, error handling, and logging in API ingestion pipelines.

Parse and transform

complex JSON / XML

structures.

3. Big Data Processing (PySpark / Databricks)

Write optimized

PySpark jobs

for data processing, cleansing, aggregation, and transformations.

Work with

Delta Lake

for ACID transactions, schema evolution, and time-travel queries.

Optimize Spark jobs using partitioning, caching, broadcast joins, and AQE.

4. Azure Cloud Data Platform

Utilize Azure services such as :

Azure Data Lake Storage (ADLS Gen2)

Azure Data Factory (ADF)

Azure Synapse Analytics (SQL Pool / Serverless SQL)

Azure Databricks

Azure Key Vault

Azure Functions

Deploy and manage solutions using

CI / CD pipelines

with Azure DevOps.

5. Data Modeling & SQL

Design and develop

data warehouse models

(Star Schema, Snowflake).

Write advanced SQL queries for transformations, validation, performance tuning.

Create and manage tables, partitions, stored procedures, and views in

Azure Synapse / SQL Server

6. Data Quality & Governance

Implement data validation, profiling, and quality checks.

Maintain metadata, lineage, and documentation using appropriate tools.

Ensure compliance with security policies, encryption, and access management.

Required Technical Skills

Core Skills

Python

(5+ years)

PySpark / Spark SQL

Azure Cloud

(ADF, ADLS, Functions, Synapse, Databricks)

REST API integration

JSON, XML Parsing

SQL (T-SQL / Spark SQL / Synapse SQL)

ETL / ELT development

Delta Lake