Digital : Databricks

Experience : 3-9years

Location : Hyderabad

Required Technical Skill Set :

Spark-SQL, Pyspark, Pandas,DevOps, CI / CD ,as SQL server, Oracle

3+ years experienced data engineer with sound ADF, Databricks and Pyspark knowledge and able to interact independently with customer technical team / leads

Responsibilities :

Assure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements

Develop and maintain innovative Azure solutions, Perform transformation using Databricks notebooks with Pyspark.

Solution design using Microsoft Azure services and other tools

The ability to automate tasks and deploy production standard code (with unit testing, continuous integration, versioning etc.)

Load transformed data into storage and reporting structures in destinations including data warehouse, high speed indexes, real-time reporting systems and analytics applications

Build data pipelines to collectively bring together data and setting up their orchestration and schedules ( DAGs)