Data Engineer

Position Summary:

We are seeking a talented Databricks Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing building and maintaining data pipelines using a variety of technologies with a focus on the Microsoft Azure cloud platform.

Responsibilities:

• Design develop and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
• Write efficient SQL queries to extract transform and load (ETL) data from various sources into Azure Synapse Analytics.
• Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
• Collaborate with data analysts to understand data requirements and ensure data quality.
• Hands-on experience in designing and developing Datalakes and Warehouses
• Implement data governance practices to ensure data security and compliance.
• Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
• Develop and maintain unit tests for data pipeline code.
• Work collaboratively with other engineers and data professionals in an Agile development environment.

Must Have tech skills

• Databricks
• Fabric
• Sql
• Pyspark
• Python

Must Have competencies

• Pipeline - Data Ingestion and data quality transformation Databricks ADF
• Consumption of rest api
• Optimization of pyspark job sql procedure views

Nice to have ( These are design and admin skills)

• Warehouse and lake house creation
• Modeling
• Semantic model design
• Refresh management
• Logging and monitoring
• Rls and permissions

New Vision is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees

Required Experience:

IC

Key Skills
Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala
Employment Type : Full-Time
Experience: years
Vacancy: 1