MonoSpear Technologies - Azure Databricks Engineer - Spark / Python

Job Title : Azure Databricks Expert (3+ years of experience)

Job Description :

We are seeking an experienced Azure Databricks Expert with hands-on expertise in Spark and PySpark to join our team.

The ideal candidate will have a minimum of 3 years of experience in delivering Azure-based projects and working with various Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse Analytics, and Azure SQL Data Warehouse (SQL DW).

As an Azure Databricks Expert, you will be responsible for developing and optimizing data pipelines, implementing scalable data solutions, and contributing to end-to-end data workflows.

You will also be required to provide guidance on best practices for working with Databricks in the Azure cloud environment.

Key Responsibilities :

• Design, implement, and optimize data pipelines using Databricks, Spark, and PySpark for large-scale data processing.
• Work with Azure Data Lake Storage (ADLS) to manage and store large volumes of structured and unstructured data.
• Utilize Azure Data Factory (ADF) for orchestrating and automating data workflows.
• Implement end-to-end data solutions using Azure Synapse Analytics and Azure SQL Data Warehouse (SQL DW) for data warehousing and analytics.
• Collaborate with data engineers, data scientists, and stakeholders to build scalable, high-performance data pipelines.
• Troubleshoot and optimize performance issues related to Databricks and Spark jobs.
• Contribute to the development of best practices for data processing, pipeline design, and data integration.
• Ensure data solutions are designed and implemented with a focus on scalability, reliability, and security.
• Document processes, workflows, and code to ensure maintainability and knowledge sharing.

Qualifications :

Preferred Skills :

(ref : hirist.tech)