Description :
Azure Data Bricks + SQL. Azure Data Bricks + SQL. by HR Neev Systems.
Experience : 5 + Years.
Location : Description :
We are seeking an experienced Azure Databricks + SQL Developer / Big Data Engineer to design, develop, and maintain scalable data solutions on Azure. The role will focus on building efficient ETL / ELT pipelines, optimizing SQL queries, and leveraging Databricks and other Azure services for advanced data processing, analytics, and data platform engineering. The ideal candidate will have a strong background in both traditional SQL development and modern big data technologies on Azure.
Key Responsibilities :
- Develop, maintain, and optimize ETL / ELT pipelines using Azure Databricks (PySpark / Spark SQL).
- Write and optimize complex SQL queries, stored procedures, triggers, and functions in
Microsoft SQL Server.
Design and build scalable, metadata-driven ingestion pipelines for both batch and streaming datasets.Perform data integration and harmonization across multiple structured and unstructured data sources.Implement orchestration, scheduling, exception handling, and log monitoring for robust pipeline management.Collaborate with peers to evaluate and select appropriate tech stack and tools.Work closely with business, consulting, data science, and application development teams todeliver analytical solutions within timelines.
Support performance tuning, troubleshooting, and debugging of Databricks jobs and SQL queries.Work with other Azure services such as Azure Data Factory, Azure Data Lake, SynapseAnalytics, Event Hub, Cosmos DB, Streaming Analytics, and Purview when required.
Support BI and Data Science teams in consuming data securely and in compliance withgovernance Skills & Experience :
5 - 9 years of overall IT experience with at least 4+ years in Big Data Engineering on MicrosoftAzure.
Proficiency in Microsoft SQL Server (T-SQL) stored procedures, indexing, optimization, and performance tuning.Strong experience with Azure Data Factory (ADF), Databricks, ADLS, PySpark, and Azure SQL Database.Working knowledge of Azure Synapse Analytics, Event Hub, Streaming Analytics, Cosmos DB,and Purview.
Proficiency in SQL, Python, and either Scala or Java with debugging and performanceoptimization skills.
Hands-on experience with big data technologies such as Hadoop, Spark, Airflow, NiFi, Kafka,Hive, Neo4J, and Elastic Search.
Strong understanding of file formats such as Delta Lake, Avro, Parquet, JSON, and CSV.Solid background in data modeling, data transformation, and data governance best practices.Experience designing and building REST APIs with practical exposure to Data Lake orLakehouse projects.
Ability to work with large and complex datasets, ensuring data quality, governance, andsecurity standards.
Certifications such as DP-203 : Data Engineering on Microsoft Azure or Databricks CertifiedDeveloper (DE) are a plus.
(ref : hirist.tech)