Data Engineer

Job Title : Databricks SQL Engineer (with Pharma / Life Sciences background)

About the Role :

We are seeking a highly skilled Databricks SQL Engineer with Pharma background to join our Data Engineering team. The ideal candidate will have strong expertise in Databricks, Pyspark, SQL, Spark SQL, Delta Lake, and data lakehouse architectures, and will be responsible for building scalable and efficient data models, queries, and pipelines that power analytical insights and business decisions.

Key Responsibilities :

Design, develop, and optimize SQL-based data models and queries in Databricks for large-scale datasets.

Work with Delta Lake, Unity Catalog, and Databricks Notebooks to manage and secure data assets.

Implement and maintain ETL / ELT pipelines using Databricks workflows and SQL.

Develop documentation and data dictionaries to promote self-service analytics.

Ensure data quality and consistency through testing, validation, and monitoring.

Optimize data lakehouse architecture for cost and performance, leveraging Spark SQL best practices.

Support CI / CD practices for deploying notebooks and SQL jobs to production.

Required Qualifications :

5+ years of experience working with Databricks SQL, Pyspark, Spark SQL, or similar distributed SQL engines.

Experience in the Pharma Domain

Strong experience with Delta Lake, structured streaming, and data lakehouse paradigms.

Proficiency in SQL development, query tuning, and working with large datasets.

Experience with version control tools (Git) and collaborative development practices.

Experience working in cloud environments (AWS, Azure, or GCP), preferably with Azure Databricks or Databricks on AWS.

Understanding of data governance, access control, and tools like Unity Catalog or Purview.

Familiarity with CI / CD for data pipelines and automation tools like Databricks Repos, Terraform, or dbx.

Exposure to Python or PySpark for scripting and data manipulation.