Data Engineer/Architect

Role : Data Architect / Engineers Databricks - Hinjewadi, Pune (Hybrid) or Remote

About the Role :

We are looking for a hands-on Data Architect/Engineer with deep expertise in Databricks to design, build, and optimise enterprise-scale data platforms.

You will own the end-to-end data engineering lifecycle from ingestion and transformation to serving while ensuring reliability, scalability, and governance across our lakehouse architecture.

You will collaborate closely with data engineers, analytics engineers, and product teams to translate business requirements into robust, reusable data solutions on the Databricks Lakehouse Platform.

Key Responsibilities :

Data Architecture & Design :

- Design and maintain the organisation's lakehouse architecture using Databricks and Delta Lake.

- Define data modelling standards (dimensional, Data Vault 2.0, or medallion architecture) across Bronze, Silver, and Gold layers.

- Architect scalable ingestion frameworks using structured and unstructured data sources (Kafka, JDBC, REST APIs, cloud storage).

- Own schema evolution strategy and ensure backward-compatibility across data assets.

Pipeline Development & Optimisation :

- Build and maintain production-grade ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows.

- Implement Delta Live Tables (DLT) for declarative, auto-scaling pipeline development.

- Optimise Spark jobs for performance partitioning, Z-ordering, caching, and cluster right-sizing.

- Establish CI/CD practices for data pipelines using tools such as GitHub Actions, Azure DevOps, or Databricks Asset Bundles.

Data Governance & Quality :

- Implement Unity Catalog for data discovery, lineage tracking, fine-grained access control, and compliance.

- Define and enforce data quality rules using Great Expectations, DLT expectations, or equivalent frameworks.

- Work with data governance teams to document metadata, business glossary, and data contracts.

Platform & Infrastructure :

- Manage Databricks workspace configuration : clusters, pools, secrets, and access policies.

- Collaborate with cloud and DevOps teams on infrastructure-as-code (Terraform) for Databricks on Azure / AWS / GCP.

- Monitor platform health, SLAs, and cost using Databricks system tables and cloud-native monitoring tools.

Collaboration & Mentorship :

- Partner with data consumers (analysts, data scientists, ML engineers) to define SLAs and publish clean, well-documented data products.

- Review code and provide architectural guidance to junior engineers.

- Contribute to and champion internal data engineering best practices, runbooks, and documentation.

Required Skills & Experience :

Core Databricks & Spark :

- 5+ years of hands-on experience with Databricks (Unified Data Analytics Platform).

- Strong proficiency in PySpark and Spark SQL for large-scale data transformation.

- Deep knowledge of Delta Lake ACID transactions, time travel, OPTIMIZE, VACUUM.

- Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT).

- Familiarity with Unity Catalog and Databricks governance features.

Data Engineering Fundamentals :

- Solid understanding of data modelling paradigms : dimensional modelling, Data Vault, or medallion architecture.

- Experience designing and operating streaming pipelines (Structured Streaming, Kafka, Event Hubs, or Kinesis).

- Proficiency in SQL; experience with dbt is a strong plus.

- Hands-on experience with cloud platforms : Azure (ADLS, ADF), AWS (S3, Glue), or GCP (BigQuery, GCS).

Software Engineering Practices :

- Version control with Git; experience with branching strategies and code review workflows.

- Ability to write testable, modular pipeline code with unit and integration tests.

- Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform preferred).

- Data Architecture, Databricks, PySpark, SQL, ETL/ELT, Kafka with any cloud experience is totally mandatory.

(ref:hirist.tech)