Databricks Engineer/Architect

Databricks Engineer/Databricks Architect :

Role Overview :

Define and lead the end-to-end Lakehouse architecture on Databricks ensuring scalability, governance, and performance.

Key Responsibilities :

- Design medallion architecture and canonical models.

- Define standards for ingestion, transformation, orchestration, and CI/CD.

- Architect event-driven ingestion and workflows using EventBridge.

- Ensure governance, lineage, and cost optimization.

- Guide Spark/Delta performance best practices.

- Leverage AI copilots for design and documentation.

Must Have Skills :

- Expertise in Databricks Lakehouse, Delta Lake, Unity Catalog, and Workflows.

- Strong AWS architecture (S3, IAM, EventBridge).

- Deep Spark performance tuning.

- Strong usage of AI coding copilots (GitHub Copilot, ChatGPT) for code generation, debugging, optimization, and documentation.

- Hands-on experience with Databricks Lakehouse (Delta Lake, DLT/Workflows), PySpark/Spark, SQL, Python, AWS S3/Glue, and CI/CD.

- Deep experience in PySpark/Spark transformations and optimization.

- Strong SQL and Python for ETL/ELT.

- Experience building medallion pipelines on Databricks.

Good to Have Skills :

- Healthcare domain exposure including Quality measures, Risk Adjustment (HCC/RAF), and HEDIS.

- Exposure to FHIR/HL7 data models and patient/member/provider entities.

- Experience with event-driven patterns using AWS EventBridge for ingestion/orchestration.

- Delta Live Tables expectations and data quality frameworks.

- HealthLake architecture.

- Streaming architectures.

(ref:hirist.tech)