Databricks Developer

Position Description :

We are seeking a skilled Databricks Developer to join our Data Development team. Reporting to the Team Lead, Data Development, the Databricks Developer implements robust data pipelines using Apache Spark on Databricks, supports advanced data transformation, and enables scalable data products that serve enterprise analytics and reporting needs. Youll work closely with data engineers and analysts to ensure high-performance, reliable data pipelines and quality outputs.

This is a hands-on development role focused on engineering scalable, maintainable, and optimized data flows in a modern cloud-based environment.

Job Responsibilities :

• Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
• Develop efficient ETL / ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks / jobs.
• Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
• Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
• Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
• Apply best practices in data modeling and data warehousing concepts.
• Perform data validation and quality checks.
• Work with various data formats, including JSON, Parquet, and Avro.
• Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, and SQL Server.
• Create parameterized and dynamic ADF pipelines, and trigger Databricks notebooks from ADF.
• Collaborate closely with Data Scientists, Data Analysts, Business Analysts, and Data Architects to deliver trusted, high-quality datasets.
• Contribute to data governance, metadata documentation, and ensure adherence to data quality standards.
• Use version control tools (e.g., Git) and CI / CD pipelines to manage code deployment and workflow changes.
• Develop real-time and batch processing pipelines for streaming data sources such as MQTT, Kafka, and Event Hub.

Requirements :

Additional Desired Qualifications :

Job Demands and / or Physical Requirements :

As Seaspan is a global company, occasional work outside of regular office hours may be required.

(ref : hirist.tech)