Databricks Data Engineer

About the Role

The Databricks Data Engineer will play a crucial role in migrating and optimizing data workflows, ensuring high performance and reliability in our data operations.

Experience- 3 to 6 years

Location - Kolkata / Chennai / Trivandrum

Required Technical Skill Set Databricks, AWS, Python & PySpark

Good to have- Experience in AWS platform (S3, EC2, EMR, MWAA, and Redshift).

Responsibilities

• Collaborate with stakeholders to gather requirements and design solutions using the Databricks platform.
• Migrate Python, Pyspark, MWAA DAG to Databricks Notebook and Workflow.
• Work with BJs Computer operations team and repoint Talend / MWAA Autosys jobs to Databricks workflow.
• Implement medallion architecture in Databricks Enterprise Data Platform with industry best practices and recommend the best framework for Redshift to Delta tables.
• Provide technical expertise and guidance to the team members and stakeholders.
• Contribute to the development of documentation, training materials, and user guides.
• Participate in testing and quality assurance activities to ensure reliability and performance.
• Optimize long-running Databricks workflows.
• Development experience in Python, Pyspark.
• Hands-on experience in writing complex SQLs and performance optimization of very complex SQL.