Celebal Technologies - AWS Data Engineer

DATA ENGINEER (Databricks & AWS)

Overview :

As a Data Engineer, you will work with multiple teams to deliver solutions on the AWS Cloud using core cloud data engineering tools such as Databricks on AWS, AWS Glue, Amazon Redshift, Athena, and other Big Data-related technologies. This role focuses on building the next generation of application-level data platforms and improving recent implementations. Hands-on experience with Apache Spark (PySpark, SparkSQL), Delta Lake, Iceberg, and Databricks is essential.

Responsibilities :

• Define, design, develop, and test software components / applications using AWS-native data services : Databricks on AWS, AWS Glue, Amazon S3, Amazon Redshift, Athena, AWS Lambda, Secrets Manager
• Build and maintain ETL / ELT pipelines for both batch and streaming data.
• Work with structured and unstructured datasets at scale.
• Apply Data Modeling principles and advanced SQL techniques.
• Implement and manage pipelines using Apache Spark (PySpark, SparkSQL) and Delta Lake / Iceberg formats.
• Collaborate with product teams to understand requirements and deliver optimized data solutions.
• Utilize CI / CD pipelines with DBX and AWS for continuous delivery and deployment of Databricks code.
• Work independently with minimal supervision and strong ownership of deliverables.

Must Have :

1. Apache Spark (PySpark, SparkSQL)

2. Delta Lake / Iceberg formats

3. Databricks on AWS

4. AWS Glue, Amazon Athena, Amazon Redshift

Good To Have :

Key Skills :

(ref : hirist.tech)