Job Summary
A senior data engineer leads a team of big data engineers in developing architectures to meet client and team requirements. They provide technical leadership, guidance, and support in setting up the Data Lake on Cloud and implementing standardized Data Model.
Key Responsibilities :
- Build pipelines for various data sources within the organization and social media / public data sources.
- Collaborate with cross-functional teams to source data and make it available for downstream consumption.
- Work with the team to design effective solutions meeting business needs.
- Ensure regular communication with stakeholders, understand concerns, and escalate issues.
- Manage dependencies, timelines, and quality while achieving business value.
- Communicate initiative progress and final handover to business-as-usual processes.
Requirements :
Team player with excellent problem analysis skills.Experienced in Azure Databricks, cloud infrastructure (Azure / AWS), Apache Spark, Hive / Hadoop file formats, and CI / CD tools.Knowledge of NOSQL databases, Big data ETL processing, data modeling, and mapping.