Principal Data Engineer

K&K Talents is an international recruiting agency that has been providing technical resources globally since 1993. This position is with one of our clients in India , who is actively hiring candidates to expand their teams.

Title : Principal Data Engineer

Location : Bangalore, Bhopal, Chennai, Gurugram, Hyderabad, Jaipur, Pune

Mode of Operation : Hybrid

Employment Type : Full-time Permanent

Experience : 10 -14 years

Summary of JD : GCP or AWS, Spark, Pyspark, Python, SQL, Data Warehouse design and development, Databricks and dbt, architecting and implementing workflow management systems, Airflow, Spark applications, CI / CD processes, experience with GitHub Actions

Role :

As a Principal Data Engineer in the Data Warehouse team you will be relied on to lead a team of Sr. Data Engineers, propose Architecture, solve complex situations and data model, interact directly with client for requirement gathering, build a holistic roadmap, identify risks and have mitigation strategy, develop and deliver high-quality features and finish tasks to a high standard. You’ll be part of an emerging team, focussing on building and improving our innovative BNL Data Warehouse offerings applying marketing suites.

Required Experience :

• Overall 10 + years of experience in data engineering with 4+ years leading team.
• 8+ years of hands-on experience in using Python, JVM and Shell Scripting with production experience.
• 5+ years of proven experience on building cloud native data intensive applications.
• Experience on Data Warehouse design and development per the business nature using any of the modern data warehouses.
• Having experience in Databricks and dbt experience is required.
• Highly proficient in architecting and implementing workflow management systems, Airflow is a must.
• Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes).
• Hands-on experience of building and maintaining Spark applications.
• Fundamental understanding of Parquet, Delta Lake and other OTFs file formats.
• Proficiency on an IaC tool such as Terraform, CDK or CloudFormation is a must.
• Experience with observability systems (such as Datadog or Prometheus / Grafana).
• Strong experience on building CI / CD processes, experience with GitHub Actions, ArgoCD or Jenkins is good to have.
• Experience with unit testing and test automation frameworks.
• Propose and review Architecture.
• Solve complex situations and data model.

Soft skills :