Architect - Data Engineering

Role Overview

Architect- Data Engineering is responsible for overseeing the strategy, design, development, and management of data infrastructure and pipelines within an organization. This role involves a strong technical leadership and collaboration with other teams to ensure the efficient collection, storage, processing, and analysis of large datasets. The Architect typically Leads a team of data engineers, associate data architects, and analysts, ensuring that data workflows are scalable, reliable, and meet the business's requirements.

Responsibilities

• Lead the design, development, and maintenance of data pipelines and ETL processes architect and implement scalable data solutions using Databricks and AWS.
• Optimize data storage and retrieval systems using Rockset, Clickhouse, and CrateDB.
• Develop and maintain data APIs using FastAPI.
• Orchestrate and automate data workflows using Airflow.
• Collaborate with data scientists and analysts to support their data needs.
• Ensure data quality, security, and compliance across all data systems.
• Mentor junior data engineers and promote best practices in data engineering.
• Evaluate and implement new data technologies to improve the data infrastructure.
• Participate in cross-functional projects and provide technical leadership.
• Manage and optimize data storage solutions using AWS S3, implementing best practices for data lakes and data warehouses.
• Implement and manage Databricks Unity Catalog for centralized data governance and access control across the organization.

Qualifications

Preferred Qualifications

Tech Stack

Databricks, Python, PySpark, SQL, Airflow, FastAPI, AWS (S3, IAM, ECR, Lambda), Rockset, Clickhouse, CrateDB

We Offer

This role presents an exciting opportunity for a motivated individual to contribute to the development of cutting-edge solutions while advancing their career in a dynamic and collaborative environment.

(ref : hirist.tech)

Skills Required

Airflow, Git, Pyspark, Databricks, FastAPI, Sql, Python, Aws