Lead/Principal Data Architect

Job Description Responsibilities :

Architecture & Strategy :

- Design end-to-end scalable, secure, and highly available data architectures leveraging modern cloud data ecosystems (Databricks and Snowflake).

- Establish data governance frameworks and strategic direction for enterprise data platforms.

Pipeline Engineering :

- Architect, optimize, and oversee the deployment of reliable streaming and batch data pipelines (ETL/ELT) to process complex, large-scale datasets.

- Ensure fault-tolerance, performance optimization, and cost-efficiency across all pipeline implementations.

Cloud Architecture :

- Architect and deploy scalable enterprise data platform components natively within the AWS ecosystem, ensuring tight integration with core security, IAM, and networking protocols.

- Design cloud-native solutions that maximize performance and minimize operational overhead.

API Ingestion & Orchestration :

- Design and implement robust data ingestion frameworks leveraging Databricks APIs and external REST/GraphQL APIs for automated workflows, platform orchestration, and data delivery.

- Build scalable ingestion solutions that support diverse data sources and formats.

Real-time Processing :

- Design and implement robust frameworks for real-time data ingestion and processing to solve business-critical, low-latency use cases.

- Architect streaming solutions that deliver insights with minimal latency while maintaining data quality and reliability.

Hybrid Data Modelling :

- Harmonize traditional relational data warehousing patterns (Kimball/Inmon, Star/Snowflake schemas) with unstructured and semi-structured modern paradigms.

- Create flexible, scalable data models that support diverse analytical and operational use cases.

Technical Leadership :

- Act as a core problem-solver for complex data bottlenecks and performance challenges.

- Provide technical governance, establish best practices, and mentor engineering teams on data architecture and optimization strategies. Drive technical excellence across the organization.

AI Integration :

- Collaborate with Data Science and AI teams to architect data layers that seamlessly support LLMs, Machine Learning pipelines, and advanced analytics solutions.

- Design feature stores and data infrastructure optimized for AI/ML workloads.

Required Qualifications :

Experience :

- 10+ years of progressive experience in Data Engineering, Data Warehousing, and Data Architecture.

- Proven track record of designing and implementing enterprise-scale data platforms.

- Demonstrated experience leading technical teams and influencing architectural decisions.

Educational Background :

- Bachelor of Engineering (BE) / B.Tech in Computer Science or related field.

- OR Master of Computer Applications (MCA) / M.Tech.

Mandatory Technical Skills :

- Databricks : Deep hands-on expertise with Databricks Lakehouse platform, Delta Lake, Unity Catalog, and Spark performance optimization.

- Snowflake : Strong experience in architectural design, performance tuning, query optimization, and cost-optimization strategies.

- Python : Advanced proficiency for data pipeline development and scripting.

- Scala : Solid experience for Spark-based distributed computing.

- SQL : Expert-level SQL skills for complex query optimization and data modeling.

- Structured Streaming: Hands-on experience with Apache Spark Structured Streaming for real-time data processing.

- Apache Kafka : Proven expertise in designing and implementing Kafka-based data streaming architectures.

- Flink/AWS Kinesis : Experience with Apache Flink or AWS Kinesis for stream processing and real-time analytics.

Technical Expertise :

- Data Pipeline Excellence : Exceptional expertise in designing distributed, fault-tolerant data pipelines using Python, Scala, or SQL.

- Real-time Systems : Proven track record with stream processing technologies for real-time, low-latency use cases.

- Polyglot Persistence : Solid foundation in traditional Data Warehousing and relational database management systems (RDBMS); hands-on experience with NoSQL ecosystems.

- Cloud Platforms : Deep understanding of AWS services including Lambda, S3, EC2, IAM, and VPC configurations.

- Data Governance : Experience implementing data governance, lineage tracking, and metadata management solutions.

Soft Skills & Competencies :

- Problem-Solving : Elite analytical mindset with a proven track record of troubleshooting complex distributed systems and resolving performance degradation issues.

- Communication : Ability to articulate complex technical architectures clearly to both engineering teams and non-technical business stakeholders.

- Leadership : Natural ability to mentor, guide, and elevate technical teams; strong influence without authority.

- Collaboration : Proven ability to work cross-functionally with data scientists, engineers, and business stakeholders.

- Attention to Detail : Meticulous approach to system design, documentation, and quality assurance.

Preferred / Good-to-Have Qualifications :

- AI/ML Data Readiness : Exposure to architecting data solutions tailored for AI, such as vector databases (e.g., Pinecone, Milvus), feature stores, or building data pipelines for generative AI/LLM applications.

- Certifications : Databricks Certified Data Architect, Snowflake Certified Advanced Architect, or AWS Solutions Architect certifications.

- Experience with data quality frameworks and tools (e.g., Great Expectations, dbt).

- Knowledge of containerization technologies (Docker, Kubernetes) and CI/CD pipelines.

- Experience with data cataloging and metadata management platforms.

- Exposure to graph databases and advanced NoSQL technologies.

- Experience with cost optimization strategies for cloud data platforms.

(ref:hirist.tech)