Data Pipeline Architect

About the Role

As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics.

Responsibilities

• Develop and maintain scalable ETL / ELT pipelines using Python, Airflow, and dbt
• Build and optimize real-time streaming pipelines using Kafka, RabbitMQ, Spark and event-driven microservices
• Leverage AI coding agents (Claude Code, Cursor AI, GitHub Copilot) to accelerate data pipeline development, data transformation, and optimization
• Design and manage data storage solutions with PostgreSQL, ClickHouse and Elasticsearch
• Define and implement data modeling best practices for efficient and scalable data storage.
• Ensure high availability, performance, and security of data infrastructure on AWS cloud
• Design and implement BI-ready datasets in ClickHouse optimized for OLAP workloads and reporting use cases.
• Develop and manage denormalized schemas and data models to support analytics and amCharts-based reporting.
• Optimize query performance, partitioning, and indexing strategies in ClickHouse to handle large-scale analytical workloads.
• Implement best practices for data quality, validation, and observability
• Work on data ingestion using Airbyte or other connector frameworks
• Implement data transformation pipelines supporting analytical and ML workloads
• Collaborate with cloud and DevOps teams for containerized deployments (Docker, Kubernetes, AWS)
• Support integration of ML workflows and model APIs into production pipelines

Qualifications

Required Skills

Preferred Skills