Data Pipeline Specialist

Data Engineer Job Description

About Sirius AI

Sirius AI is a US headquartered AI Consulting services and products company with operations in India. Sirius AI focuses on Financial Services enterprises and solutions / services delivered across multiple geographies. We are an innovation-driven AI and data driven consulting services firm with a high focus on delivering business outcomes for our clients. The Sirius AI team is a mix of talent from a diverse background of Consulting, Industry and Product companies which bring together their Data, Platform, Ecosystem, Technology, Project management and AI skills to develop new solutions and deliver client engagements.

Key Responsibilities :

• Data Architecture & Management : Design and implement scalable, cloud agnostic Data Lake, Data LakeHouse, Data Mesh and Data Fabric architectures to efficiently store, process, and manage structured and unstructured data from various sources.
• Data Pipeline Development : Design, develop, and maintain robust data pipelines to ingest, process, and transform data from multiple sources into usable formats for analytics and reporting using services like AWS Glue, Azure Data Factory, GCP Dataflow, Apache Spark, or Apache Airflow.
• Data Integration and ETL : Develop and optimize Extract, Transform, Load (ETL) and ELT processes to integrate disparate data sources into the data lake, ensuring high data quality, consistency, and reliability across multiple cloud platforms.
• Cloud-Agnostic Data Engineering : Develop data solutions that are cloud agnostic, leveraging open-source technologies like Apache Spark, Delta Lake, Presto, and Kubernetes, ensuring compatibility across AWS, Azure, and GCP.
• Big Data Processing & Analytics : Utilize big data technologies such as Apache Spark, Hive, and Presto for distributed computing, enabling large-scale data transformations and analytics.
• Data Governance and Security : Implement robust data governance policies, security frameworks, and compliance controls, including role-based access control (RBAC), encryption, and monitoring to meet industry standards (GDPR, HIPAA, PCI-DSS).
• DevOps Integration for Data Platforms : Leverage cloud-agnostic DevOps tools and practices for source control, build automation, release management, and Infrastructure as Code (IaC) to streamline the development, deployment, and management of data lake and data architecture solutions across multiple cloud providers. Solutions should support CI / CD pipelines, automated testing, and scalable data workflows.
• Continuous Integration and Deployment (CI / CD) : Establish automated CI / CD pipelines to streamline deployment, testing, and monitoring of data infrastructure and workflows.
• Performance Optimization : Optimize data workflows and query performance using indexing, caching, and partitioning strategies to improve efficiency and cost-effectiveness.
• Monitoring and Troubleshooting : Implement observability solutions using tools like Prometheus, Grafana, or cloud-native monitoring services to proactively detect and resolve data pipeline issues.
• Collaboration and Documentation : Work with cross-functional teams, including data scientists, analysts, and business stakeholders, to design and implement scalable data solutions. Maintain comprehensive documentation of data architectures, processes, and best practices.

Job Qualifications :