Key Responsibilities :
Data Architecture & Pipeline Development :
- Design, implement, and maintain scalable, reliable, and efficient data pipelines.
- Build and optimize data models for structured, semi-structured, and unstructured datasets.
- Develop real-time and batch processing solutions.
Data Integration & Management :
Integrate data from multiple internal and external sources into a unified data platform.Ensure data quality, consistency, integrity, and governance across all systems.Implement best practices for data ingestion, transformation (ETL / ELT), and storage.Performance Optimization :
Optimize data workflows for performance, scalability, and cost-effectiveness.Monitor and troubleshoot issues with data pipelines and cloud & Leadership :Partner with data scientists, analysts, and product managers to deliver data solutions.Provide technical mentorship to junior engineers.Drive the adoption of modern data engineering practices, tools, and frameworks.Security & Compliance :
Implement data security standards, compliance policies, and governance protocols.Ensure adherence to GDPR, HIPAA, or other relevant regulatory requirements.Required Qualifications :
Bachelors or Masters degree in Computer Science, Engineering, Information Systems, or related field.5 - 8 years of experience in data engineering or related roles.Strong proficiency in SQL and programming languages (Python, Scala, or Java).Hands-on experience with big data frameworks (e.g., Spark, Hadoop, Kafka, Flink).Expertise in data modeling, warehousing, and ETL / ELT development.Solid experience with cloud platforms (AWS, Azure, GCP) and data services like Redshift, BigQuery, Snowflake, or Databricks.Strong understanding of data governance, security, and compliance.Preferred Qualifications :
Experience with containerization & orchestration tools (Docker, Kubernetes, Airflow).Familiarity with real-time streaming technologies (Kafka, Kinesis, Pulsar).Knowledge of machine learning pipelines and integration with AI / ML workflows.Strong problem-solving and analytical skills with attention to detail.Excellent communication skills with the ability to work across technical and business teams.(ref : hirist.tech)