Key Responsibilities :
- Develop, test, and maintain robust data pipelines using Python to ingest, clean, transform,
and store data from multiple sources.
Design scalable and reliable data workflows that support real-time and batch processing.Collaborate with business stakeholders, data analysts, and other engineers to understanddata requirements and deliver optimal solutions.
Optimize performance of data pipelines and ensure efficient processing and low-latency access to data.Implement best practices in data architecture, data quality, security, and governance.Work with cloud platforms (e.g., AWS, Azure, GCP) for data storage, processing, and pipeline orchestration.Integrate data from APIs, databases, flat files, streaming services, and third-party applications.Monitor, troubleshoot, and support data pipelines in production environments.Maintain proper documentation of data workflows, schema definitions, and metadata.Required Skills & Qualifications :
Bachelors or Masters degree in Computer Science, Engineering, Information Systems, or related field.37 years of experience as a Data Engineer or in a similar role.Strong proficiency in Python for data manipulation, scripting, and automation.Solid understanding of data structures, algorithms, and data modeling concepts.Experience with ETL / ELT development using tools like Airflow, Prefect, or similar.Strong SQL skills and experience working with relational databases (e.g., PostgreSQL, MySQL,SQL Server).
Experience with NoSQL databases (e.g., MongoDB, Cassandra) is a plus.Familiarity with data lake and data warehouse concepts.Hands-on experience with at least one cloud platform : AWS (e.g., S3, Lambda, Redshift, Glue), Azure, or GCP.Experience with containerization and version control tools (Docker, Git).Understanding of CI / CD processes in data engineering environments.Preferred Skills :
Experience with big data tools like Apache Spark, Kafka, Databricks, or Snowflake.Knowledge of data security practices, GDPR, and data compliance standards.Familiarity with data cataloging and lineage tools (e.g., Apache Atlas, Collibra).Exposure to monitoring tools such as Prometheus or Grafana is a plus.Soft Skills :
Excellent problem-solving and analytical thinking.Strong communication and interpersonal skills.Ability to work independently and collaboratively in a fast-paced environment.Attention to detail and a commitment to delivering high-quality workref : hirist.tech)