Data Engineer - Python / SQL

Description

Key Responsibilities :

• Data Pipeline Development : Design, build, and optimize scalable, secure, and reliable data pipelines to ingest, process, and transform large volumes of structured and unstructured data.
• Data Architecture : Architect and maintain data storage solutions, including data lakes, data warehouses, and databases, ensuring performance, scalability, and cost-efficiency.
• Data Integration : Integrate data from diverse sources, including APIs, third-party systems, and streaming platforms, ensuring data quality and consistency.
• Performance Optimization : Monitor and optimize data systems for performance, scalability, and cost, implementing best practices for partitioning, indexing, and caching.
• Collaboration : Work closely with data scientists, analysts, and software engineers to understand data needs and deliver solutions that enable advanced analytics, machine learning, and reporting.
• Data Governance : Implement data governance policies, ensuring compliance with data security, privacy regulations (e.g., GDPR, CCPA), and internal standards.
• Automation : Develop automated processes for data ingestion, transformation, and validation to improve efficiency and reduce manual intervention.
• Mentorship : Guide and mentor junior data engineers, fostering a culture of technical excellence and continuous learning.
• Troubleshooting : Diagnose and resolve complex data-related issues, ensuring high availability and reliability of data systems.

Required Qualifications

Technical Skills

Soft Skills

(ref : hirist.tech)

Skills Required

BigQuery, Hadoop, Kafka, Redshift, Sql, Gcp, Docker, snowflake , Spark, Kubernetes, Python, Aws