Job Role :
As a Data Engineer, you will be an integral part of our team, working on data pipelines, data warehousing, and data integration for various analytics and AI use cases.
You will collaborate closely with Delivery Managers, ML Engineers and other stakeholders to ensure seamless data flow and accessibility. Your expertise will be crucial in enabling data-driven decision-making for our clients.
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines and ETL processes to support downstream analytics and AI applications.
- Collaborate with ML Engineers to integrate data solutions into machine learning models and workflows.
- Work closely with clients to understand their data requirements and deliver tailored data solutions.
- Ensure data quality, integrity, and security across all projects.
- Optimize and manage data storage solutions in cloud environments (AWS, Azure, GCP).
- Utilize Databricks for data processing and analytics tasks, leveraging its capabilities to enhance data workflows.
- Monitor the performance of data pipelines, identify bottlenecks or failures, and implement improvements to enhance efficiency and reliability.
- Implement best practices for data engineering, including documentation, testing, and version control.
- Troubleshoot and resolve data-related issues in a timely :
- Bachelors or Masters degree in Computer Science, Information Technology, or a related field. 3 to 5 years of experience as a Data Engineer or in a similar role.
Must Have Skills :
Strong proficiency in SQL, Python, and other relevant programming languages.Hands-on experience with Databricks and its ecosystem.Familiarity with major cloud environments (AWS, Azure, GCP) and their data services.Experience with data warehousing solutions like Snowflake, Redshift, or BigQuery.Experience with SQL, NoSQL and graph databases like PostgreSQL and MongoDB.Understanding of data modeling, data architecture, and database design.Excellent problem-solving skills and attention to detail.Strong communication and collaboration skills.Good to Have Skills :
Experience with real-time data processing frameworks (e.g., Apache Kafka, Spark Streaming).Knowledge of data visualization tools (e.g., Tableau, Power BI).Familiarity with machine learning concepts and frameworks.Experience working in a client-facing role.(ref : hirist.tech)