Senior Data Engineer

Role Overview:

As a Senior Data Engineer based in Hyderabad, you will serve as a cornerstone of our data infrastructure team, architecting and maintaining robust pipelines that transform raw data into actionable business intelligence. You will collaborate closely with cross-functional teams, including data scientists, product managers, and business stakeholders, to translate complex requirements into scalable technical solutions. By leveraging cutting-edge cloud technologies, you will play a pivotal role in optimizing our data ecosystem, directly influencing strategic decision-making and enhancing the performance of our data-driven products for our global user base.

Key Responsibilities:

- Design and implement high-performance ETL/ELT pipelines using PySpark and Azure Databricks to ensure seamless data ingestion and transformation for downstream analytics.

- Architect real-time data streaming solutions utilizing Kafka to provide low-latency insights that empower stakeholders to make rapid, data-backed decisions.

- Manage and optimize Snowflake DB environments to ensure high availability, cost-efficiency, and superior query performance for enterprise-wide reporting.

- Integrate advanced AI and GPT-based models into existing data workflows to automate data enrichment and improve the quality of predictive insights.

- Mentor junior engineers and conduct rigorous code reviews to maintain high engineering standards and foster a culture of technical excellence within the team.

- Collaborate with infrastructure teams to automate deployment processes, ensuring the reliability and scalability of our data platforms as business demands grow.

Required Skillset:

- Demonstrated expertise in building complex data architectures using Python and SQL, with a deep understanding of data modeling and warehousing principles.

- Proven ability to manage large-scale data processing tasks in Azure Databricks and Snowflake environments, ensuring data integrity and security across all pipelines.

- Strong proficiency in developing event-driven architectures with Kafka, coupled with the ability to troubleshoot and resolve complex distributed system issues.

- Exceptional communication skills, with the ability to articulate technical concepts to non-technical stakeholders and influence project direction through data-driven storytelling.

- A proactive mindset toward learning, with a demonstrated interest in applying emerging technologies like GPT to solve traditional data engineering challenges.

- A Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field, reflecting a strong foundation in algorithmic thinking.

- Ability to thrive in a hybrid work environment in Hyderabad, balancing independent problem-solving with active participation in collaborative team sprints.

- A minimum of 5 to 8 years of hands-on experience in data engineering roles, showcasing a track record of delivering impactful data products in high-growth environments.

(ref:hirist.tech)