DataOps Engineer

We are seeking a highly skilled and experienced DataOps Engineer to join our dynamic team. The ideal candidate will have a strong background in DataOps practices with a focus on AWS and Azure DevOps, Databricks setup and management, PostgreSQL administration, Docker management, CI / CD setup, and Azure / AWS infrastructure management. This role is critical in ensuring the seamless integration and deployment of data infrastructure, enabling efficient and reliable data operations.

• Responsibilities : DevOps Management :
• Design, implement, and manage DevOps pipelines for automated build, test, and deployment processes with tools like Git, Azure DevOps, Jenkins, GitActions etc., Mostly for data workloads.
• Build and manage Databricks, snowflake, Kafka, and other cloud native data services / tools setup.
• Collaborate with development teams to integrate code changes and ensure seamless delivery to production.
• Databricks Setup and Management :
• Set up and manage Azure Databricks environments for large-scale data processing and analytics.
• Optimize Databricks clusters and manage costs while ensuring high availability and performance.
• PostgreSQL Administration :
• Administer PostgreSQL databases, ensuring their optimal performance, security, and reliability.
• Perform routine database maintenance tasks such as backups, restoration, and tuning queries for performance.
• Docker Management :
• Develop and manage Docker containers to ensure consistency across development, testing, and production environments.
• Monitor containerized applications for performance and resolve any issues related to container orchestration.
• CI / CD Setup :
• Design and implement CI / CD pipelines to automate software deployments and data pipelines.
• Ensure that the CI / CD pipelines are scalable, secure, and capable of handling large volumes of data.
• Azure / AWS Infrastructure Management :
• Manage Azure infrastructure components such as Virtual Networks, Storage Accounts, and Resource Groups.
• Monitor and optimize the performance of the Azure environment, ensuring scalability and reliability.
• Implement security best practices across Azure services to protect data and applications.
• Collaboration and Communication :
• Work closely with data engineers, software developers, and IT teams to integrate DataOps processes across the organization.
• Provide technical guidance and mentorship to junior team members on DataOps best practices.
• Monitoring and Optimization :
• Implement monitoring solutions to track the health and performance of data pipelines and infrastructure.
• Continuously optimize processes and infrastructure for cost-effectiveness and efficiency.

Requirements :

Skills Required

Docker, Postgresql, Databricks, Azure Devops, Aws