Infrastructure Consultant

Senior Cloud Infrastructure Engineer for Data Platform

JD for DevOps requirement in Data Lake team. Pls share suitable profiles.

Job Title : Senior Cloud Infrastructure Engineer for Data Platform- Periscope

The ideal candidate will play a critical role in designing, implementing, and maintaining cloud infrastructure and CI / CD pipelines to support scalable, secure, and efficient data and analytics solutions. This role requires a strong understanding of cloud-native technologies, DevOps best practices, and hands-on experience with Azure and Databricks.

Key Responsibilities

1. Cloud Infrastructure Design & Management

o Architect, deploy, and manage scalable and secure cloud infrastructure on Microsoft Azure.

o Implement best practices for Azure Resource Management, including resource groups, virtual networks, and storage accounts.

o Optimize cloud costs and ensure high availability and disaster recovery for critical systems.

2. Databricks Platform Management

o Set up, configure, and maintain Databricks workspaces for data engineering, machine learning, and analytics workloads.

o Automate cluster management, job scheduling, and monitoring within Databricks.

o Collaborate with data teams to optimize Databricks performance and ensure seamless integration with Azure services.

3. CI / CD Pipeline Development

o Design and implement CI / CD pipelines for deploying infrastructure, applications, and data workflows using tools like Azure DevOps, GitHub Actions, or similar.

o Automate testing, deployment, and monitoring processes to ensure rapid and reliable delivery of updates.

4. Monitoring & Incident Management

o Implement monitoring and alerting solutions using tools like Dynatrace, Azure Monitor, Log Analytics, and Databricks metrics.

o Troubleshoot and resolve infrastructure and application issues, ensuring minimal downtime.

5. Security & Compliance

o Enforce security best practices, including identity and access management (IAM), encryption, and network security.

o Ensure compliance with organizational and regulatory standards for data protection and cloud operations.

6. Collaboration & Documentation

o Work closely with cross-functional teams, including data engineers, software developers, and business stakeholders, to align infrastructure with business needs.

o Maintain comprehensive documentation for infrastructure, processes, and configurations.

Required Qualifications

• Education : Bachelor’s degree in Computer Science, Engineering, or a related field.
• Experience :

o 6+ years of experience in DevOps or Cloud Engineering roles.

o Proven expertise in Microsoft Azure services, including Azure Data Lake, Azure Databricks, Azure Data Factory (ADF), Azure Functions, Azure Kubernetes Service (AKS), and Azure Active Directory.

o Hands-on experience with Databricks for data engineering and analytics.

o Proficiency in Infrastructure as Code (IaC) tools like Terraform, ARM templates, or Bicep.

o Strong scripting skills in Python, or Bash.

o Experience with containerization and orchestration tools like Docker and Kubernetes.

o Familiarity with version control systems (e.g., Git) and CI / CD tools (e.g., Azure DevOps, GitHub Actions).

o Strong problem-solving and analytical skills.

o Excellent communication and collaboration abilities.