Technical Lead - Data Engineer

Job Description :

Experience Range : 8-10 years

Location : Bengaluru / Hyderabad

Primary (Must have skills)

• 4+ years of experience in Azure Databricks with PySpark.
• 2+ years of experience in Databricks workflow & Unity catalog.
• 3+ years of experience in ADF (Azure Data Factory).
• 3+ years of experience in ADLS Gen 2.
• 3+ years of experience in Azure SQL.
• 5+ years of experience in Azure Cloud platform.
• 2+ years of experience in Python programming & package builds.

Roles & Responsibilities :

Hands-on experience in designing and building scalable data pipelines using Databricks with PySpark, supporting batch and near-real-time ingestion, transformation, and processing.

Ability to optimize Spark jobs and manage large-scale data processing using RDD and DataFrame APIs.

Experience in building efficient orchestration solutions using Azure Data Factory, with the ability to design modular, reusable workflows using tasks, triggers, and dependencies. Proficiency in using dynamic expressions, parameterized pipelines, custom activities, and triggers.

Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.

Good experience in implementing secure, hierarchical namespace-based data lake storage for structured and semi-structured data, aligned to bronze-silver-gold layers with ADLS Gen2. Knowledge of lifecycle policies, access control (RBAC / ACLs), and folder-level security. Understanding of best practices in file partitioning, retention management, and storage performance optimization.

Capability to develop T-SQL queries, stored procedures, and manage metadata layers on Azure SQL Database.

Experience in writing modular, testable Python code used in data transformations, utility functions, and packaging reusable components.

Familiarity with Python environments, dependency management using pip, Poetry, or Conda, and packaging libraries.

Ability to write clear and concise documentation leveraging industry-standard practices and guidelines during development.

Adoption of code analysers and unit testing frameworks is expected.

Communication Skills

Ability to communicate effectively with internal and customer stakeholders through verbal interactions, emails, and instant messages

Interpersonal Skills

Strong interpersonal skills to build and maintain productive relationships with team members. Ability to provide constructive feedback during code reviews and openness to receiving feedback

Problem-Solving and Analytical Thinking

Ability to troubleshoot and resolve issues efficiently with an analytical mindset

Work Updates

Prior experience working on Agile or Scrum projects with familiarity in tools like Jira or Azure DevOps. Should provide regular updates and demonstrate a proactive and diligent approach in carrying out responsibilities.