Data Engineers

Job Summary

We are seeking an experienced Data Engineering Lead / Architect with strong expertise in building enterprise-scale Lakehouse solutions using Azure, Databricks, PySpark, and modern cloud-native data engineering practices.

The ideal candidate should possess deep hands-on expertise in data engineering, semantic data modeling, metadata-driven architectures, and AI-assisted engineering approaches. This role requires a highly solution-oriented engineer who can go beyond pipeline development and contribute toward scalable architecture design, performance optimization, governance, automation, and business enablement.

The candidate should demonstrate strong ownership, problem-solving ability, adaptability to evolving priorities, and a continuous learning mindset toward emerging Azure and Databricks capabilities.

Key Responsibilities

Lakehouse Architecture & Engineering

• Design and implement scalable end-to-end Lakehouse solutions using Azure, Databricks, Python, and PySpark.
• Develop enterprise-grade data pipelines sourcing data from ERP, CRM, and field-device/on-field operational applications.
• Implement Medallion Architecture (Bronze, Silver, Gold) with focus on scalability, governance, and maintainability.
• Design robust Silver Layer semantic data models aligned with enterprise common data models.

Metadata & Semantic Layer Engineering

• Develop and maintain semantic data layers including:
  • Data Catalogs
  • Data Dictionaries
  • Business Glossaries
• Implement metadata-driven transformation and engineering frameworks.
• Manage metadata lifecycle, lineage, and governance processes.
• Handle schema drift and schema evolution through regenerative metadata and automated data dictionary approaches.

AI-Driven Automation & Productivity Engineering

• Design AI-assisted solutions for extracting and generating metadata, data dictionaries, and semantic mappings using:
  • Source datasets
  • Application code
  • Defined business rules
• Develop Databricks-based automation frameworks capable of dynamically generating transformation notebooks using metadata definitions.
• Leverage AI capabilities to improve engineering productivity, automation, governance, and operational efficiency.
• Continuously explore and adopt emerging Azure and Databricks AI capabilities.

Data Processing, Modeling & Optimization

• Develop scalable data transformations, functions, procedures, and reusable business computation logic.
• Build curated aggregates, summaries, snapshots, and analytics-ready datasets for downstream consumption.
• Optimize distributed processing workloads for:
  • Performance
  • Scalability
  • Cost efficiency
• Implement data quality frameworks including validation, reconciliation, observability, and governance controls.

Self-Service BI & Consumption Enablement

• Enable governed Self-Service BI capabilities using Databricks Genie/API/embed tools.
• Securely expose semantic/silver-layer datasets to business and functional stakeholders.
• Capture and analyze usage telemetry/logs to continuously enhance data platform usability and adoption.

Required Skills & Expertise

Core Technical Skills

• Strong expertise in:
  • SQL
  • Data Modeling
  • Python
  • PySpark
  • Azure
  • Databricks
• Strong understanding of:
  • Medallion Architecture
  • Lakehouse Architecture
  • Metadata-driven engineering
  • Distributed data processing
  • Semantic data layers
• Experience integrating data from:
  • ERP systems
  • CRM platforms
  • Field-device / operational applications
• Experience with Databricks system tables and platform observability.

AI & Advanced Engineering Capabilities

• Exposure to AI/ML-enabled data engineering solutions.
• Experience leveraging AI capabilities for engineering productivity and automation.
• Understanding of metadata automation and AI-assisted semantic enrichment approaches preferred.

Candidate Profile

We are looking for candidates who:

• Possess strong analytical and problem-solving skills with ability to propose effective and scalable solutions.
• Are flexible and adaptable to evolving business priorities and changing technical landscapes.
• Demonstrate creativity and engineering innovation in solving enterprise data challenges.
• Have strong communication and stakeholder management capabilities.
• Are self-driven learners with continuous upskilling mindset on latest Azure and Databricks technologies.
• Can operate beyond maintenance/support activities and contribute meaningfully toward solution architecture, optimization, and engineering transformation.

Skills Required
Data Modeling, Pyspark, Sql, Azure, Databricks, Python