Job Description:
Experience Range
10+ years
Primary (Must have skills)* - To be Screened by TA Team
10+ years total experience in Data Engineering, Cloud Data Platforms
5+ years hands-on experience with Databricks (PySpark/Scala, Delta Lake, Lakehouse architecture)
4+ years' experience in AWS Glue (Glue ETL jobs, Workflows, Crawlers, Glue Catalog).
3+ years' experience designing data ingestion and transformation pipelines using Spark-based frameworks.
Job Description of Role
Technical Leadership
Drive end-to-end technical solutions, ensuring performance, scalability, and reliability.
Establish coding standards, best practices, and development guidelines.
Conduct code reviews, provide technical mentorship, and resolve complex technical issues.
Data Platform Architecture
Architect and implement large-scale data ingestion, transformation, and storage layers.
Design Lakehouse architecture using Delta Lake, Unity Catalog, Medallion model.
Ensure data quality, lineage, and governance across data assets.
Collaborate with cloud architects to integrate platform services across Azure and AWS.
Solution Development
Develop optimized Spark jobs (PySpark/Scala) with strong performance tuning practices.
Implement ETL/ELT pipelines using Databricks notebooks, Workflows, and Glue Jobs.
Integrate data from multiple sources into ADLS / S3 with appropriate security and metadata management.
Build CI/CD pipelines using Azure DevOps / GitHub Actions.
Cloud Infrastructure & Security
Define IAM roles, managed identities, ACLs, and storage permissions for secure operations.
Work with Terraform/CloudFormation for infrastructure automation (preferred).
Monitor cloud resource utilization and optimize cost.
Cross-Functional Collaboration
Interact with stakeholders, analysts, architects, and business teams to translate requirements into technical designs.
Present architectural recommendations and solution proposals to leadership.
Coordinate with DevOps and Cloud Engineering for deployment and operational readiness.
Technical Lead • Bengaluru / Bangalore, India
L&T Technology Services has futuristic VLSI IPs that address some of the pressing needs of the semiconductor industry.With Talented engineers globally and state-of-the-art R&D infrastructure, we ar... Show more
As the CDP Tech Lead, you will play a pivotal role in shaping and implementing architectural designs for a complete view of our customers within the Customer Data Platform (CDP).You will be respons... Show more
