Team Lead - Data Group / 2
ConfidentialBengaluru / Bangalore21 days ago
Job descriptionTechnical Leadership Lead and mentor a team of data engineers, fostering best practices in coding, design, and delivery. Drive the adoption of modern data engineering frameworks, tools, and methodologies to ensure high-quality and scalable solutions. Translate complex business requirements into effective data pipelines, architectures, and workflows. Data Pipeline Development Architect, develop, and optimize scalable ETL / ELT pipelines using Apache Spark, Hive, AWS Glue, and Trino. Handle complex data workflows across structured and unstructured data sources, ensuring performance and cost-efficiency. Develop real-time and batch processing systems to support business intelligence, analytics, and machine learning applications. Cloud & Infrastructure Management Build and maintain cloud-based data solutions using AWS services like S3, Athena, Redshift, EMR, DynamoDB, and Lambda. Design and implement federated query capabilities using Trino for diverse data sources. Manage Hive Metastore for schema and metadata management in data lakes. Performance Optimization Optimize Apache Spark jobs and Hive queries for performance, ensuring efficient resource utilization and minimal latency. Implement caching and indexing strategies to accelerate query performance in Trino. Continuously monitor and improve system performance through diagnostics and tuning. Collaboration & Stakeholder Engagement Work closely with data scientists, analysts, and business teams to understand requirements and deliver actionable insights. Ensure that data infrastructure aligns with organizational goals and compliance standards. Data Governance & Quality Establish and enforce data quality standards, governance practices, and monitoring processes. Ensure data security, privacy, and compliance with regulatory frameworks. Innovation & Continuous Learning Stay ahead of industry trends, emerging technologies, and best practices in data engineering. Proactively identify and implement improvements in data architecture and processes. Required Technical Expertise Advanced proficiency with Apache Spark (core, SQL, streaming) for large-scale data processing. Strong expertise in Hive for querying and managing structured data in data lakes. In-depth knowledge of Trino (Presto) for federated querying and high-performance SQL execution. Solid programming skills in Python with frameworks like PySpark and Pandas. Hands-on experience with AWS Glue, including Glue ETL jobs, Glue Data Catalog, and Glue Crawlers. Deep understanding of data formats such as Parquet, ORC, Avro, and their use cases. Cloud Proficiency Expertise in AWS services, including S3, Redshift, Athena, EMR, DynamoDB, and IAM. Experience designing scalable and cost-efficient cloud-based data solutions. Performance Tuning Strong ability to optimize Apache Spark jobs, Hive queries, and Trino workloads for distributed environments. Experience with advanced techniques like partitioning, bucketing, and query plan optimization. Leadership & Collaboration Proven experience leading and mentoring data engineering teams. Strong communication skills, with the ability to interact with technical and non-technical stakeholders effectively. Education & Experience Bachelors or Masters degree in Computer Science, Data Engineering, or a related field. 8+ years of experience in data engineering with a minimum of 2 years in a leadership role. 8+ years of experience in building data pipelines from scratch in large data volume environments AWS certifications, such as AWS Certified Data Analytics or AWS Certified Solutions Architect. Experience with Kafka or Kinesis for real-time data streaming would be a plus. Familiarity with containerization tools like Docker and orchestration platforms like Kubernetes. Knowledge of CI / CD pipelines and DevOps practices for data engineering. Prior experience with data lake architectures and integrating ML workflows. Mandatory Key Skills - Spark,Python,Hive,ETL,Redshift,Athenda,IAM,S3,Kafka,AWS.
Position Overview :
- We seek a highly skilled and experienced Data Engineering Lead to join our team. This role demands deep technical expertise in Apache Spark, Hive, Trino (formerly Presto), Python, AWS Glue, and the broader AWS ecosystem. The ideal candidate will possess strong hands-on skills and the ability to design and implement scalable data solutions, optimise performance, and lead a high-performing team to deliver data-driven insights.
Key Responsibilities :
Qualifications :
Qualifications :
Skills Required
Spark, Python, Hive, Etl, Iam, S3, Kafka, Aws
Create a job alert for this search
Team Lead • Bengaluru / Bangalore
Related jobs
- Promoted
Team lead - powerbi
Programmers.ioBangalore Urban, Karnataka, India Shift Timings : Should be available till 10 : 30 PM IST.Experience Required : 9 - 15 Years.Strong expertise in Power BI, DAX, SQL, Data Modelling, PowerBi services,.
Should have good experience in Data vi...Show moreLast updated: 26 days ago
- Promoted
Team Lead -Power Bi
Programmers.iohosur, tamil nadu, in Shift Timings : Should be able to work until 10 : 30 PM IST.Location : PAN INDIA Remote, Work from Home.Experience Required : 7 - 15 Years.
Strong expertise in Power BI, DAX, SQL, Data Modelling, PowerBi...Show moreLast updated: 17 days ago
- Promoted
Team Lead
ALTISOURCE BUSINESS SOLUTIONS PRIVATE LIMITEDBengaluru, IN Willing to work in night shift.Lead the property inspection operations in a multi-client environment ensuring adherence to service level agreements and quality standards.
Track team perfoJob Descrip...Show moreLast updated: 8 days ago
- Promoted
Team Lead
ConfidentialBengaluru / Bangalore, India Responsible for Markets Account Opening and Maintenance.To maintain high quality Client Static Data in SCI and TP system.
Implement suitable mitigations for the identified risk issues.To ensure erro...Show moreLast updated: 30+ days ago
- Promoted
- New!
Sales Team Lead
HexaHealthBengaluru, IN Location : Gurugram, Sector - 14.Salary : As per market standards.Lead and manage a team of sales representatives, providing guidance, motivation, and coaching to achieve sales targets.Develop and im...Show moreLast updated: 16 hours ago
- Promoted
Team Lead
Acmegradebangalore, karnataka, in Acmegrade is a dynamic e-learning platform committed to providing a holistic environment where students can develop essential skills for industry readiness.
We aim to empower and upskill the future ...Show moreLast updated: 1 day ago
- Promoted
Team Lead
Zensar TechnologiesBengaluru, IN ZENSAR -TEAM LEAD | PROJECT MANAGER OPPORTUNITY FOR GEN AI PROJECT.Dear Aspirant, Greetings from Zensar!!.We are a technology consulting and services company with over 11,500 associates in 33 globa...Show moreLast updated: 14 days ago
- Promoted
Team Lead-Software Development
Sosuv ConsultingBengaluru, Karnataka, India Company Profile : We are a technology firm in the UK, and India focussed on the financial services domain.Our clients are the world's major fintech players.
We are looking for intelligent, talented a...Show moreLast updated: 30+ days ago
- Promoted
Development Team Lead
ConfidentialBengaluru / Bangalore, India We're looking for No / Low code with expertise in Unqork Platform to join the Unqork team.They support the implementation of Unqork's platform.
This includes gathering application requirements.Oversee...Show moreLast updated: 21 days ago
- Promoted
Lead Generation Team Lead
ConfidentialBengaluru / Bangalore, India Job Title : Lead Generation Team Lead.We are seeking a dynamic and experienced.With 10 years of experience in the ITES sector, the ideal candidate will lead a team of analysts and specialists to dri...Show moreLast updated: 21 days ago
- Promoted
Senior Team Lead
ConfidentialBengaluru / Bangalore, India ConnectWise a US Product IT company is hiring passionate Senior Team leads for its development team.Join us if you have the passion for development.
ConnectWise a US IT product company is hiring a S...Show moreLast updated: 30+ days ago
- Promoted
Jr. Team Lead MDG
ConfidentialBengaluru / Bangalore, India At ABB, we help industries outrun - leaner and cleaner.Here, progress is an expectation - for you, your team, and the world.
As a global market leader, we'll give you what you need to make it happen...Show moreLast updated: 1 day ago
- Promoted
Technical Team Lead
ConfidentialBengaluru / Bangalore, India As a Product Specialist Team Lead, you are recognized as an expert in technology and a thought leader.We seek a natural team leader capable of managing our Product Specialist team's day-to-day inte...Show moreLast updated: 10 days ago
- Promoted
Team Lead - Data and Operations
ConfidentialBengaluru / Bangalore, India Stronghold Investment Management ('Stronghold,' 'SIM,' or 'the Firm') is a technology-driven, vertically integrated investment manager focused on real-assets and related private markets.Stronghold ...Show moreLast updated: 17 days ago
- Promoted
Azure Cloud Team Lead
LTIMindtreeBengaluru, Karnataka, India Notice Period : Immediate to 30 days.Please do not apply Notice Period with more than 30 days of Notice Period.Azure Services (must include all services).Show moreLast updated: 30+ days ago
- Promoted
Team Lead
FirstsourceBengaluru, Karnataka, India Firstsource Solutions is a leading provider of customized Business Process Management (BPM) services.Firstsource specializes in helping customers stay ahead of the curve through transformational so...Show moreLast updated: 30+ days ago