Lead Data Engineer - PySpark / Spark

About the Role :

We are looking for an experienced Lead Data Engineer with deep expertise in Big Data technologies, particularly within the Google Cloud Platform (GCP) ecosystem. The ideal candidate should have a strong command of PySpark / Spark, SQL, and Python, and a proven track record in building, optimizing, and managing large- scale data pipelines and cloud- native data platforms.

Key Responsibilities :

• Lead the design and implementation of scalable ETL / ELT pipelines using Spark (batch and stream) and Python
• Architect and optimize BigQuery solutions using advanced SQL, partitioning, clustering, and materialized views
• Guide the team on GCP services : Dataproc, GCS, BigQuery, Cloud Composer, Cloud Functions, IAM, and Cloud Logging
• Conduct code reviews and mentor team members on Spark optimization (caching, memory management, broadcast joins, skew handling)
• Drive Airflow DAG development, configuration management, and orchestration workflows
• Solve complex data engineering problems and contribute to architectural decisions for performance, scalability, and cost- efficiency
• Ensure data quality, governance, and security best practices are enforced across all data platforms
• Support team readiness through technical ramp- up and ongoing skill enhancement

Must- Have Skills :

(ref : hirist.tech)