Senior Engineer IT

Job Description:

Key Responsibilities

• Design develop and maintain scalable data pipelines using Databricks and PySpark
• Build robust ETL/ELT workflows for batch and near realtime data processing
• Write optimized and efficient SQL queries for data transformation validation and analysis
• Work with large-scale distributed datasets using Spark-based processing frameworks
• Perform data ingestion from multiple sources (APIs files databases) into data lake/lakehouse architecture
• Ensure data quality integrity and consistency through validations and monitoring
• Optimize pipeline performance and cost (partitioning caching query tuning etc.)
• Collaborate with cross-functional teams (Data Ops Business Analytics) to deliver data solutions
• Troubleshoot pipeline failures and implement proactive monitoring and alerting
• Maintain proper documentation for data models transformations and workflows

Required Skills & Experience

• Strong hands-on experience in:
  • Databricks (Lakehouse platform notebooks jobs clusters)
  • PySpark / Apache Spark
  • Advanced SQL (joins window functions performance tuning)
• Solid understanding of ETL concepts data warehousing and data modeling
• Experience building scalable data pipelines for structured and semi-structured data
• Familiarity with data lake / delta lake architecture
• Good understanding of performance optimization techniques in Spark and SQL
• Experience with version control (Git) and deployment practices
• Strong problem-solving and analytical skills

Good to Have

• Experience with cloud platforms (AWS preferably)
• Exposure to CI/CD pipelines (Jenkins)
• Knowledge of Delta Lake Unity Catalog or similar governance tools
• Understanding of data quality frameworks / monitoring tools
• Experience working in Agile / Scrum teams

Location:

This position can be based in any of the following locations:

Chennai

Current Guardian Colleagues: Please apply through the internal Jobs Hub in Workday

Required Experience:

Senior IC

Employment Type : Full-Time
Experience: years
Vacancy: 1