Job Description :
We are looking for a highly skilled Senior Data Engineer with strong expertise in AWS, Python, and PySpark to join our team.
Key Responsibilities :
- Build scalable PySpark applications using Spark DataFrames in Python.
- Optimize Spark jobs for large-scale data processing.
- Work with AWS analytics services : EMR, Athena, Glue.
- Work with AWS compute / storage services : Lambda, EC2, S3, SNS.
- Collaborate with cross-functional teams to design and implement efficient data pipelines.
- Maintain codebase using Git or other version control systems.
- Ensure data quality, performance tuning, and process improvements.
Required Skills :
5+ years of IT experience, with hands-on in Big Data technologies.Strong expertise in Python and PySpark.Practical experience with data compression & storage formats : Parquet, Avro, ORC, Snappy, Gzip.Good understanding of data warehousing concepts dimensions, facts, schemas (snowflake, star).Good to Have :
Knowledge of AWS databases such as Aurora, RDS, Redshift, ElastiCache, DynamoDB.(ref : hirist.tech)