Data Engineer (Immediate Start)

Overview :

We are looking for a highly skilled Python Data Engineer to join our team in an on-premise data engineering environment. The ideal candidate will have experience in ETL tools, data processing technologies, data orchestration, and relational databases. Additionally, you should be proficient in Python scripting for data engineering tasks and have experience working with Spark, PySpark, and other relevant data technologies. While cloud tools are a good-to-have, this position primarily focuses on on-premise data infrastructure.

This is an excellent opportunity to work on exciting projects that require developing scalable data pipelines, real-time data streaming, and optimizing data processing tasks using Python.

Key Responsibilities :

• ETL Development & Optimization : Design, develop, and optimize ETL pipelines using open-source or cloud ETL tools (e.g., Apache Nifi, Talend, Pentaho, Airflow, AWS Glue).
• Python Scripting for Data Engineering : Write Python scripts to automate data extraction, transformation, and loading (ETL) processes. Ensure that the code is optimized for performance and scalability.
• Big Data Processing : Work with Apache Spark and PySpark to process large datasets in a distributed computing environment. Optimize Spark jobs for performance and resource efficiency.
• Job Orchestration : Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.
• Data Streaming : Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
• File Formats & Table Formats : Work with open-source table formats like Apache Parquet, Apache Avro, or Delta Lake, and other structured / unstructured data formats for efficient data storage and access.
• Database Management : Work with relational databases (e.g., PostgreSQL, MySQL, SQL Server) for data storage, management, and optimization. Understand database concepts such as normalization, indexing, and query optimization.
• SQL Expertise : Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets. Ensure queries are efficient and scalable.
• BI & Data Warehouse Knowledge : Exposure to BI tools and data warehousing concepts is a plus, ensuring the data is structured in a way that supports analytics and reporting.

Required Skills & Experience :

Nice to Have :

Educational Qualifications :

Additional Qualities :