Data Engineer

Overview : We are looking for a highly skilled Python Data Engineer to join our team in an on-premise data engineering environment. The ideal candidate will have experience in ETL tools , data processing technologies , data orchestration , and relational databases . Additionally, you should be proficient in Python scripting for data engineering tasks and have experience working with Spark , PySpark , and other relevant data technologies. While cloud tools are a good-to-have , this position primarily focuses on on-premise data infrastructure .This is an excellent opportunity to work on exciting projects that require developing scalable data pipelines, real-time data streaming, and optimizing data processing tasks using Python.Key Responsibilities : - ETL Development & Optimization : Design, develop, and optimize ETL pipelines using open-source or cloud ETL tools (e.g., Apache Nifi , Talend , Pentaho , Airflow , AWS Glue).- Python Scripting for Data Engineering : Write Python scripts to automate data extraction, transformation, and loading (ETL) processes. Ensure that the code is optimized for performance and scalability.- Big Data Processing : Work with Apache Spark and PySpark to process large datasets in a distributed computing environment. Optimize Spark jobs for performance and resource efficiency.- Job Orchestration : Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.- Data Streaming : Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.- File Formats & Table Formats : Work with open-source table formats like Apache Parquet , Apache Avro , or Delta Lake , and other structured / unstructured data formats for efficient data storage and access.- Database Management : Work with relational databases (e.g., PostgreSQL , MySQL , SQL Server ) for data storage, management, and optimization. Understand database concepts such as normalization, indexing, and query optimization.- SQL Expertise : Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets. Ensure queries are efficient and scalable.- BI & Data Warehouse Knowledge : Exposure to BI tools and data warehousing concepts is a plus, ensuring the data is structured in a way that supports analytics and reporting.Required Skills & Experience : - ETL Tools : Experience working with open-source ETL tools such as Apache Nifi , Talend , or Pentaho . Cloud-based tools like AWS Glue or Azure Data Factory are good to have.- Python Scripting : Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries such as Pandas , Dask , PySpark , etc.- Big Data Technologies : Experience with Apache Spark and PySpark for distributed data processing, along with optimization techniques.- Data Orchestration : Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.- Data Streaming : Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.- Open-Source File Formats : Knowledge of Apache Parquet , Apache Avro , Delta Lake , or similar open-source table formats for efficient data storage and retrieval.- Relational Databases : Strong experience with at least one relational database (e.g., PostgreSQL , MySQL , SQL Server ) and a solid understanding of database concepts like indexing, normalization, and query optimization.- SQL Expertise : Strong skills in writing and optimizing complex SQL queries for data extraction, transformations, and aggregation.Nice to Have : - BI / Analytics Tools : Familiarity with BI tools like Power BI , Tableau , Looker , or similar reporting and data visualization platforms.- Data Warehousing : Knowledge of data warehousing principles, schema design (e.g., star / snowflake), and optimization techniques for large datasets.- Cloud Technologies : Experience with cloud data platforms like Databricks , Snowflake , or Azure Synapse is beneficial, though the role is focused on on-prem environments.- Containerization : Familiarity with containerization tools like Docker or Kubernetes for deploying data engineering workloads.Educational Qualifications : - Bachelor’s or Master’s degree in Computer Science , Engineering , Information Systems , or a related field (or equivalent work experience).Additional Qualities : - Excellent problem-solving and troubleshooting skills.- Ability to work both independently and in a collaborative environment.- Strong communication skills , both written and verbal.- Detail-oriented with a focus on data quality and performance optimization .- Proactive attitude and the ability to take ownership of projects.