Description :
Position : Data Integration Engineer
Experience : 2-5 Years
Location : Bangalore
Industry Type : Logistics / Supply Chain Technology (Assumed)
Education : Bachelor's degree in Computer Science or a related field.
Job Summary :
We are seeking a proactive Data Integration Engineer with 2-5 years of hands-on experience in building and optimizing robust data pipelines. The role requires strong proficiency in Python (including Pandas / NumPy), an excellent grasp of data structures and algorithms, and practical experience with ETL / data integration workflows. The Engineer will be responsible for designing and managing data exchange across various formats (JSON / XML, CSV, EDI) and sources (REST APIs, relational / NoSQL databases), with mandatory experience in workflow orchestration using Apache Airflow.
Job Description :
Data Pipeline Design and Development (ETL) :
- Design, build, and maintain robust ETL / data pipelines or data integration workflows to ensure the seamless, high-volume flow of data between internal and external systems.
- Demonstrate mandatory Proficiency in Python, utilizing related libraries such as Pandas and NumPy for complex data manipulation, cleansing, and transformation tasks.
- Apply a strong understanding of object-oriented programming (OOP) principles, data structures, and algorithms to develop scalable and efficient code.
- Ensure the quality and reliability of integrated data by implementing rigorous testing and error handling within all ETL processes.
Data Management and Integration Technologies :
Possess comprehensive knowledge of both relational and non-relational databases, including expertise in data modeling and normalization techniques to optimize storage and retrieval efficiency.Work hands-on with various data formats, including structured files like JSON and XML, and flat file formats such as CSV and EDI (Electronic Data Interchange), which are critical for the logistics domain.Develop and manage data retrieval mechanisms by working extensively with REST APIs for integrating external partner and client data sources.Create, schedule, and manage complex workflows using Apache Airflow to orchestrate pipelines, manage task dependencies, and monitor job execution effectively.Collaboration and System Optimization :
Analyze data flow requirements and perform detailed impact analysis for changes to upstream and downstream systems.Demonstrate the ability to collaborate effectively within teams and exhibit the discipline to work independently when necessary to meet project deadlines.Contribute to system optimization efforts, focusing on improving the performance, resilience, and cost-effectiveness of data integration infrastructure.Required Skills & Qualifications :
Experience : Mandatory 2-5 years of experience in data engineering or integration.Core Programming : Proficiency in Python and related libraries (Pandas and NumPy).Fundamentals : Strong understanding of object-oriented programming, data structures, and algorithms.Databases : Knowledge of relational and non-relational databases, data modeling, and normalization.Pipelines : Hands-on experience designing and building ETL / data pipelines or data integration workflows.Orchestration : Experience creating workflows using Apache Airflow.Formats / APIs : Experience with structured (JSON / XML) and flat files (CSV, EDI), and working with REST APIs.Education : Bachelor's degree in Computer Science or a related field.Preferred Skills :
Cloud : Exposure to AWS or other cloud platforms (e.g., Azure, GCP) and cloud-native data services.Client Engagement : Prior client-facing experience for requirement gathering and technical discussion.Domain : Familiarity with data standards and integration challenges within the Logistics or Supply Chain domain.(ref : hirist.tech)
