Data Engineer I [T500-21404]

ADMBengaluru, Karnataka, India

10 days ago

Job description

About ADM :

We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition. We unlock the power of nature to provide nourishing quality of life by transforming crops into ingredients and solutions for foods, beverages, supplements, livestock, aquaculture, and pets.

About ADM India Hub :

At ADM, we have long recognized the strength and potential of India’s talent pool, which is why we have maintained a presence in the country for more than 25 years. Building on this foundation, we have now established ADM India Hub, our first GCC in India.

At ADM India Hub, we are hiring for IT and finance roles across diverse technology and business functions. We stand at the intersection of global expertise and local excellence, enabling us to drive innovation and support our larger purpose of unlocking the power of nature to enrich quality of life.

Job Title : Data Engineer I

Job Overview :

We are seeking an experienced and highly motivated Data Ingestion Engineer to join our dynamic team. The ideal candidate will have strong hands-on experience with Azure Data Factory (ADF), a deep understanding of relational and non-relational data ingestion techniques, and proficiency in Python programming. You will be responsible for designing and implementing scalable data ingestion solutions that interface with Azure Data Lake Storage Gen 2 (ADLS Gen 2), Databricks, and various other Azure ecosystem services.

Key Responsibilities :

Data Ingestion Strategy & Development :

Design, develop, and deploy scalable and efficient data pipelines in Azure Data Factory (ADF) to move data from multiple sources (relational, non-relational, files, APIs, etc.) into Azure Data Lake Storage Gen 2 (ADLS Gen 2), Azure SQL Database, and other target systems.
Implement ADF activities (copy, lookup, execute pipeline, etc.) to integrate data from on-premises and cloud-based systems.
Build parameterized and reusable pipeline templates in ADF to standardize the data ingestion process, ensuring maintainability and scalability of ingestion workflows.
Integrate custom data transformation activities within ADF pipelines, utilizing Python, Databricks, or Azure Functions when required

ADF Data Flows Design & Development :

Leverage Azure Data Factory Data Flows for visually designing and orchestrating data transformation tasks, enabling complex ETL (Extract, Transform, Load) logic to process large datasets at scale.

Design data flow transformations such as filtering, aggregation, joins, lookups, and sorting to process and transform data before loading it into target systems like ADLS Gen 2 or Azure SQL Database.

Implement incremental loading strategies in Data Flows to ensure efficient and optimized data ingestion for large volumes of data while minimizing resource consumption.

Develop reusable data flow components to streamline transformation processes, ensuring consistency and reducing development time for new data ingestion pipelines.

Utilize debugging tools in Data Flows to troubleshoot, test, and optimize data transformations, ensuring accurate results and performance.

ADF Orchestration & Automation :

Use ADF triggers and scheduling to automate pipeline execution based on time or events, ensuring timely and efficient data ingestion.

Configure ADF monitoring and alerting capabilities to proactively track pipeline performance, handle failures, and address issues in a timely manner.

Implement ADF version control practices using Git to manage code changes, collaborate effectively with other team members, and ensure code integrity.

Data Integration with Various Sources :

Ingest data from diverse sources such as on-premises SQL Servers, REST APIs, cloud databases (e.g., Azure SQL Database, Cosmos DB), file-based systems (CSV, Parquet, JSON), and third-party services using ADF.

Design and implement ADF linked services to securely connect to external data sources (databases, file systems, APIs, etc.).

Develop and configure ADF datasets and dataflows to efficiently transform, clean, and load data into Azure Data Lake or other destinations.

Pipeline Monitoring and Optimization :

Continuously monitor and optimize ADF pipelines to ensure they run with high performance and minimal cost. Apply techniques like data partitioning, parallel processing, and incremental loading where appropriate.

Implement data quality checks within the pipelines to ensure data integrity and handle data anomalies or errors in a systematic manner.

Review pipeline execution logs and performance metrics regularly and apply tuning recommendations to improve execution times and reduce operational costs.

Collaboration and Communication :

Work closely with business and technical stakeholders to capture and translate data ingestion requirements into ADF pipeline designs.

Provide ADF-specific technical expertise to both internal and external teams, guiding them in the use of ADF for efficient and cost-effective data pipelines.

Document ADF pipeline designs, error handling strategies, and best practices to ensure the team can maintain and scale the solutions.

Conduct training sessions or knowledge transfer with junior engineers or other team members on ADF best practices and architecture.

Security and Compliance :

Ensure all data ingestion solutions built in ADF follow security and compliance guidelines, including encryption at rest and in transit, data masking, and identity and access management.

Implement role-based access control (RBAC) and managed identities within ADF to manage access securely and reduce the risk of unauthorized access to sensitive data.

Integration with Azure Ecosystem :

Leverage other Azure services, such as Azure Logic Apps, Azure Function Apps, and Azure Databricks, to augment the capabilities of ADF pipelines, enabling more advanced data processing, event-driven workflows, and custom transformations.

Incorporate Azure Key Vault to securely store and manage sensitive data (e.g., connection strings, credentials) used in ADF pipelines.

Integrate ADF with Azure Data Lake Analytics, Synapse Analytics, or other data warehousing solutions for advanced querying and analytics after ingestion.

Best Practices & Continuous Improvement :

Develop and enforce best practices for building and maintaining ADF pipelines and data flows, ensuring the solutions are modular, reusable, and follow coding standards.

Identify opportunities for pipeline automation to reduce manual intervention and improve operational efficiency.

Regularly review and suggest new tools or services within the Azure ecosystem to enhance ADF pipeline performance and increase the overall efficiency of data ingestion workflows.

Required Skills and Qualifications :

Experience with Azure Data Services :

Strong experience with Azure Data Factory (ADF) for orchestrating data pipelines.

Hands-on experience with ADLS Gen 2, Databricks, and various data formats (e.g., Parquet, JSON, CSV).

Solid understanding of Azure SQL Database, Azure Logic Apps, Azure Function Apps, and Azure Container Apps

Programming and Scripting :

Proficient in Python / PySpark for data ingestion, automation, and transformation tasks.

Ability to write clean, reusable, and maintainable code.

API

Data Ingestion Techniques :

Solid understanding of relational and non-relational data models and their ingestion techniques.

Experience working with file-based data ingestion, API-based data ingestion, and integrating data from various third-party systems.

Problem Solving & Analytical Skills :

Strong troubleshooting and debugging skills to quickly resolve ingestion-related issues.

Ability to analyze and optimize complex data workflows for performance and efficiency.

Communication Skills :

Excellent communication skills to articulate complex technical solutions to non-technical stakeholders.

Ability to document processes and solutions clearly and concisely.

Work Ethic & Responsibility :

Demonstrated ability to take full responsibility for tasks, from requirements gathering to delivery.

Comfortable working independently and collaborating in a team environment.

Preferred Qualifications :

Bachelor’s degree in computer science, Engineering, or related field.

2 Year of relevant experience

Experience with version control systems like Git.

Experience in other Azure services such as Azure Synapse Analytics and Azure Data Share.

Familiarity with cloud security best practices and data privacy regulations.

Create a job alert for this search

Data Engineer • Bengaluru, Karnataka, India

Related jobs

Promoted

Senior Data Engineer – II

MakeMyTripBangalore Urban, Karnataka, India

At MakeMyTrip (MMT), technology is at the heart of everything we do.As a leading player in the travel industry, we leverage cutting-edge solutions like AI, machine learning, and cloud infrastructur...Show moreLast updated: 18 days ago

Promoted

Data Engineer 2

YubiBangalore Urban, Karnataka, India

As a Data Engineer, you will be part of a highly talented Data Engineering team.Responsible for developing reusable capabilities and tools to automate various types of data processing pipelines.You...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Insight GlobalBangalore, IN

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

HCLTechGreater Bengaluru Area, India

F2F Hackathon scheduled on 22nd Nov 2025.If you are passionate about building scalable data solutions on Google Cloud and enjoy solving real-time data engineering challenges, this is the perfect op...Show moreLast updated: 30+ days ago

Promoted

Data Engineer II [T500-21346]

ADMBengaluru, Karnataka, India

We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition.We unlock the power of nature to provide nourishing quality of life by transforming crops int...Show moreLast updated: 14 days ago

Promoted

Data Engineer I [T500-21304]

ADMBengaluru, Karnataka, India

About ADM : We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition.We unlock the power of nature to provide nourishing quality of life by transformi...Show moreLast updated: 16 days ago

Promoted

Lead Data Engineer [T500-21309]

ANSRBengaluru, Karnataka, India

ANSR is hiring for one of its clients.Martens is an iconic British brand founded in 1960 in Northamptonshire.Produced originally for workers looking for tough, durable boots, the brand was quickly ...Show moreLast updated: 15 days ago

Promoted
New!

Data Bricks Engineer

Relevance LabGreater Bengaluru Area, India

SQL skills and hands-on experience using.AI-assisted development tools).The ideal candidate will be responsible for designing, developing, and optimizing data pipelines, supporting analytics worklo...Show moreLast updated: 2 hours ago

Promoted
New!

Data Engineer I

LiciousGreater Bengaluru Area, India

SDE 1 – Data Platform Engineer.We are looking for a passionate and motivated.Software Development Engineer (SDE 1) – Data Platform. Data Platform Engineering team at.In this role, you’ll design, bui...Show moreLast updated: 2 hours ago

Promoted
New!

Data Engineer

VIDA Digital IdentityGreater Bengaluru Area, India

At VIDA we’re building the future of digital identity.As a Data Engineer you’ll work across the stack—from platform and infrastructure to data pipelines and end-user tooling.You’ll help us moderniz...Show moreLast updated: 2 hours ago

Promoted

Data Engineer II (AI / ML) [T500-21411]

ADMBengaluru, Karnataka, India

We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition.We unlock the power of nature to provide nourishing quality of life by transforming crops int...Show moreLast updated: 10 days ago

Promoted

Data Engineer

SelectorBengaluru, Karnataka, India

Selector is building an operational intelligence platform for digital infrastructure.By adopting an AI / ML-based analytics approach, the platform provides actionable multi-dimensional.It enables ope...Show moreLast updated: 30+ days ago

Promoted
New!

AWS Data Engineer

Tata Consultancy ServicesGreater Bengaluru Area, India

TCS has always been in the spotlight for being adept in the next big technologies.What we can offer you is a space to explore varied technologies and quench your techie soul.Required Skill Set : AWS...Show moreLast updated: 2 hours ago

Promoted
New!

Senior Data Engineer

ArcanaGreater Bengaluru Area, India

Join our dynamic team at the forefront of cutting-edge technology as we seek a seasoned Senior Data Engineer.Embark on a journey where your deep-rooted expertise in computer science fundamentals, a...Show moreLast updated: 2 hours ago

Promoted
New!

GCP Data Engineer

HCLTechGreater Bengaluru Area, India

HCLTech is a global technology company, home to more than 226,600 people across 60 countries, delivering industry-leading capabilities centered around AI, digital, engineering, cloud and software, ...Show moreLast updated: 2 hours ago

Promoted

Data Engineer I [T500-21306]

ADMBengaluru, Karnataka, India

Promoted

Data Engineer

ACL Digitalbangalore, karnataka, in

International experience with global projects and collaboration with intercultural team is preferred.Strong and Expert-like proficiency in Python frameworks FAST API, Knowledge Graph Libraries.Expe...Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer Technical Lead(Azure)

SigmoidGreater Bengaluru Area, India

Sigmoid empowers enterprises to make smarter, data-driven decisions by blending advanced data engineering with AI consulting. We collaborate with some of the world’s leading data-rich organizations ...Show moreLast updated: 2 hours ago