Shift Timings : UK Shift
Location : Noida office (Hybrid)
In this role, we are seeking a data engineer to design, implement, and optimize cloud-based data
pipelines using Microsoft Azure services including ADF, Synapse, and ADLS.
Responsibilities
- Collaborate with our business analytics and data science teams, gathering requirements and
delivering complete business intelligence solutions
Mentor junior software developers and build a strong teamModel data and metadata to support discovery, ad-hoc, and pre-built reportingDesign and implement data pipelines using Hadoop, Spark, and Azure services such as BlobStorage, SQL Database, Event Hubs, Data Factory, Synapse Analytics, and Databricks
Should be able to write Programs & Scripting, Strong in SQL, Proficiency in Python or Scala.Experience with PowerShell or Azure CLI for automation is a plus
Partner with security, privacy, and legal teams to deliver solutions that comply with security andprivacy policies
Own the design, development, and maintenance of datasets our business analytics teams willuse to drive key business decisions
Develop and promote standard methodologies in data engineering, including scalability,reusability, maintainability, and usability
Tune and ensure compute performance by optimizing queries, databases, files, tables, andprocesses
Ensure data and report service level agreements are metAnalyze and solve problems at their root, stepping back to understand the broader contextOwn continuous engineering operational excellence of the datasets that drive key businessdecisions
Learn and understand a broad range of data resources and know when, how, and which to useand which not to use
Keep up to date with advances in big data technologies and run pilots to design the dataarchitecture to scale with increased data volume using Azure
Continually improve ongoing reporting and analysis processes, automating or simplifyingself-service support for datasets
Triage many possible courses of action in a high-ambiguity environment, making use of bothquantitative analysis and business judgment
Qualifications we seek in you!
Minimum Qualifications / Skills
Bachelor’s degree in computer science or related technical field10+ years of experience in data architecture and business intelligence5+ years of experience in developing solutions in distributed technologies such as Hadoop, SparkExperience in delivering end-to-end solutions using Azure services – Blob Storage, SQL Database,Event Hubs, Data Factory, Synapse Analytics, and HDInsight
Experience in programming using Python, Java, or ScalaExpert in data modeling, metadata management, and data qualitySQL performance tuningStrong interpersonal and multitasking skills with the ability to balance competing prioritiesExcellent communication (verbal and written) and interpersonal skills and an ability to effectivelycommunicate with both business and technical teams
An ability to work in a fast-paced ambiguous environment where continuous innovation isoccurring
Experience with a business intelligence reporting toolPreferred Qualifications / Skills
Experience with Databricks for advanced analytics and data processingUnderstanding of well-architected data pipeline designsExpertise in monitoring and fault-tolerant pipeline designsKnowledge of cluster configuration for optimal performanceAbility to create cost-optimal solutionsExperience in exploratory data analysis (dashboarding, plotting) using machine learningtechnologies and algorithms is desirable
Good knowledge of standard machine learning techniques (like regression, classification,anomaly detection, forecasting) by using standard machine learning libraries part of Spark,
Python is desirable
Prior experience in gen AI and related tools and techniques (such as large language models,prompt engineering) is desirable
Having a relevant Azure certification (architecture / data / machine learning) is desirable