Senior Data Engineer

The Data Engineer should have strong data infrastructure and data architecture skills, a proven track record of administration for data engineering teams, strong operational skills to drive efficiency and speed, strong project management skills, and a core value base for how analytics can proactively improve organizations. You will be part of the Data & Analytics team and will partner closely with a team of data scientists, business analysts & business intelligence leading a cloud-based strategy for our clients.

About the Role

You'll work on our Azure based data pipeline projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data to turn information into insights using multiple technology platforms. Therefore, this role requires an understanding of how a secure data cloud environment is architected to gain real insights faster, with less friction and complexity. Ideal candidate should be a champion of building a distributed and scalable architecture that is metadata driven. You're also a detail-oriented person that believes data quality should never be sacrificed.

Mandatory Skills : Python, SQL, Snowflake, Cloud

Experience - 5 to 7 yrs

Location - Bangalore

Work Mode - Hybrid

Responsibilities :

Master the use of technology & modern tools to optimize solutions and continuous process improvements

Facilitate team to design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation

Help build out and support an Enterprise Data Marketplace

Administer the Big data platform on the cloud

Implement automation of jobs and testing

Optimize the data pipeline to support ML workloads and use cases

Support mission critical applications and near real time data needs from the data platform

Build and support reusable framework to ingest, integration and provision data

Capture and publish metadata and new data to subscribed users

Work collaboratively with product managers, data scientists as well as business partners and actively participate in design thinking session

Motivate, coach, and serve as a role model and mentor for other development team associates / members that leverage the platform

Establish relationship with key business lines, Infosec and infrastructure team

Ensure security is integrated into all cloud services.

Build, monitor, and manage the cloud infrastructure.

Monitor and optimize cloud workloads.

Review current and future Azure implementations and suggest ways to improve and standardize the cloud offerings.

Qualifications

Bachelor’s degree in Computer Science, Physics, Mathematics, Statistics or other Engineering disciplines

Required Skills :

5+ years' experience as Data engineer

Experience with big data tools : Spark, Databricks

Experience with building relational SQL Data Warehouses

Good Hands on experience with data pipeline and workflow management tools : Azure Data Factory, SSIS

Experience with object-oriented / object function scripting languages : Python, C#, Scala, VBA, etc.

Experience with systems scripting languages : Batch Scripts, Powershell

Experience with Git / GitHub and automating builds

Strong interpersonal, verbal and written communication skills and ability to present complex technical / analytical concepts to executive audience

Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and delivery

Preferred Skills :

Experience building distributed architecture and IoT in a Dev / Ops environment

Experience in both Agile and Waterfall methodologies

Experience building and optimizing claims data pipelines, architectures / data models and data sets.

Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

Experience with the Agile workflow, Test Driven Development, and continuous improvement / continuous deployment.

Strong analytic skills related to working with structured and unstructured datasets.

Build processes supporting data transformation, data structures, metadata, dependency and workload management using Azure Automation, Azure Function Apps, and Azure Data Factory.

A successful history of manipulating, processing and extracting value from large disconnected datasets.

Working knowledge of Azure Automation, Azure Data Factory, and highly scalable 'big data' data stores such as Azure Blob Storage, Azure Data Lake

Experience supporting and working with cross-functional teams in a dynamic environment.

Excellent planning, prioritization, problem solving skills.

Experience with any of the following scripting languages, R, JSON, Parquet, JavaScript, Regex, PowerShell, shell scripting.

Experience with Data Bricks, HDInsight, Python.