Big Data Engineer

Description

Data Engineer : Primary : ADF, SQL

Seconday : Databricks, Pyspark / Scala, CICD, Data Security, encryption

Responsibilities :

• Work closely with scrum master, technical leads and provide daily progress updates on the tasks assigned.
• Provide support in defining the scope and sizing of the work.
• Working closely with various enterprise architects, Information security teams, Data management team, to ensure the architected solution meets all the needs of a customer, from a functionality perspective and IT solution engineering perspective.
• Translate business requirements into technology solutions.
• Lead designing of all aspects of our data solution including artifact creation such as diagrams, playbooks, and other technical documents Skills :
• Over all 5 to 8 years of experience with Data Management & Big Data and Analytics.
• At least 2 to 3 years of experience in implementing the data solutions on MS Azure using Azure Data Factory.
• Must have strong SQL background and ability to tune the SQL for optimized ETL jobs.
• one to two years of experience in Azure Synapse Analytics is plus.
• Extensive hands-on experience in data warehousing design, tuning and ETL / ELT process development by using cloud native technologies.
• Apache Spark experience using Scala or PySpark or pre-packaged tools like Databrick is plus.
• At least one year of experience with API based / real time integrations using Kafka or similar technologies.
• At least 1 to 2 years of experience with big data using spark and java for data processing.
• Developing the CICD pipeline for Azure Infrastructure, version control strategy and Integrate source control (git, GitHub, azure repos).
• At least one year experience with unified data governance solution using MS Purview.
• In-depth understanding of various storage services offered by Azure.
• Experience with implementation of data security, encryption, PII / PSI legislation, identity and access management across sources and environments.
• Experience with data process Orchestration, end-to-end design and build process of Near-Real Time and Batch Data Pipelines.
• Certification in Azure fundamentals (AZ-900) and Azure data engineering (DP-203) is must.
• Strong client-facing communication and facilitation skills.