Senior Software Engineer - Data Engineering / Migration

Job Profile :

• Data Engineer / Migration Specialist

Contract Duration :

6-7 months (ext)

Mode of work :

Remote

Working Hours :

UK Shift

Job Overview :

We are seeking an experienced Data Engineer / Migration Specialist to lead the assessment, planning, execution, and validation of a complex data migration project involving Databricks Unity Catalog and Hive Metastore. The ideal candidate will have deep technical expertise in cloud platforms (Azure), data governance, and hands-on experience with data platform engineering and migrations.

Qualifications & Skills :

9-10 years of hands-on experience in data platform engineering or migration projects

Strong proficiency with Databricks, Unity Catalog, Hive Metastore (HMS), and UCX assessment tools

Expertise in Azure cloud platform, including Identity and Access Management (IAM) and networking configurations

Familiarity with BI / reporting tools integration and enforcing security policy governance

Excellent documentation skills and experience collaborating across cross-functional teams

Good to Have :

Experience designing cluster policies and managing Unity Catalog-enabled environments

Knowledge of ACL mapping strategies and data governance best practices

Ability to create migration dashboards or reports for leadership visibility

Roles & Responsibilities :

HMS Assessment & Migration Planning :

Execute UCX Assessment Tool runs to evaluate Hive Metastore states, migration complexity, and readiness

Identify external and managed tables, BI dependencies, and environmental separation requirements

Prepare and deliver a comprehensive migration plan and finalize a detailed migration runbook

Platform Setup :

Configure necessary cloud resources : storage accounts, IAM roles, and networking

Set up Unity Catalog Metastore(s) and assign workspace mappings

Establish and validate admin roles such as Account Admin, Metastore Admin, and Workspace Admin

Unity Catalog Design & Cluster Enablement :

Inventory and categorize data assets including catalogs, schemas, tables, views, and files

Design catalog / schema hierarchy aligned with business units or data domains

Develop and test ACL / permission migration strategies and validate cluster policies to enable Unity Catalog

Migration Execution :

Use UCX tools and upgrade notebooks to migrate tables and views from Hive Metastore to Unity Catalog

Migrate and validate ACLs and permissions using audit scripts

Update BI tool configurations and modify application connection strings as needed

Transition and test job cluster policies for post-migration stability and functionality

Post-Migration Validation & Handover :

Verify data integrity, job executions, BI report accuracy, and access control mechanisms

Document rollback procedures and highlight any unresolved issues

Conduct handover workshops and knowledge transfer sessions with operational and support teams

(ref : hirist.tech)