Talent.com
Data Engineer

Data Engineer

Ubique SystemsSrikakulam, Andhra Pradesh, India
30+ days ago
Job description

Primary skills : Python, SQL, data lakes, azure

Experience : 4+ years

Immediate Joiner

Key Responsibilities

Pipeline Development & Automation

  • Design, build, and maintain CI / CD pipelines to automate deployment of DQ rules and data services across environments.
  • Optimize data pipelines and jobs for efficiency, scalability, and enterprise-grade reliability within Azure and Databricks.
  • Implement best practices in version control, testing, and automated deployments.

Data Quality & Rule Development

  • Develop, optimize, and maintain DQ rules in PySpark / Python for self-serve capabilities.
  • Design and implement profiling frameworks for rule generation and automated remediation.
  • Ensure DQ frameworks align with enterprise standards, governance, and audit requirements.

    • UI & Self-Serve Integration

  • Collaborate with front-end teams (Node.js / Angular) to enable rule configuration, validation, and monitoring via a low-code UI.
  • Develop APIs and services to expose DQ results and outputs for dashboards and self-service tools.

    • Collaboration & Governance

  • Partner with Data Stewards, GPOs, and business owners to translate requirements into technical solutions.
  • Contribute to BRDs, SRDs, and design reviews for DQ rules and pipelines.
  • Provide inputs on governance, compliance, and change management processes.
  • Support periodic reviews and continuous improvements of pipelines, rules, and dashboards.

    • Required Skills & Qualifications

    Strong programming skills in PySpark, Python, and SQL.

    Hands-on experience with Databricks (clusters, notebooks, Delta Lake).

    Experience designing and implementing CI / CD pipelines (Azure DevOps / GitHub Actions / Jenkins).

    Familiarity with Node.js & Angular for UI integration.

    Solid understanding of data governance, DQ frameworks, and MDM concepts.

    Strong analytical, problem-solving, and stakeholder engagement skills.

    Cloud expertise, preferably Azure Data Lake, Data Factory, and Synapse.

    Education & Experience

    Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field.

    4–8 years of relevant experience in data engineering, CI / CD, and DQ rule development.

    Proven expertise in Databricks, Azure data services, and low-code self-serve frameworks.

    Create a job alert for this search

    Data Engineer • Srikakulam, Andhra Pradesh, India