Talent.com
Senior Data Engineer

Senior Data Engineer

American InferenceThrissur, IN
12 hours ago
Job description

About the Company

We are an AI and Data Consulting Startup transforming how businesses leverage technology through four core service lines :

  • Consulting Services : AI Strategy, Automation, and Digital Transformation for enterprises.
  • SaaS Platform Development : Building a business application suite similar to Odoo and Zoho that is AI-native and user friendly.
  • Data Lakehouse Solutions : Unified data pipelines for aggregation, cleaning, governance, and advanced analytics.
  • Government Contracting : Developing secure, compliant AI solutions for the public sector.

Our tech stack includes Python, TypeScript, React, Next.js, Go, Rust, Azure, Kubernetes, Spark, MLflow, Postgres, graph databases, and vector stores.

We're a small, fast-moving team delivering enterprise-grade solutions with startup agility.

Role Overview

You will design, build, and optimize data lakehouse solutions, enabling analytics, AI, and MLOps across multiple clients.

Key Responsibilities

  • Design lakehouse architectures (Bronze / Silver / Gold) using Azure Databricks, Delta Lake, or Iceberg
  • Build scalable PySpark pipelines for batch and streaming data
  • Implement data quality, validation, and lineage tracking
  • Integrate pipelines with MLflow for experiment tracking
  • Optimize Spark performance and cloud cost
  • Ensure data governance, PII protection, and security compliance
  • Mentor developers and document patterns and runbooks

    • Required Skills

  • Apache Spark (PySpark) and SQL (complex joins, CTEs, optimization)
  • Delta Lake / Iceberg and Azure Data Stack (ADLS, Synapse, Databricks)
  • MLflow integration for MLOps pipelines
  • Kafka / Event Hubs for streaming data
  • Strong Python and data modeling (dimensional, Data Vault)
  • 5+ years hands-on in data engineering with production-grade systems

    • Nice to Have

  • Vector databases, dbt, Airflow, Terraform, FinOps exposure, and Great Expectations data validation
  • Experience with data governance technologies (such as MS Purview or similar)
  • Certifications : Azure Data Engineer / Databricks DE

    • Education

  • UG : B.Tech / B.E. – Computer Science / Information Technology
  • PG : M.Tech / M.E. / Any Postgraduate (preferred)

    • Why Join Us

  • Impact : You'll have significant influence over our data architecture and enable AI / ML capabilities across the organization.
  • Growth : Help shape and build our data-driven culture from the ground up.
  • Learning : Work with the latest in AI, code, and data technologies.
  • Culture : Flat hierarchy, collaborative, and outcome-driven team.
    • Create a job alert for this search

    Senior Data Engineer • Thrissur, IN