Talent.com
Data Engineer

Data Engineer

OWOWJodhpur, IN
30+ days ago
Job description

Data Engineer – AI-Powered Marketing Personalization Platform

We’re seeking an experienced Data Engineer to help build and scale our next-generation AI-powered marketing personalization platform (V2.0) . You’ll design and implement a robust multi-database infrastructure that enables real-time personalization, vector search, graph analytics, and large-scale data processing.

This is a greenfield opportunity to architect data pipelines from the ground up using vector and graph databases and LLM-based systems . You’ll play a key role in migrating our existing platform while creating a scalable foundation powering AI agents across thousands of marketing campaigns.

Core Responsibilities

Data Architecture & Infrastructure (40%)

  • Design and implement multi-database systems (MongoDB, Redis, Milvus, Neo4j, BigQuery)
  • Build scalable real-time pipelines and ETL / ELT workflows
  • Implement data governance, quality, and high-throughput optimization

Vector & Graph Systems (25%)

  • Optimize Milvus collections for semantic search
  • Design Neo4j schemas for customer journeys and relationships
  • Develop hybrid search (vector + graph + text) with performance tuning

    • ML Data Infrastructure (20%)

  • Build data pipelines for LLM fine-tuning and GNN training
  • Manage data versioning, lineage, and A / B testing systems
  • Implement real-time feature computation for contextual models

    • Analytics & Monitoring (15%)

  • Create BigQuery schemas and real-time dashboards
  • Implement observability (Prometheus, Grafana) and anomaly detection
  • Optimize data costs across cloud environments

    • Tech Stack

    Databases : MongoDB, Redis, Milvus, Neo4j, BigQuery

    Processing : Airflow / Prefect, Pandas / Polars, dbt, Spark

    ML Pipeline : vLLM, MLflow, Sentence Transformers, PyTorch, TensorFlow

    Cloud & Infra : GCP, Docker, Kubernetes, Terraform, GitHub Actions

    Languages : Python (3.10+), SQL, Bash

    Requirements

    Must-Have

  • 5+ years of data engineering experience in production systems
  • Advanced Python and SQL expertise
  • Experience with 3+ database types (SQL, NoSQL, Vector, Graph)
  • Proven ability to build high-scale data pipelines (>

    • 1M records / day)

  • Strong data modeling, validation, and optimization skills
  • Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake)

    • Preferred

  • Experience with Milvus, Pinecone, or Weaviate
  • Graph databases (Neo4j, Neptune) and embedding-based search
  • ML / Feature store experience
  • Background in marketing technology or CDPs

    • Key Projects

    Phase 1 – Foundation : Migrate 10M+ vectors, implement MongoDB schemas, Neo4j models, and BigQuery warehouse

    Phase 2 – Optimization : Build data quality monitoring, caching (Redis), and

    Phase 3 – ML Infrastructure : Create LLM fine-tuning pipelines, GNN feature stores, and A / B testing systems

    Why Join Us

  • Collaborate with ML engineers and data scientists on cutting-edge AI systems
  • High ownership and real product impact
  • Modern tools and a flexible environment
  • Clear growth path to Senior, Lead, or Principal roles
  • Shape the future of AI-driven marketing personalization
    • Create a job alert for this search

    Data Engineer • Jodhpur, IN