This job offer is not available in your country.

Data Engineer

OWOWBengaluru, IN

12 hours ago

Job description

What You'll Build

Core Responsibilities

Data Architecture & Infrastructure (40%)

Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
Build scalable data pipelines for real-time conversation processing and personalization
Architect ETL / ELT workflows for data migration from legacy systems
Implement data partitioning, sharding, and optimization strategies for high-throughput systems
Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
Build graph schemas in Neo4j for customer journey mapping and persona relationships
Implement HNSW indexing strategies and similarity search optimization
Create hybrid search systems combining vector, full-text, and graph queries
Monitor and tune database performance (query latency, throughput, resource utilization)

ML Data Infrastructure (20%)

Build data collection pipelines for LLM fine-tuning (conversation logs, tool executions)

Create feature stores for GNN training (customer interactions, engagement signals)

Implement data versioning and lineage tracking for ML experiments

Design A / B testing data infrastructure with CUPED variance reduction

Build real-time feature computation pipelines for contextual bandits

Analytics & Monitoring (15%)

Design BigQuery schemas for marketing analytics and performance tracking

Create materialized views and aggregation pipelines for real-time dashboards

Implement data quality monitoring and anomaly detection

Build observability infrastructure (Prometheus metrics, Grafana dashboards)

Develop cost optimization strategies for cloud data warehousing

Technical Stack You'll Work With

Databases & Storage

MongoDB (conversation state, active sessions)

Redis (caching, rate limiting, real-time data)

Milvus (vector embeddings, semantic search)

Neo4j (customer journey graphs, persona networks)

BigQuery (analytics warehouse, historical data)

Data Processing & Orchestration

Apache Airflow or Prefect (workflow orchestration)

Pandas , Polars (data transformation)

Apache Spark (optional - for large-scale processing)

dbt (data transformation and modeling)

ML / AI Data Pipeline

vLLM (LLM inference serving)

MLflow (model registry, experiment tracking)

Sentence Transformers (embedding generation)

PyTorch , TensorFlow (ML model training)

Cloud & Infrastructure

Google Cloud Platform (BigQuery, Cloud Storage, Compute)

Docker & Kubernetes (containerization, orchestration)

Terraform (infrastructure as code)

GitHub Actions or GitLab CI (CI / CD pipelines)

Programming & Tools

Python 3.10+ (primary language)

SQL (complex queries, query optimization)

Shell scripting (Bash / Zsh)

Git (version control)

Requirements

Must-Have Skills

5+ years of data engineering experience with production systems

Expert-level SQL and database design skills

Strong Python programming (async / await, type hints, testing)

Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph)

Proven track record building high-scale data pipelines (>

1M records / day)

Deep understanding of data modeling (dimensional, normalized, denormalized)

Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake)

Strong knowledge of data quality, validation, and governance

Excellent debugging and optimization skills

Highly Desirable

Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant)

Experience with graph databases (Neo4j, ArangoDB, Neptune)

Knowledge of embedding models and semantic search

Experience with ML data pipelines (feature stores, model training data)

Understanding of A / B testing and experimental design

Experience with real-time streaming (Kafka, Pub / Sub, Kinesis)

Knowledge of LLMs and conversational AI systems

Experience with data migration projects (especially large-scale)

Background in marketing technology or customer data platforms

Nice-to-Have

Experience with PyTorch Geometric or graph neural networks

Knowledge of marketing analytics (attribution, segmentation, personalization)

Familiarity with LangChain , LangGraph , or agent frameworks

Experience with cost optimization in cloud environments

Contributions to open-source data engineering projects

Experience with data compliance (GDPR, CCPA)

Key Projects You'll Own

Phase 1 : Foundation

Migrate 10M+ conversation vectors from Pinecone to Milvus

Design and implement MongoDB schemas for real-time agent state

Set up Neo4j graph database with customer journey models

Create BigQuery data warehouse with partitioned tables

Phase 2 : Optimization

Build automated data quality monitoring system

Implement caching strategies (Redis) for 10x latency reduction

Optimize vector search queries (target :

Create real-time analytics dashboards (Grafana)

Phase 3 : ML Infrastructure

Build LLM fine-tuning data pipeline

Implement feature store for GNN training

Create A / B testing data infrastructure

Design multi-armed bandit state management

Work Environment

Collaborative team : Work with ML engineers, backend developers, and data scientists

Modern stack : Latest technologies and tools

Impact : Your work directly affects millions of marketing interactions

Autonomy : Own your projects end-to-end

Growth : Clear path to Senior / Lead / Principal roles

Create a job alert for this search

Data Engineer • Bengaluru, IN

Related jobs

Promoted

Data Engineer 2

YubiBangalore Urban, Karnataka, India

As a Data Engineer, you will be part of a highly talented Data Engineering team.Responsible for developing reusable capabilities and tools to automate various types of data processing pipelines.You...Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer

Velodata Global Pvt Ltdhosur, tamil nadu, in

Job Title : Data Engineer (Contract – 6 Months) and (Permanent positions are open ).Joining Date : October 22 (Candidate should work in our client office in Trivandrum from Oct 22 to October 31) aft...Show moreLast updated: 22 hours ago

Promoted

Data Engineer

EVERSANABengaluru, Karnataka, India

At EVERSANA, we are proud to be certified as a Great Place to Work across the globe.We’re fueled by our vision to create a healthier world. How? Our global team of more than 7,000 employees is commi...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

MareanaBengaluru, Karnataka, India

Mareana is looking for experienced Data Engineer with a strong development background using, RDMS, graph database and NoSQL platforms. This role is responsible for developing and enhancing Mareana’s...Show moreLast updated: 5 days ago

Promoted

Data Engineer

RecroBengaluru, Karnataka, India

You will work with business teams to transform raw data into actionable insights.Azure Data Factory, Databricks, and Synapse Analytics. Troubleshoot and optimize data workflows.ADF, Databricks, Syna...Show moreLast updated: 30+ days ago

Promoted

Data Engineer – Databricks & PySpark

Capgemini EngineeringBengaluru, Karnataka, India

Data Engineer – Databricks & PySpark.Choosing Capgemini means choosing a place where you’ll be empowered to shape your career, supported by a collaborative global community, and inspired to reimagi...Show moreLast updated: 4 days ago

Promoted

Data Engineer

Talogybangalore, karnataka, in

This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

EveriseBangalore, IN

Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 26 days ago

Promoted

Data Engineer

AcqueonBengaluru, Karnataka, India

CX) across our products and services.This role offers the opportunity to design and scale a platform that unifies customer data from multiple sources, ensures data quality and governance, and provi...Show moreLast updated: 5 days ago

Promoted

Data Engineer

One Tapp ConsultingBengaluru, Karnataka, India

Design and Development : Design, build, and maintain robust, scalable, and optimized ETL / ELT data.Python and standard data warehousing principles. Data Platform Management : Implement and manage data ...Show moreLast updated: 3 days ago

Promoted

Data Engineer

SelectorBengaluru, Karnataka, India

Selector is building an operational intelligence platform for digital infrastructure.AI / ML-based analytics approach, the platform provides actionable multi-dimensional. It enables operations teams t...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

V2Softhosur, tamil nadu, in

Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala. Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 5 days ago

Promoted

Data Engineer

INFEC Serviceshosur, tamil nadu, in

Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 24 days ago

Promoted

Data Engineer

ThoughtFocusBengaluru, Karnataka, India

Looking for Immediate joiners only.Shift time : 1 : 00 PM to 10 : 00 PM.Skills : Python, pyspark, Databricks.The Data Engineer responsible for implementing and managing the operational aspects of cloud-n...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

AAA Globalhosur, tamil nadu, in

Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 5 days ago

Promoted

Data Engineer

Keasishosur, tamil nadu, in

We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS / Hive / Impala) into Apache Iceberg tables, with dow...Show moreLast updated: 5 days ago

Promoted

Data Engineer

VAANTECHhosur, tamil nadu, in

Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago

Promoted

Lead Data Engineer

CoffeeBeansBengaluru, Karnataka, India

Founded in the year 2017, CoffeeBeans specialises in offering high end consulting services in technology, product, and processes. We help our clients attain significant improvement in quality of del...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Magma Consultancybangalore, karnataka, in

Part-time (20–25 hours per week).The ideal candidate has hands-on experience in building, optimizing, and maintaining data pipelines and architectures. You’ll work closely with analysts, developers,...Show moreLast updated: 3 days ago

Promoted

Data Engineer

Manuh TechnologiesBengaluru, IN

S3, Glue, Redshift, EMR, Lambda).Develop automation scripts and tools using.Collaborate with data analysts, data scientists, and business stakeholders to ensure data availability and reliability.Tr...Show moreLast updated: 30+ days ago