No longer accepting applications

Data Engineer

OWOWMeerut, IN

8 days ago

Job description

What You'll Build

Core Responsibilities

Data Architecture & Infrastructure (40%)

Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
Build scalable data pipelines for real-time conversation processing and personalization
Architect ETL / ELT workflows for data migration from legacy systems
Implement data partitioning, sharding, and optimization strategies for high-throughput systems
Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
Build graph schemas in Neo4j for customer journey mapping and persona relationships
Implement HNSW indexing strategies and similarity search optimization
Create hybrid search systems combining vector, full-text, and graph queries
Monitor and tune database performance (query latency, throughput, resource utilization)

ML Data Infrastructure (20%)

Build data collection pipelines for LLM fine-tuning (conversation logs, tool executions)

Create feature stores for GNN training (customer interactions, engagement signals)

Implement data versioning and lineage tracking for ML experiments

Design A / B testing data infrastructure with CUPED variance reduction

Build real-time feature computation pipelines for contextual bandits

Analytics & Monitoring (15%)

Design BigQuery schemas for marketing analytics and performance tracking

Create materialized views and aggregation pipelines for real-time dashboards

Implement data quality monitoring and anomaly detection

Build observability infrastructure (Prometheus metrics, Grafana dashboards)

Develop cost optimization strategies for cloud data warehousing

Technical Stack You'll Work With

Databases & Storage

MongoDB (conversation state, active sessions)

Redis (caching, rate limiting, real-time data)

Milvus (vector embeddings, semantic search)

Neo4j (customer journey graphs, persona networks)

BigQuery (analytics warehouse, historical data)

Data Processing & Orchestration

Apache Airflow or Prefect (workflow orchestration)

Pandas , Polars (data transformation)

Apache Spark (optional - for large-scale processing)

dbt (data transformation and modeling)

ML / AI Data Pipeline

vLLM (LLM inference serving)

MLflow (model registry, experiment tracking)

Sentence Transformers (embedding generation)

PyTorch , TensorFlow (ML model training)

Cloud & Infrastructure

Google Cloud Platform (BigQuery, Cloud Storage, Compute)

Docker & Kubernetes (containerization, orchestration)

Terraform (infrastructure as code)

GitHub Actions or GitLab CI (CI / CD pipelines)

Programming & Tools

Python 3.10+ (primary language)

SQL (complex queries, query optimization)

Shell scripting (Bash / Zsh)

Git (version control)

Requirements

Must-Have Skills

5+ years of data engineering experience with production systems

Expert-level SQL and database design skills

Strong Python programming (async / await, type hints, testing)

Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph)

Proven track record building high-scale data pipelines (>

1M records / day)

Deep understanding of data modeling (dimensional, normalized, denormalized)

Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake)

Strong knowledge of data quality, validation, and governance

Excellent debugging and optimization skills

Highly Desirable

Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant)

Experience with graph databases (Neo4j, ArangoDB, Neptune)

Knowledge of embedding models and semantic search

Experience with ML data pipelines (feature stores, model training data)

Understanding of A / B testing and experimental design

Experience with real-time streaming (Kafka, Pub / Sub, Kinesis)

Knowledge of LLMs and conversational AI systems

Experience with data migration projects (especially large-scale)

Background in marketing technology or customer data platforms

Nice-to-Have

Experience with PyTorch Geometric or graph neural networks

Knowledge of marketing analytics (attribution, segmentation, personalization)

Familiarity with LangChain , LangGraph , or agent frameworks

Experience with cost optimization in cloud environments

Contributions to open-source data engineering projects

Experience with data compliance (GDPR, CCPA)

Key Projects You'll Own

Phase 1 : Foundation

Migrate 10M+ conversation vectors from Pinecone to Milvus

Design and implement MongoDB schemas for real-time agent state

Set up Neo4j graph database with customer journey models

Create BigQuery data warehouse with partitioned tables

Phase 2 : Optimization

Build automated data quality monitoring system

Implement caching strategies (Redis) for 10x latency reduction

Optimize vector search queries (target :

Create real-time analytics dashboards (Grafana)

Phase 3 : ML Infrastructure

Build LLM fine-tuning data pipeline

Implement feature store for GNN training

Create A / B testing data infrastructure

Design multi-armed bandit state management

Work Environment

Collaborative team : Work with ML engineers, backend developers, and data scientists

Modern stack : Latest technologies and tools

Impact : Your work directly affects millions of marketing interactions

Autonomy : Own your projects end-to-end

Growth : Clear path to Senior / Lead / Principal roles

Create a job alert for this search

Data Engineer • Meerut, IN

Related jobs

Promoted

Python AWS Data Engineer

Digitrix Software LLPMeerut, IN

Mandatory skills - python , pyspark, who can write codes , any cloud exp - aws / gcp / azure.Python, AWS Python (core language skill) Backend, Pandas, PySpark (DataFrame API), interacting with AW...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

RevXMeerut, IN

RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

TalogyMeerut, IN

This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago

Promoted

Freelance Data Engineer

upGradMeerut, IN

We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 11 days ago

Promoted

Senior Data Engineer

AIQUMeerut, IN

We are hiring for Senior Data Engineer to join one of our major clients based out of KSA.Employment Type : Contract – 12 months & extendable. The Senior Data Engineer plays a lead role in designing, ...Show moreLast updated: 2 days ago

Promoted

Lead Data Engineer

Searce IncMeerut, IN

Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 22 days ago

Promoted

Data Engineer

Jaipur RugsMeerut, IN

Jaipur Rugs is a social enterprise that connects rural craftsmanship with global markets through its luxurious handmade carpets. It is a family-run business that offers an exclusive range of hand-kn...Show moreLast updated: 9 days ago

Promoted

Data Engineer - Scala

Idyllic ServicesMeerut, IN

Code Optimization , Shell Scripting, Data Engineering & Scala.Big Data tools like Spark, Hadoop & Hive.The Senior Data Engineer will be responsible for designing, developing, and maintaining scalab...Show moreLast updated: 11 days ago

Promoted

Data Engineer with Snowflake Experience- Contract

Gravity Infosolutions, Inc.Meerut, IN

Data Engineer with Snowflake & Data Vault 2.The project involves setting up a new software tool to meet audit requirements and is therefore critical. Most data to be migrated is already in Snowflake...Show moreLast updated: 1 day ago

Promoted
New!

Data Engineer – Azure & AWS

Yoda TechMeerut, IN

We are seeking a skilled and motivated Data Engineer to join our team and help build scalable, secure, and efficient data pipelines and platforms. The ideal candidate will have 2 to 4 years of hands...Show moreLast updated: 8 hours ago

Promoted

Data Engineer

ImpetusNoida, Uttar Pradesh, India

Strong ability to write clean and efficient code.Good understanding of Spark SQL for distributed data processing.Experience with large datasets and structured data manipulation.Ability to write que...Show moreLast updated: 4 days ago

Promoted

Data Engineer

thinkbridgemeerut, uttar pradesh, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

VeraxionMeerut, IN

Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 2 days ago

Promoted

Data Engineer

Randstad EnterpriseMeerut, IN

Shift Timing : 2 : 00 Pm - 11 : 00 Pm.Experience : 2- 4 years relevant Experience only ( this is a Junior position with us ). GCP - 2 years minimum working Experience.Worked with global stakeholders.Ran...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Kline + CompanyGhaziabad, IN

Here at Kline our data capabilities have grown exponentially over the last four years.Having gone through a rapid digitization process and becoming a cloud-native corporation, we are looking for to...Show moreLast updated: 27 days ago

Promoted

Data Engineer

AAA GlobalMeerut, IN

Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 12 days ago

Promoted

Data Engineer

IntraEdgeMeerut, IN

The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 30+ days ago

Promoted

AI / ML & Data Engineer

Mindfire SolutionsMeerut, IN

We are looking for an experienced AI / ML & Data Engineer to design, develop, and deploy scalable machine learning models and data infrastructure on AWS. You will work closely with cross-functional te...Show moreLast updated: 4 days ago

Promoted

GCP Data Engineer

EXLMeerut, IN

Must Have- GCP Data Engineer with Banking / Finance Institutions Experience.Google Cloud Platform (GCP) Engineers.This solution will integrate data from diverse sources including.SAS systems, Excel ...Show moreLast updated: 12 days ago

Promoted

Data Engineer

Response InformaticsMeerut, IN

We’re Hiring | Data Engineering Experts (Hyderabad / Remote).We’re looking for passionate and experienced.Data Engineering professionals. If you love building scalable data pipelines, optimizing clo...Show moreLast updated: 12 days ago