No longer accepting applications
Data Engineer
OWOWChennai, IN3 days ago
Job descriptionBuild data collection pipelines for LLM fine-tuning (conversation logs, tool executions) Create feature stores for GNN training (customer interactions, engagement signals) Implement data versioning and lineage tracking for ML experiments Design A / B testing data infrastructure with CUPED variance reduction Build real-time feature computation pipelines for contextual bandits Design BigQuery schemas for marketing analytics and performance tracking Create materialized views and aggregation pipelines for real-time dashboards Implement data quality monitoring and anomaly detection Build observability infrastructure (Prometheus metrics, Grafana dashboards) Develop cost optimization strategies for cloud data warehousing MongoDB (conversation state, active sessions) Redis (caching, rate limiting, real-time data) Milvus (vector embeddings, semantic search) Neo4j (customer journey graphs, persona networks) BigQuery (analytics warehouse, historical data) Apache Airflow or Prefect (workflow orchestration) Pandas , Polars (data transformation) Apache Spark (optional - for large-scale processing) dbt (data transformation and modeling) vLLM (LLM inference serving) MLflow (model registry, experiment tracking) Sentence Transformers (embedding generation) PyTorch , TensorFlow (ML model training) Google Cloud Platform (BigQuery, Cloud Storage, Compute) Docker & Kubernetes (containerization, orchestration) Terraform (infrastructure as code) GitHub Actions or GitLab CI (CI / CD pipelines) Python 3.10+ (primary language) SQL (complex queries, query optimization) Shell scripting (Bash / Zsh) Git (version control) 5+ years of data engineering experience with production systems Expert-level SQL and database design skills Strong Python programming (async / await, type hints, testing) Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph) Proven track record building high-scale data pipelines (> Deep understanding of data modeling (dimensional, normalized, denormalized) Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Strong knowledge of data quality, validation, and governance Excellent debugging and optimization skills Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant) Experience with graph databases (Neo4j, ArangoDB, Neptune) Knowledge of embedding models and semantic search Experience with ML data pipelines (feature stores, model training data) Understanding of A / B testing and experimental design Experience with real-time streaming (Kafka, Pub / Sub, Kinesis) Knowledge of LLMs and conversational AI systems Experience with data migration projects (especially large-scale) Background in marketing technology or customer data platforms Experience with PyTorch Geometric or graph neural networks Knowledge of marketing analytics (attribution, segmentation, personalization) Familiarity with LangChain , LangGraph , or agent frameworks Experience with cost optimization in cloud environments Contributions to open-source data engineering projects Experience with data compliance (GDPR, CCPA) Migrate 10M+ conversation vectors from Pinecone to Milvus Design and implement MongoDB schemas for real-time agent state Set up Neo4j graph database with customer journey models Create BigQuery data warehouse with partitioned tables Build automated data quality monitoring system Implement caching strategies (Redis) for 10x latency reduction Optimize vector search queries (target : Create real-time analytics dashboards (Grafana) Build LLM fine-tuning data pipeline Implement feature store for GNN training Create A / B testing data infrastructure Design multi-armed bandit state management Collaborative team : Work with ML engineers, backend developers, and data scientists Modern stack : Latest technologies and tools Impact : Your work directly affects millions of marketing interactions Autonomy : Own your projects end-to-end Growth : Clear path to Senior / Lead / Principal roles
What You'll Build
Core Responsibilities
Data Architecture & Infrastructure (40%)
- Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
- Build scalable data pipelines for real-time conversation processing and personalization
- Architect ETL / ELT workflows for data migration from legacy systems
- Implement data partitioning, sharding, and optimization strategies for high-throughput systems
- Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
- Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
- Build graph schemas in Neo4j for customer journey mapping and persona relationships
- Implement HNSW indexing strategies and similarity search optimization
- Create hybrid search systems combining vector, full-text, and graph queries
- Monitor and tune database performance (query latency, throughput, resource utilization)
ML Data Infrastructure (20%)
Analytics & Monitoring (15%)
Technical Stack You'll Work With
Databases & Storage
Data Processing & Orchestration
ML / AI Data Pipeline
Cloud & Infrastructure
Programming & Tools
Requirements
Must-Have Skills
1M records / day)
Highly Desirable
Nice-to-Have
Key Projects You'll Own
Phase 1 : Foundation
Phase 2 : Optimization
Phase 3 : ML Infrastructure
Work Environment
Create a job alert for this search
Data Engineer • Chennai, IN
Related jobs
- Promoted
Data Engineer
Impetuschennai, tamil nadu, in Strong ability to write clean and efficient code.Good understanding of Spark SQL for distributed data processing.Experience with large datasets and structured data manipulation.Ability to write que...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Response Informaticschennai, India AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture.
CICD - Terraform or CloudFo...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
Cleantech Industry ResourcesChennai, Tamil Nadu, India Cleantech Industry Resources accelerates United States solar, battery storage and EV projects by providing turnkey development as a service including 100% internal systems engineering.The company d...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
TalogyChennai, IN This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago
- Promoted
- New!
Data Engineer
GEMRAJ TECHNOLOGIES LIMITEDChennai, IN We are currently seeking a Data Engineer to join our technology team.In this role, you will work on building data-driven products that sit atop cloud-based data infrastructure.The ideal candidate i...Show moreLast updated: 15 hours ago
- Promoted
Data Engineer
EXLChennai, Tamil Nadu, India Collaborate with project stakeholders (client) to identify product and technical requirements.Develop, implement, and tune large-scale distributed systems and pipelines that process large volume of...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Trantormount, India We are seeking a data engineer to design, implement, and optimize cloud-based data pipelines using Microsoft Azure services, including ADF, Synapse, and ADLS.
Develop and maintain ETL / ELT pipelines ...Show moreLast updated: 1 day ago
- Promoted
Lead Data Engineer
ITQube LTDchennai, tamil nadu, in Looking for a Data Engineer to join our engineering will contribute directly to the design, automation, and optimization of our data processes, primarily developing solutions in Python within the A...Show moreLast updated: 18 days ago
- Promoted
Data Engineer
KaarTechChennai, Tamil Nadu, India Data Pipeline Development : Design, develop, and maintain scalable data pipelines to support various data analytics and reporting needs.
Data Integration : Integrate data from multiple sources and ens...Show moreLast updated: 16 days ago
- Promoted
Data Engineer
AAA GlobalChennai, IN Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team.
In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 8 days ago
- Promoted
Data Engineer
V2Softchennai, tamil nadu, in Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala.
Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 8 days ago
- Promoted
Data Engineer
RevXmount, India RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
INFEC Serviceschennai, tamil nadu, in Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 27 days ago
- Promoted
Data / ML Engineer
Lilochennai, tamil nadu, in Lilo is revolutionizing procurement for Commercial Real Estate (CRE) businesses by providing a single platform to automate and optimize procurement processes using AI.
From hotels to gyms, schools, ...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
ConfidentialChennai, Delhi, Bengaluru / Bangalore Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Data bricks.
Develop complex ETL processe...Show moreLast updated: 26 days ago
- Promoted
Data Engineer
VAANTECHChennai, Tamil Nadu, India Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning.
Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Randstad EnterpriseChennai, IN Shift Timing : 2 : 00 Pm - 11 : 00 Pm.Experience : 2- 4 years relevant Experience only ( this is a Junior position with us ).
GCP - 2 years minimum working Experience.Worked with global stakeholders.Ran...Show moreLast updated: 30+ days ago
- Promoted
Lead Data Engineer
Searce IncChennai, IN Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 18 days ago
- Promoted
Data Engineer
IntraEdgeChennai, IN The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Bahwan CyberTekchennai, tamil nadu, in We are seeking a skilled and motivated Data Engineer with 5–8 years of experience in building scalable data pipelines using Python, PySpark, and AWS services.
The ideal candidate will have hands-on ...Show moreLast updated: 30+ days ago