This job offer is not available in your country.
Data Engineer
OWOWPune, IN12 hours ago
Job descriptionBuild data collection pipelines for LLM fine-tuning (conversation logs, tool executions) Create feature stores for GNN training (customer interactions, engagement signals) Implement data versioning and lineage tracking for ML experiments Design A / B testing data infrastructure with CUPED variance reduction Build real-time feature computation pipelines for contextual bandits Design BigQuery schemas for marketing analytics and performance tracking Create materialized views and aggregation pipelines for real-time dashboards Implement data quality monitoring and anomaly detection Build observability infrastructure (Prometheus metrics, Grafana dashboards) Develop cost optimization strategies for cloud data warehousing MongoDB (conversation state, active sessions) Redis (caching, rate limiting, real-time data) Milvus (vector embeddings, semantic search) Neo4j (customer journey graphs, persona networks) BigQuery (analytics warehouse, historical data) Apache Airflow or Prefect (workflow orchestration) Pandas , Polars (data transformation) Apache Spark (optional - for large-scale processing) dbt (data transformation and modeling) vLLM (LLM inference serving) MLflow (model registry, experiment tracking) Sentence Transformers (embedding generation) PyTorch , TensorFlow (ML model training) Google Cloud Platform (BigQuery, Cloud Storage, Compute) Docker & Kubernetes (containerization, orchestration) Terraform (infrastructure as code) GitHub Actions or GitLab CI (CI / CD pipelines) Python 3.10+ (primary language) SQL (complex queries, query optimization) Shell scripting (Bash / Zsh) Git (version control) 5+ years of data engineering experience with production systems Expert-level SQL and database design skills Strong Python programming (async / await, type hints, testing) Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph) Proven track record building high-scale data pipelines (> Deep understanding of data modeling (dimensional, normalized, denormalized) Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Strong knowledge of data quality, validation, and governance Excellent debugging and optimization skills Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant) Experience with graph databases (Neo4j, ArangoDB, Neptune) Knowledge of embedding models and semantic search Experience with ML data pipelines (feature stores, model training data) Understanding of A / B testing and experimental design Experience with real-time streaming (Kafka, Pub / Sub, Kinesis) Knowledge of LLMs and conversational AI systems Experience with data migration projects (especially large-scale) Background in marketing technology or customer data platforms Experience with PyTorch Geometric or graph neural networks Knowledge of marketing analytics (attribution, segmentation, personalization) Familiarity with LangChain , LangGraph , or agent frameworks Experience with cost optimization in cloud environments Contributions to open-source data engineering projects Experience with data compliance (GDPR, CCPA) Migrate 10M+ conversation vectors from Pinecone to Milvus Design and implement MongoDB schemas for real-time agent state Set up Neo4j graph database with customer journey models Create BigQuery data warehouse with partitioned tables Build automated data quality monitoring system Implement caching strategies (Redis) for 10x latency reduction Optimize vector search queries (target : Create real-time analytics dashboards (Grafana) Build LLM fine-tuning data pipeline Implement feature store for GNN training Create A / B testing data infrastructure Design multi-armed bandit state management Collaborative team : Work with ML engineers, backend developers, and data scientists Modern stack : Latest technologies and tools Impact : Your work directly affects millions of marketing interactions Autonomy : Own your projects end-to-end Growth : Clear path to Senior / Lead / Principal roles
What You'll Build
Core Responsibilities
Data Architecture & Infrastructure (40%)
- Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
- Build scalable data pipelines for real-time conversation processing and personalization
- Architect ETL / ELT workflows for data migration from legacy systems
- Implement data partitioning, sharding, and optimization strategies for high-throughput systems
- Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
- Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
- Build graph schemas in Neo4j for customer journey mapping and persona relationships
- Implement HNSW indexing strategies and similarity search optimization
- Create hybrid search systems combining vector, full-text, and graph queries
- Monitor and tune database performance (query latency, throughput, resource utilization)
ML Data Infrastructure (20%)
Analytics & Monitoring (15%)
Technical Stack You'll Work With
Databases & Storage
Data Processing & Orchestration
ML / AI Data Pipeline
Cloud & Infrastructure
Programming & Tools
Requirements
Must-Have Skills
1M records / day)
Highly Desirable
Nice-to-Have
Key Projects You'll Own
Phase 1 : Foundation
Phase 2 : Optimization
Phase 3 : ML Infrastructure
Work Environment
Create a job alert for this search
Data Engineer • Pune, IN
Related jobs
- Promoted
Data Engineer
IntraEdgePune, IN The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer Lead
Calfus Inc.Pune, Maharashtra, India As a Data Engineer – BI Analytics & DWH, you will play a pivotal role in designing and implementing comprehensive business intelligence solutions that empower our organization to make data-driven d...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
V2Softpune, maharashtra, in Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala.
Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 5 days ago
- Promoted
Data Engineer
VAANTECHpune, maharashtra, in Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning.
Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago
- Promoted
Snowflake Data Engineer
Impactron Globalpune, maharashtra, in We’re seeking a Snowflake Data Engineer / Sr Data Engineer passionate about building scalable, cloud-native data solutions using Snowflake, dbt, and modern ETL frameworks.You will design, implement...Show moreLast updated: 5 days ago
- Promoted
Databricks Data Engineer
Persistent SystemsPune, Maharashtra, India We are seeking a Databricks Data engineer with AWS experience.Role : Databricks Data Engineer.Location : All Persistent Location.
Job Type : Full Time Employment.Must have knowledge and working experie...Show moreLast updated: 14 days ago
- Promoted
Data Engineer
KeasisPune, IN We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS / Hive / Impala) into Apache Iceberg tables, with dow...Show moreLast updated: 5 days ago
- Promoted
Lead Data Engineer
ITQube LTDPune, Maharashtra, India Looking for a Data Engineer to join our engineering will contribute directly to the design, automation, and optimization of our data processes, primarily developing solutions in Python within the A...Show moreLast updated: 15 days ago
- Promoted
Data Engineer
Manuh TechnologiesPune, IN S3, Glue, Redshift, EMR, Lambda).Develop automation scripts and tools using.Collaborate with data analysts, data scientists, and business stakeholders to ensure data availability and reliability.Tr...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
XebiaPune, IN We’re Hiring : Data Engineer | Xebia.Any Xebia location (Hybrid, 3 days office per week).Immediate to 2 weeks – only apply if you can join early.
Databricks, Python, SQL, and Postgres.The ideal candi...Show moreLast updated: 25 days ago
- Promoted
Data Engineer
Magma Consultancypune, maharashtra, in Part-time (20–25 hours per week).The ideal candidate has hands-on experience in building, optimizing, and maintaining data pipelines and architectures.
You’ll work closely with analysts, developers,...Show moreLast updated: 3 days ago
- Promoted
Lead Data Engineer
Searce IncPune, IN Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 15 days ago
- Promoted
Data Engineer
ITC InfotechPune, Maharashtra, India Maintenance and support of Data Pipeline.Data Validation and quality checks.Identify and optimize ingestion pipelines in consultation with Customer.
Investigate Glue job failures and restart.Fix min...Show moreLast updated: 15 days ago
- Promoted
Data Engineer
The Machine Learning CompanyPune, Maharashtra, India Job Description – Data Engineer (3–5 Years Experience).About The Machine Learning Company (TMLC).AI and Data solutions that drive measurable business impact.
From predictive analytics to intelligent...Show moreLast updated: 2 days ago
- Promoted
Data Engineer
EverisePune, IN Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 26 days ago
- Promoted
Data Engineer
INFEC Servicespune, maharashtra, in Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 24 days ago
- Promoted
Data Engineer
TalogyPune, IN This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
InfarsightPune, Maharashtra, India At Infarsight, we simplify complex operations across Finance, HR, Operations, and Supply Chain using platforms like Task Sight, Trip Sight, and Fleet Sight.
Our engineering teams build scalable soft...Show moreLast updated: 30+ days ago