No longer accepting applications
Data Engineer
OWOWbhopal, madhya pradesh, in8 days ago
Job descriptionBuild data collection pipelines for LLM fine-tuning (conversation logs, tool executions) Create feature stores for GNN training (customer interactions, engagement signals) Implement data versioning and lineage tracking for ML experiments Design A / B testing data infrastructure with CUPED variance reduction Build real-time feature computation pipelines for contextual bandits Design BigQuery schemas for marketing analytics and performance tracking Create materialized views and aggregation pipelines for real-time dashboards Implement data quality monitoring and anomaly detection Build observability infrastructure (Prometheus metrics, Grafana dashboards) Develop cost optimization strategies for cloud data warehousing MongoDB (conversation state, active sessions) Redis (caching, rate limiting, real-time data) Milvus (vector embeddings, semantic search) Neo4j (customer journey graphs, persona networks) BigQuery (analytics warehouse, historical data) Apache Airflow or Prefect (workflow orchestration) Pandas , Polars (data transformation) Apache Spark (optional - for large-scale processing) dbt (data transformation and modeling) vLLM (LLM inference serving) MLflow (model registry, experiment tracking) Sentence Transformers (embedding generation) PyTorch , TensorFlow (ML model training) Google Cloud Platform (BigQuery, Cloud Storage, Compute) Docker & Kubernetes (containerization, orchestration) Terraform (infrastructure as code) GitHub Actions or GitLab CI (CI / CD pipelines) Python 3.10+ (primary language) SQL (complex queries, query optimization) Shell scripting (Bash / Zsh) Git (version control) 5+ years of data engineering experience with production systems Expert-level SQL and database design skills Strong Python programming (async / await, type hints, testing) Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph) Proven track record building high-scale data pipelines (> Deep understanding of data modeling (dimensional, normalized, denormalized) Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Strong knowledge of data quality, validation, and governance Excellent debugging and optimization skills Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant) Experience with graph databases (Neo4j, ArangoDB, Neptune) Knowledge of embedding models and semantic search Experience with ML data pipelines (feature stores, model training data) Understanding of A / B testing and experimental design Experience with real-time streaming (Kafka, Pub / Sub, Kinesis) Knowledge of LLMs and conversational AI systems Experience with data migration projects (especially large-scale) Background in marketing technology or customer data platforms Experience with PyTorch Geometric or graph neural networks Knowledge of marketing analytics (attribution, segmentation, personalization) Familiarity with LangChain , LangGraph , or agent frameworks Experience with cost optimization in cloud environments Contributions to open-source data engineering projects Experience with data compliance (GDPR, CCPA) Migrate 10M+ conversation vectors from Pinecone to Milvus Design and implement MongoDB schemas for real-time agent state Set up Neo4j graph database with customer journey models Create BigQuery data warehouse with partitioned tables Build automated data quality monitoring system Implement caching strategies (Redis) for 10x latency reduction Optimize vector search queries (target : Create real-time analytics dashboards (Grafana) Build LLM fine-tuning data pipeline Implement feature store for GNN training Create A / B testing data infrastructure Design multi-armed bandit state management Collaborative team : Work with ML engineers, backend developers, and data scientists Modern stack : Latest technologies and tools Impact : Your work directly affects millions of marketing interactions Autonomy : Own your projects end-to-end Growth : Clear path to Senior / Lead / Principal roles
What You'll Build
Core Responsibilities
Data Architecture & Infrastructure (40%)
- Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
- Build scalable data pipelines for real-time conversation processing and personalization
- Architect ETL / ELT workflows for data migration from legacy systems
- Implement data partitioning, sharding, and optimization strategies for high-throughput systems
- Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
- Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
- Build graph schemas in Neo4j for customer journey mapping and persona relationships
- Implement HNSW indexing strategies and similarity search optimization
- Create hybrid search systems combining vector, full-text, and graph queries
- Monitor and tune database performance (query latency, throughput, resource utilization)
ML Data Infrastructure (20%)
Analytics & Monitoring (15%)
Technical Stack You'll Work With
Databases & Storage
Data Processing & Orchestration
ML / AI Data Pipeline
Cloud & Infrastructure
Programming & Tools
Requirements
Must-Have Skills
1M records / day)
Highly Desirable
Nice-to-Have
Key Projects You'll Own
Phase 1 : Foundation
Phase 2 : Optimization
Phase 3 : ML Infrastructure
Work Environment
Create a job alert for this search
Data Engineer • bhopal, madhya pradesh, in
Related jobs
- Promoted
Senior Data Engineer
AIQUbhopal, India We are hiring for Senior Data Engineer to join one of our major clients based out of KSA.Employment Type : Contract – 12 months & extendable.
The Senior Data Engineer plays a lead role in designing, ...Show moreLast updated: 2 days ago
- Promoted
Senior Data Engineer (AWS / Databricks)
Accoladebhopal, India The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
Jaipur Rugsbhopal, India Jaipur Rugs is a social enterprise that connects rural craftsmanship with global markets through its luxurious handmade carpets.
It is a family-run business that offers an exclusive range of hand-kn...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
RevXbhopal, India RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
Impetusbhopal, India Strong ability to write clean and efficient code.Good understanding of Spark SQL for distributed data processing.Experience with large datasets and structured data manipulation.Ability to write que...Show moreLast updated: 4 days ago
- Promoted
Data Engineer
Kline + Companybhopal, India Here at Kline our data capabilities have grown exponentially over the last four years.Having gone through a rapid digitization process and becoming a cloud-native corporation, we are looking for to...Show moreLast updated: 3 days ago
- Promoted
Data Engineer
Veraxionbhopal, India Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions.
Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 2 days ago
- Promoted
Azure Data Engineer
9NEXUSbhopal, India Job Title : Azure Data Engineer.We are seeking an experienced Azure Data Engineer to design, develop, and optimize scalable data solutions in a cloud environment.
The role involves building high-perf...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
V2Softbhopal, India Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala.
Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
Randstad Enterprisebhopal, India Shift Timing : 2 : 00 Pm - 11 : 00 Pm.Experience : 2- 4 years relevant Experience only ( this is a Junior position with us ).
GCP - 2 years minimum working Experience.Worked with global stakeholders.Ran...Show moreLast updated: 4 days ago
- Promoted
Data Engineer
Response Informaticsbhopal, India AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture.
CICD - Terraform or CloudFo...Show moreLast updated: 6 days ago
- Promoted
Freelance Data Engineer
upGradbhopal, India We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 6 days ago
- Promoted
Senior Data Engineer
USEReadybhopal, India Job Title : Senior Databricks Engineer.As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakeho...Show moreLast updated: 6 days ago
- Promoted
- New!
AWS Data Engineer
Tata Consultancy Servicesbhopal, madhya pradesh, in Aws data engineer having strong experience of Python.Technical / Behavioral Competency.Proficient in Python, with experience in deploying Python packages and OOP, Experience in ingesting data from di...Show moreLast updated: 1 hour ago
- Promoted
Data Engineer
Talogybhopal, India This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 6 days ago
- Promoted
Lead Data Engineer
ITQube LTDbhopal, India Looking for a Data Engineer to join our engineering will contribute directly to the design, automation, and optimization of our data processes, primarily developing solutions in Python within the A...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
thinkbridgebhopal, India We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses.
We differentiate ourselves by...Show moreLast updated: 3 days ago
- Promoted
Data Engineer
IntraEdgebhopal, India The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both interna...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
Trantorbhopal, India We are seeking a data engineer to design, implement, and optimize cloud-based data pipelines using Microsoft Azure services, including ADF, Synapse, and ADLS.
Develop and maintain ETL / ELT pipelines ...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
AAA Globalbhopal, India Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team.
In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 6 days ago