This job offer is not available in your country.
Data Engineer
OWOWHyderabad, IN1 day ago
Job descriptionBuild data collection pipelines for LLM fine-tuning (conversation logs, tool executions) Create feature stores for GNN training (customer interactions, engagement signals) Implement data versioning and lineage tracking for ML experiments Design A / B testing data infrastructure with CUPED variance reduction Build real-time feature computation pipelines for contextual bandits Design BigQuery schemas for marketing analytics and performance tracking Create materialized views and aggregation pipelines for real-time dashboards Implement data quality monitoring and anomaly detection Build observability infrastructure (Prometheus metrics, Grafana dashboards) Develop cost optimization strategies for cloud data warehousing MongoDB (conversation state, active sessions) Redis (caching, rate limiting, real-time data) Milvus (vector embeddings, semantic search) Neo4j (customer journey graphs, persona networks) BigQuery (analytics warehouse, historical data) Apache Airflow or Prefect (workflow orchestration) Pandas , Polars (data transformation) Apache Spark (optional - for large-scale processing) dbt (data transformation and modeling) vLLM (LLM inference serving) MLflow (model registry, experiment tracking) Sentence Transformers (embedding generation) PyTorch , TensorFlow (ML model training) Google Cloud Platform (BigQuery, Cloud Storage, Compute) Docker & Kubernetes (containerization, orchestration) Terraform (infrastructure as code) GitHub Actions or GitLab CI (CI / CD pipelines) Python 3.10+ (primary language) SQL (complex queries, query optimization) Shell scripting (Bash / Zsh) Git (version control) 5+ years of data engineering experience with production systems Expert-level SQL and database design skills Strong Python programming (async / await, type hints, testing) Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph) Proven track record building high-scale data pipelines (> Deep understanding of data modeling (dimensional, normalized, denormalized) Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Strong knowledge of data quality, validation, and governance Excellent debugging and optimization skills Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant) Experience with graph databases (Neo4j, ArangoDB, Neptune) Knowledge of embedding models and semantic search Experience with ML data pipelines (feature stores, model training data) Understanding of A / B testing and experimental design Experience with real-time streaming (Kafka, Pub / Sub, Kinesis) Knowledge of LLMs and conversational AI systems Experience with data migration projects (especially large-scale) Background in marketing technology or customer data platforms Experience with PyTorch Geometric or graph neural networks Knowledge of marketing analytics (attribution, segmentation, personalization) Familiarity with LangChain , LangGraph , or agent frameworks Experience with cost optimization in cloud environments Contributions to open-source data engineering projects Experience with data compliance (GDPR, CCPA) Migrate 10M+ conversation vectors from Pinecone to Milvus Design and implement MongoDB schemas for real-time agent state Set up Neo4j graph database with customer journey models Create BigQuery data warehouse with partitioned tables Build automated data quality monitoring system Implement caching strategies (Redis) for 10x latency reduction Optimize vector search queries (target : Create real-time analytics dashboards (Grafana) Build LLM fine-tuning data pipeline Implement feature store for GNN training Create A / B testing data infrastructure Design multi-armed bandit state management Collaborative team : Work with ML engineers, backend developers, and data scientists Modern stack : Latest technologies and tools Impact : Your work directly affects millions of marketing interactions Autonomy : Own your projects end-to-end Growth : Clear path to Senior / Lead / Principal roles
What You'll Build
Core Responsibilities
Data Architecture & Infrastructure (40%)
- Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
- Build scalable data pipelines for real-time conversation processing and personalization
- Architect ETL / ELT workflows for data migration from legacy systems
- Implement data partitioning, sharding, and optimization strategies for high-throughput systems
- Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
- Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
- Build graph schemas in Neo4j for customer journey mapping and persona relationships
- Implement HNSW indexing strategies and similarity search optimization
- Create hybrid search systems combining vector, full-text, and graph queries
- Monitor and tune database performance (query latency, throughput, resource utilization)
ML Data Infrastructure (20%)
Analytics & Monitoring (15%)
Technical Stack You'll Work With
Databases & Storage
Data Processing & Orchestration
ML / AI Data Pipeline
Cloud & Infrastructure
Programming & Tools
Requirements
Must-Have Skills
1M records / day)
Highly Desirable
Nice-to-Have
Key Projects You'll Own
Phase 1 : Foundation
Phase 2 : Optimization
Phase 3 : ML Infrastructure
Work Environment
Create a job alert for this search
Data Engineer • Hyderabad, IN
Related jobs
- Promoted
Lead Data Engineer
Searce IncHyderabad, IN Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’.It signifies our way of working : To improve to the finest degree of excellence, ‘solving for better’ every time.Searcians...Show moreLast updated: 16 days ago
- Promoted
Data Engineer
OptumHyderabad, Telangana, India Support the full data engineering lifecycle including research, proof of concepts, design, development, testing, deployment, and maintenance of data management solutions.
Utilize knowledge of variou...Show moreLast updated: 5 days ago
- Promoted
Data Engineer
Response InformaticsSecunderabad, Telangana, India Cloud & Infrastructure • AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.
Snowflake : Moderate understanding of Snowflake architecture.CI...Show moreLast updated: 16 days ago
- Promoted
Data Engineer
EveriseHyderabad, IN Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 27 days ago
- Promoted
Data Engineer
Tanla Platforms LimitedHyderabad, Telangana, India We are seeking a skilled Data Engineer with 5–10 years of experience to join our dynamic Data Engineering team.The ideal candidate will design, build, and optimize large-scale data pipelines and wa...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Divamihyderabad, telangana, in We are now building our data capabilities to.You will collaborate closely with engineering, product, and business teams to ensure data is.
Collaborate with cross-functional teams to define.Postgres,...Show moreLast updated: 30+ days ago
- Promoted
Softobiz - Data Engineer - Synapse Analytics
Softobiz Technologies Pvt ltdHyderabad Description : Job Title : Data Engineer.Career-defining role available with Tier 1 Blue Chip Australian Client.As a Data Engineer, you will be involved in all phases...Show moreLast updated: 11 days ago
- Promoted
Lead Data Engineer
EverestDX IncHyderabad, Telangana, India Everest DX – We are a Digital Platform Services company, headquartered in Stamford.Our Platform / Solution includes Orchestration, Intelligent operations with BOTs’, AI-powered analytics for Enterpri...Show moreLast updated: 15 days ago
- Promoted
Data Engineer
TalogyHyderabad, IN This opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines.This role requires...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India Seeking an experienced Senior Data Engineer with over 9 years of industry experience to architect, build, and manage advanced data pipelines and platforms in a cloud environment.The ideal candidate...Show moreLast updated: 15 days ago
- Promoted
Data Engineer
OWOWhyderabad, telangana, in Data Architecture & Infrastructure (40%) .Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery) .
Build scalable data pipelines for real-time conversation proc...Show moreLast updated: 1 day ago
- Promoted
Data Engineer
Bahwan CyberTeksecunderabad, telangana, in We are seeking a skilled and motivated Data Engineer with 5–8 years of experience in building scalable data pipelines using Python, PySpark, and AWS services.
The ideal candidate will have hands-on ...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
CoforgeHyderabad, Telangana, India Job Title - Senior Data Engineer Job Description - - Bachelor's degree in computer science, Information Systems, or related field.
Proven expertise in Greenplum (PostgreSQL-based MPP), AWS (S3, EC...Show moreLast updated: 28 days ago
- Promoted
Data Engineer
Manuh TechnologiesHyderabad, IN S3, Glue, Redshift, EMR, Lambda).Develop automation scripts and tools using.Collaborate with data analysts, data scientists, and business stakeholders to ensure data availability and reliability.Tr...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
AAA GlobalHyderabad, IN Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team.
In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
SoftobizHyderabad, Telangana, India Career-defining role available with Tier 1 Blue Chip Australian Client.As a Data Engineer, you will be involved in all phases of the software development life cycle, collecting and documenting busi...Show moreLast updated: 5 days ago
- Promoted
Data Engineer- Snowflake
Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India We are seeking a skilled Data Engineer with strong experience in Python, Snowflake, and AWS.The ideal candidate will be responsible for building and optimizing scalable data pipelines, integrating ...Show moreLast updated: 19 days ago
- Promoted
Data Engineer
XebiaHyderabad, IN We’re Hiring : Data Engineer | Xebia.Any Xebia location (Hybrid, 3 days office per week).Immediate to 2 weeks – only apply if you can join early.
Databricks, Python, SQL, and Postgres.The ideal candi...Show moreLast updated: 26 days ago
- Promoted
Data Engineer
V2Softsecunderabad, telangana, in Experience to deal with large stream volumes.Implementation experience ( atleast couple of projects ) in MSK / Flink / Spark / Scala.
Very good knowledge on atleast 3 out of the 4 technologies.Desirable e...Show moreLast updated: 5 days ago
- Promoted
Data Engineer
INFEC ServicesHyderabad, India Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 30+ days ago