Data EngineerOWOW • Hosur, Tamil Nadu, India
No longer accepting applications
Data Engineer
OWOW • Hosur, Tamil Nadu, India
30+ days ago
Job descriptionBuild data collection pipelines for LLM fine-tuning (conversation logs, tool executions) Create feature stores for GNN training (customer interactions, engagement signals) Implement data versioning and lineage tracking for ML experiments Design A / B testing data infrastructure with CUPED variance reduction Build real-time feature computation pipelines for contextual bandits Design BigQuery schemas for marketing analytics and performance tracking Create materialized views and aggregation pipelines for real-time dashboards Implement data quality monitoring and anomaly detection Build observability infrastructure (Prometheus metrics, Grafana dashboards) Develop cost optimization strategies for cloud data warehousing MongoDB (conversation state, active sessions) Redis (caching, rate limiting, real-time data) Milvus (vector embeddings, semantic search) Neo4j (customer journey graphs, persona networks) BigQuery (analytics warehouse, historical data) Apache Airflow or Prefect (workflow orchestration) Pandas , Polars (data transformation) Apache Spark (optional - for large-scale processing) dbt (data transformation and modeling) vLLM (LLM inference serving) MLflow (model registry, experiment tracking) Sentence Transformers (embedding generation) PyTorch , TensorFlow (ML model training) Google Cloud Platform (BigQuery, Cloud Storage, Compute) Docker & Kubernetes (containerization, orchestration) Terraform (infrastructure as code) GitHub Actions or GitLab CI (CI / CD pipelines) Python 3.10+ (primary language) SQL (complex queries, query optimization) Shell scripting (Bash / Zsh) Git (version control) 5+ years of data engineering experience with production systems Expert-level SQL and database design skills Strong Python programming (async / await, type hints, testing) Experience with at least 3 different database technologies (SQL, NoSQL, Vector, Graph) Proven track record building high-scale data pipelines (> Deep understanding of data modeling (dimensional, normalized, denormalized) Experience with cloud data warehouses (BigQuery, Redshift, or Snowflake) Strong knowledge of data quality, validation, and governance Excellent debugging and optimization skills Experience with vector databases (Milvus, Pinecone, Weaviate, Qdrant) Experience with graph databases (Neo4j, ArangoDB, Neptune) Knowledge of embedding models and semantic search Experience with ML data pipelines (feature stores, model training data) Understanding of A / B testing and experimental design Experience with real-time streaming (Kafka, Pub / Sub, Kinesis) Knowledge of LLMs and conversational AI systems Experience with data migration projects (especially large-scale) Background in marketing technology or customer data platforms Experience with PyTorch Geometric or graph neural networks Knowledge of marketing analytics (attribution, segmentation, personalization) Familiarity with LangChain , LangGraph , or agent frameworks Experience with cost optimization in cloud environments Contributions to open-source data engineering projects Experience with data compliance (GDPR, CCPA) Migrate 10M+ conversation vectors from Pinecone to Milvus Design and implement MongoDB schemas for real-time agent state Set up Neo4j graph database with customer journey models Create BigQuery data warehouse with partitioned tables Build automated data quality monitoring system Implement caching strategies (Redis) for 10x latency reduction Optimize vector search queries (target : Create real-time analytics dashboards (Grafana) Build LLM fine-tuning data pipeline Implement feature store for GNN training Create A / B testing data infrastructure Design multi-armed bandit state management Collaborative team : Work with ML engineers, backend developers, and data scientists Modern stack : Latest technologies and tools Impact : Your work directly affects millions of marketing interactions Autonomy : Own your projects end-to-end Growth : Clear path to Senior / Lead / Principal roles
What You'll Build
Core Responsibilities
Data Architecture & Infrastructure (40%)
- Design and implement a multi-database architecture (MongoDB, Redis, Milvus, Neo4j, BigQuery)
- Build scalable data pipelines for real-time conversation processing and personalization
- Architect ETL / ELT workflows for data migration from legacy systems
- Implement data partitioning, sharding, and optimization strategies for high-throughput systems
- Create data governance frameworks ensuring quality, security, and compliance Vector & Graph Database Systems (25%)
- Design and optimize Milvus vector collections for semantic search (1024-dim embeddings)
- Build graph schemas in Neo4j for customer journey mapping and persona relationships
- Implement HNSW indexing strategies and similarity search optimization
- Create hybrid search systems combining vector, full-text, and graph queries
- Monitor and tune database performance (query latency, throughput, resource utilization)
ML Data Infrastructure (20%)
Analytics & Monitoring (15%)
Technical Stack You'll Work With
Databases & Storage
Data Processing & Orchestration
ML / AI Data Pipeline
Cloud & Infrastructure
Programming & Tools
Requirements
Must-Have Skills
1M records / day)
Highly Desirable
Nice-to-Have
Key Projects You'll Own
Phase 1 : Foundation
Phase 2 : Optimization
Phase 3 : ML Infrastructure
Work Environment
Create a job alert for this search
Data Engineer • Hosur, Tamil Nadu, India
Related jobs
Data Engineer
TerraGiG • hosur, tamil nadu, in
Lead the design, development, and implementation of data solutions using AWS and Snowflake.Collaborate with cross-functional teams to understand business requirements and translate them into techni...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Synechron • Hosur, Tamil Nadu, India
We have immediate opportunity for AWS Big Data Engineer Job Role : AWS Big Data Engineer Job Location : Bangalore Experience- 7+Years About Company : At Synechron, we believe in the power of di...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
People Prime Worldwide • Hosur, Tamil Nadu, India
About Client : - Our client is a French multinational information technology (IT) services and consulting company, headquartered in Paris, France.
Founded in 1967, It has been a leader in business tr...Show more
Last updated: 30+ days ago • Promoted
Data AI Engineer
Verdantas • Hosur, Tamil Nadu, India
Join Verdantas – A Top #ENR 81 Firm! Position : Data AI Engineer Key Responsibilities : Your duties will include but are not limited to the following : Architect and develop AI-powered agents using ...Show more
Last updated: 8 hours ago • Promoted • New!
Data Engineer
Arete Healthtech Pvt. Ltd. • Hosur, Tamil Nadu, India
Job Position : Data Engineer Location : Delhi (Work from Office) CTC Range : As per the Industry Standards Years of Experience : 1-3 Years About the Role We are looking for a skilled and detail...Show more
Last updated: 8 hours ago • Promoted • New!
Data Engineer
Grantify • hosur, tamil nadu, in
Grantify is an innovative education platform that connects students and universities through a transparent admissions and tuition-matching ecosystem.
By aligning student budgets and academic aspirat...Show more
Last updated: 3 days ago • Promoted
Freelance Data Quality Engineer
Leading MNC • hosur, tamil nadu, in
Freelance Data Quality Engineer.The candidate should have a minimum of 8+ yrs.If you're looking for freelance / part time opportunity (along with your day job) & a chance to work with the top 0.You ...Show more
Last updated: 27 days ago • Promoted
Data Engineer
MRF • Hosur, Tamil Nadu, India
Azure / SQL / Application - Data Engineer Job description : Responsible to maintain the data required for managed application like Advanced Planning System, Dealer Management System, etc.Responsible fo...Show more
Last updated: 8 hours ago • Promoted • New!
Senior Data Engineer
Sonatype • Hosur, Tamil Nadu, India
Sonatype is the software supply chain security company.We provide the world’s best end-to-end software supply chain security solution, combining the only proactive protection against malicious open...Show more
Last updated: 8 hours ago • Promoted • New!
AWS Data Engineer
9NEXUS • Hosur, Tamil Nadu, India
Job Title : Senior AWS Data Engineer Experience : 6 - 9 Years Location : Remote Job Description We are seeking a Senior AWS Data Engineer to design, build, and optimize large-scale data pipelines and...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
AS Technology Corporation • Hosur, Tamil Nadu, India
We are seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and data infrastructure solutions.
This role involves working with cloud platforms, big data framew...Show more
Last updated: 30+ days ago • Promoted
Senior Databricks Data Engineer
Squash Apps • Hosur, Tamil Nadu, India
Company Description Squash Apps is a top-rated full-stack consulting company dedicated to building the next generation of scalable and robust web applications for visionary clients.We specialize in...Show more
Last updated: 6 hours ago • Promoted • New!
Senior Data Engineer
Globant • Hosur, Tamil Nadu, India
At Globant, we are working to make the world a better place, one step at a time.We enhance business development and enterprise solutions to prepare them for a digital future.With a diverse and tale...Show more
Last updated: 8 hours ago • Promoted • New!
Big Data Engineer
K&K Talents - India • hosur, tamil nadu, in
This position is with one of our.Title : Big Data Engineer (Scala & Spark).Required Experience : 8 Years and above.We are seeking a highly skilled.
Azure cloud, Big Data technologies, and distributed ...Show more
Last updated: 12 hours ago • Promoted • New!
Senior Data Engineer
Synechron • Hosur, Tamil Nadu, India
Good-day, We have opprotunity for AWS Data Engineer.Job Role : AWS Data Engineer Job Location : Synechron ( Bengaluru) Experience- 7 to 12 years Notice : Immediate joiner.About Company : At Synechr...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineer
CXC • hosur, tamil nadu, in
Please apply only if you are available to work in Australian time zone and comfortable with 6 months contract duration • •.
We’re seeking a highly skilled and autonomous.Power BI implementations to jo...Show more
Last updated: 1 day ago • Promoted
Senior Data Engineer
Primesoft Inc • hosur, tamil nadu, in
Primesoft Enterprise IT Services Pvt.As a Software Engineer II - Data, you will contribute to the design and development of data systems including pipelines, APIs, analytics, AI and machine learnin...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
BLJ Tech Geeks • Hosur, Tamil Nadu, India
Job Title : Data Engineer Company - Big4 (PERMANENT) Location : Bangalore (Hybrid) Job Type : Full-time Experience Level : 5Yrs above Notice Period - Serving only till 31st Dec not more than this A...Show more
Last updated: 8 hours ago • Promoted • New!