Senior Data Platform Engineer
CogniteBengaluru, Republic Of India, IN24 days ago
Job descriptionEngineer Spark applications with a deep understanding of Catalyst optimiser, partitioning strategies, and performance tuning Develop real-time streaming solutions processing millions of events per second with Kafka and Flink. Design efficient data lake architectures using S3 / GCS with optimised partitioning and file formats (Parquet, ORC). Implement query optimisation techniques for OLAP datastores like ClickHouse, Pinot, or Druid. Scale systems to 10K+ QPS while maintaining high availability and data consistency. Optimise JVM performance through garbage collection tuning and memory management. Implement comprehensive monitoring using Prometheus, Grafana, and distributed tracing. Design fault-tolerant architectures with proper circuit breakers and retry mechanisms. Contribute to open-source projects in the big data ecosystem (Spark, Kafka, Airflow). Research and prototype new technologies for industrial data challenges. Collaborate with product teams to translate complex requirements into scalable technical solutions. Participate in architectural reviews and technical design discussions. Distributed Systems Experience (4-6 years) - Production Spark experience - built and optimised large-scale Spark applications with understanding of internals - Streaming systems proficiency - implemented real-time data processing using Kafka, Flink, or Spark Streaming - JVM Language expertise - strong programming skills in Java, Scala, or Kotlin with performance optimisation experience. Data Platform Foundations (3+ years) - Big data storage systems - hands-on experience with data lakes, columnar formats, and table formats (Iceberg, Delta Lake) - OLAP query engines - worked with Presto / Trino, ClickHouse, Pinot, or similar high-performance analytical databases - ETL / ELT pipeline development - built robust data transformation pipelines using tools like DBT, Airflow, or custom frameworks Infrastructure and Operations - Kubernetes production experience -deployed and operated containerised applications in production environments. Cloud platform proficiency - hands-on experience with AWS, Azure, or GCP data services. Monitoring and observability - implemented comprehensive logging, metrics, and alerting for data systems. Performance Engineering - System optimisation experience - delivered measurable performance improvements (2x+ throughput gains). Resource efficiency - optimised systems for cost while maintaining performance requirements. Concurrency expertise - designed thread-safe, high-concurrency data processing systems. Data Engineering Best Practices - Data quality frameworks -implemented validation, testing, and monitoring for data pipelines. Schema evolution - managed backwards-compatible schema changes in production systems. Data modelling expertise - designed efficient schemas for analytical workloads. Technical Collaboration - Cross-functional partnership - worked effectively with product managers, ML engineers, and data scientists. Codereview excellence - provided thoughtful technical feedback and maintained high code quality standards. Documentation and knowledge sharing - created technical documentation and participated in knowledge transfer. Continuous Learning - Technology adoption - quickly learned and applied new technologies to solve business problems. Industry awareness - stayed current with big data ecosystem developments and best practices. Problem-solving approach - demonstrated a systematic approach to debugging complex distributed system issues. Execution Excellence - Rapid delivery - consistently shipped high-quality features within aggressive timelines. Technical pragmatism - made smart trade-offs between technical debt, velocity, and system reliability. End-to-end ownership - took responsibility for features from design through production deployment and monitoring. Ambiguity comfort - thrived in environments with evolving requirements and unclear specifications. Technology flexibility - adapted to new tools and frameworks based on project needs. Customer focus - understood how technical decisions impact user experience and business metrics. Open-source contributions to major Apache projects in the data space (e. g. Apache Spark or Kafka) are a big plus. Conference speaking or technical blog writing experience, Industrial domain knowledge - previous experience with IoT, manufacturing, or operational technology systems. Languages : Kotlin, Scala, Python, Java. Big Data : Apache Spark, Apache Flink, Apache Kafka. Storage : PostgreSQL, ClickHouse, Elasticsearch, S3-compatible systems. Infrastructure : Kubernetes, Docker, Terraform. Table Formats : Apache Iceberg, Delta Lake, Apache Hudi. Query Engines : Trino / Presto, Apache Pinot, DuckDB. Orchestration : Apache Airflow, Dagster. Monitoring : Prometheus, Grafana, Jaeger, ELK Stack.
Cognite is revolutionising industrial data management through our flagship product, Cognite Data Fusion - a state-of-the-art SaaS platform that transforms how industrial companies leverage their data. We're seeking a Senior Data Platform Engineer who excels at building high-performance distributed systems and thrives in a fast-paced startup environment. You'll be working on cutting-edge data infrastructure challenges that directly impact how Fortune 500 industrial companies manage their most critical operational data.
Responsibilities :
- High-Performance Data Systems : Design and implement robust data processing pipelines using Apache Spark, Flink, and Kafka for terabyte-scale industrial datasets.
- Build efficient APIs and services that serve thousands of concurrent users with sub-second response times.
- Optimise data storage and retrieval patterns for time-series, sensor, and operational data.
- Implement advanced caching strategies using Redis and in-memory data structures.
Distributed Processing Excellence :
Scalability and Performance :
Technical Innovation :
Requirements :
Technical Depth Indicators :
Collaboration and Growth :
Startup Mindset :
Bonus Points :
Technical Stack :
Primary Technologies :
Technologies You May Work With :
Create a job alert for this search
Senior Data Engineer • Bengaluru, Republic Of India, IN
Related jobs
NASDAQ : CMPR) is the world leader in mass customisation.Our unmatched technology, production, and supply chain operations allow us to offer products that can be personalised by an individual custom...Show moreLast updated: 22 days ago
- Promoted
Senior Data Engineer
Straivehosur, tamil nadu, in The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show moreLast updated: 30+ days ago
- Promoted
Data Platform Engineer
Grizmo LabsBengaluru, Karnataka, India Design, build, and maintain scalable data platforms and pipelines to support the needs of the business.Ensure data quality and accuracy by implementing data validation and testing processes.Partici...Show moreLast updated: 24 days ago
- Promoted
Senior Data Platform Engineer
Black Dog LabsBangalore, IN Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps.
We’re looking for a hands-on Senior Data Platform...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Recrohosur, tamil nadu, in Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake.
AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
thinkbridgehosur, tamil nadu, in We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses.
We differentiate ourselves by...Show moreLast updated: 30+ days ago
- Promoted
- New!
Senior Platform Engineer
Vista Applied Solutions Group Inchosur, tamil nadu, in Client is looking for Senior Platform Engineer and this is remote position from India.Design, build, and manage scalable Kubernetes platforms and automation using Terraform.Support GitOps workflows...Show moreLast updated: 22 hours ago
- Promoted
Senior Data Engineer (AWS / Databricks)
Accoladebangalore district, karnataka, in The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer
Incept Labshosur, tamil nadu, in Incept Labs is an AI research lab based in San Francisco, California.Our small team of scientists, engineers, and builders is passionate about developing domain-specific, next-generation AI solutio...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer-Remote
Cimpress Indiahosur, tamil nadu, inRemote
- Promoted
Data Engineer
Insight Globalhosur, tamil nadu, in GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts.
Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer
Guidanz IncBangalore, IN BI Connector is the industry leading solution for integrating Oracle Fusion Cloud data into modern BI platforms like Power BI, Tableau, and Data Warehouse, without complex ETL.Our Data Architecture...Show moreLast updated: 3 days ago
- Promoted
Senior Data Engineer
SAIVA AIhosur, tamil nadu, in We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer
HealthyMe AIhosur, tamil nadu, in We are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data.You will report to the Head of AI and Engineering and be responsible for designi...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Integration Engineer
Indsafrihosur, tamil nadu, in Senior Data Integration Engineer.We are seeking a highly experienced and strategic Senior Data Integration Engineer to architect, build, and manage the data pipelines that power our customer intell...Show moreLast updated: 23 days ago
- Promoted
Senior Data Engineer
Mastekhosur, tamil nadu, in We are looking for a Senior Data Engineer with good exposure to Cloud Services in AWS(IAM, RDS, Lambda, DMS) along with Snowflake / Postgresql, ETL and Streaming Services like Kafka.Show moreLast updated: 30+ days ago
- Promoted
Platform Engineer-Data
ValueLabsBengaluru, Karnataka, India We’re Hiring : Senior Technology Engineer – DevOps & Big Data.Senior Technology Engineer (STE).Build robust CI / CD pipelines using Jenkins & GitOps.
Deploy applications using Kubernetes & OpenShift fo...Show moreLast updated: 24 days ago
- Promoted
Senior Data Engineer
MeghGen TechnologiesBengaluru, Karnataka, India MeghGen Technologies is a trusted technology solutions partner specializing in data engineering, cloud-native applications, and cloud migrations.
We design and implement end-to-end cloud solutions t...Show moreLast updated: 3 days ago
- Promoted
Senior Azure Data Engineer
Programmers.iohosur, tamil nadu, in Job Title : Senior Azure Data Engineer.Skills Required : Azure Synapse, Azure Data Factory, Azure Databricks, and ADLS Gen2, PySpark, SQL,Microsoft Fabrics.
Experience Range : 5 Years - 12 years.We are...Show moreLast updated: 30+ days ago