Platform Engineer

Job Responsibilities -

• Architect and implement a scalable, offline Data Lake for structured, semi-structured, and unstructured data in an on-premises, air-gapped environment.
• Collaborate with Data Engineers, Factory IT, and Edge Device teams to enable seamless data ingestion and retrieval across the platform.
• Integrate with upstream systems like MES, SCADA, and process tools to capture high-frequency manufacturing data efficiently.
• Monitor and maintain system health, including compute resources, storage arrays, disk I / O, memory usage, and network throughput.
• Optimize Data Lake performance via partitioning, deduplication, compression (Parquet / ORC), and implementing effective indexing strategies.
• Select, integrate, and maintain tools like Apache Hadoop, Spark, Hive, HBase, and custom ETL pipelines suitable for offline deployment.
• Build custom ETL workflows for bulk and incremental data ingestion using Python, Spark, and shell scripting.
• Implement data governance policies covering access control, retention periods, and archival procedures with security and compliance in mind.
• Establish and test backup, failover, and disaster recovery protocols specifically designed for offline environments.
• Document architecture designs, optimization routines, job schedules, and standard operating procedures (SOPs) for platform maintenance.
• Conduct root cause analysis for hardware failures, system outages, or data integrity issues.
• Drive system scalability planning for multi-fab or multi-site future expansions.

Essential Attributes (Tech-Stacks) -

Qualifications  -

Desired Experience Level -

Skills Required

Python, Data Lake, Hadoop, Hbase, Yarn, Zookeeper