Data Engineer

Data Engineer – NiFi / Cloudera / Iceberg / Snowflake / Databricks

Overview

We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS / Hive / Impala) into Apache Iceberg tables, with downstream integration into Snowflake and Databricks. The ideal candidate will have hands-on experience with modern data lakehouse architectures and will play a critical role in enabling scalable, governed, and high-performance data platforms.

Key Responsibilities :

• Data Ingestion & Pipeline Development
• Design, configure, and maintain NiFi data flows to extract, transform, and load data from Cloudera into Iceberg tables.
• Implement streaming and batch ingestion pipelines with NiFi processors and custom scripting where needed.
• Optimize NiFi workflows for scalability, reliability, and monitoring.
• Data Lakehouse Enablement
• Build and manage Apache Iceberg-based datasets for structured, semi-structured, and unstructured data.
• Ensure schema evolution, partitioning, and metadata management in Iceberg.
• Develop integration flows from Iceberg to Snowflake and Databricks for analytics, ML, and reporting use cases.
• Integration & Orchestration
• Work with Snowflake to ingest curated data from Iceberg for enterprise reporting and commercial insights.
• Collaborate with Databricks teams to enable advanced analytics and machine learning use cases.
• Integrate NiFi pipelines with orchestration tools (Airflow, Oozie, or AWS / Azure / GCP schedulers).
• Performance, Security & Governance
• Tune NiFi flows and Snowflake / Databricks ingestion for performance and cost optimization.
• Implement role-based security and ensure compliance (HIPAA, GDPR, SOX if applicable).
• Work with governance teams to enable lineage, metadata tracking, and auditability.

Qualifications :