Freelance Apache NiFi + GCP Data Engineer

Company Description

ThreatXIntel is a startup cybersecurity company specializing in protecting businesses and organizations from cyber threats. We offer services including cloud security, web and mobile security testing, DevSecOps, and cloud security assessments to deliver customized, affordable solutions tailored to client needs. Committed to proactive security, we continuously monitor and test digital environments to mitigate vulnerabilities before they can be exploited. Our mission is to empower businesses of all sizes with high-quality cybersecurity services to protect their digital assets and enable business growth.

Role Description

We are seeking an experienced Freelance Data Engineer with strong expertise in Apache NiFi and the Google Cloud Platform (GCP) ecosystem. The consultant will design and implement real-time data ingestion pipelines, integrate with GCP services, and ensure secure, scalable, and high-performance data delivery for analytics and downstream platforms.

Key Responsibilities

Design and build real-time ingestion pipelines using Apache NiFi for structured and unstructured datasets.

Integrate NiFi pipelines with the GCP ecosystem , including BigQuery, Pub / Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow) .

Implement low-latency, high-throughput data movement across ingestion, storage, and processing layers.

Develop and optimize SQL queries for BigQuery and relational systems.

Build Python scripts for data transformations, automation tasks, and custom NiFi processors.

Apply data quality , validation, deduplication, and error-handling frameworks within ingestion flows.

Work with analysts, data scientists, and platform teams to deliver clean, reliable, analytics-ready datasets.

Enforce GCP IAM , VPC Service Controls , encryption policies, and secure-by-design practices.

Monitor, troubleshoot, and tune NiFi processor performance and GCP data pipelines.

Document data flows, architecture standards, and operational best practices.

Required Skills & Qualifications

Strong hands-on experience with Apache NiFi for real-time ingestion, routing, transformation, and flow orchestration.

Deep expertise across the GCP ecosystem :

Pub / Sub (event ingestion & streaming)

BigQuery (data warehousing & SQL optimization)

Dataflow / Apache Beam (stream & batch processing)

Cloud Storage (data lake)

Dataproc (Spark / Hadoop processing)

Composer / Airflow (workflow orchestration)

Strong proficiency in SQL (analytic functions, optimization, partitioning, clustering).

Strong Python development for automation and data processing.

Solid understanding of real-time streaming concepts : windowing, late data handling, deduplication, event ordering.

Knowledge of data architecture patterns (ETL / ELT, data lakes, data warehouses).

Experience with CI / CD , Git-based workflows, and infrastructure-as-code (nice to have).

Understanding of cloud security , IAM roles, governance controls, and compliance in GCP environments.