Freelance GCP Data Engineer - Apache NiFi & SQL Expert

Company Description

ThreatXIntel is a startup specializing in cybersecurity, offering tailored and cost-effective solutions for businesses and organizations of all sizes. Our expertise spans cloud security, web and mobile security testing, cloud security assessments, DevSecOps, and more. Driven by a proactive approach, we monitor and test our clients' digital environments to identify vulnerabilities before they can be exploited. Our mission is to deliver reliable and accessible cybersecurity services, empowering businesses to focus on growth while safeguarding their digital assets. Based on a commitment to quality, our team strives to provide peace of mind through customized protection strategies.

Role Description

We are seeking a skilled Freelance GCP Data Engineer with expertise in Apache NiFi , SQL , and API integrations to design, implement, and optimize real-time data pipelines . This role involves working with the Google Cloud Platform (GCP) ecosystem, including BigQuery , Pub / Sub , Dataflow , and Cloud Storage , to ensure high-performance data processing, transformation, and delivery to downstream systems for analytics and reporting.

Key Responsibilities

Design and implement real-time data ingestion pipelines using Apache NiFi for structured and unstructured data.

Integrate data pipelines with the GCP ecosystem , including BigQuery , Pub / Sub , Dataflow , Dataproc , Cloud Storage , and Composer / Airflow .

Ensure low-latency, high-throughput data delivery to downstream systems for analytics and reporting .

Develop and optimize SQL queries for BigQuery and relational databases to ensure data transformation and analysis.

Write Python scripts for data transformation, automation, and custom NiFi processors when required.

Implement data quality , validation, and error-handling mechanisms within ingestion pipelines to ensure reliability and accuracy.

Collaborate with data analysts , data scientists , and platform engineers to deliver consistent and high-quality datasets.

Ensure data security , governance , and compliance with GCP IAM , VPC Service Controls , and encryption standards .

Monitor, troubleshoot , and optimize data pipeline performance to ensure smooth operations.

Document data flows , architecture, and best practices to maintain consistency across teams.

Required Skills & Qualifications

Proven experience with Apache NiFi for real-time ingestion, routing, and transformation of data.

Strong expertise in the GCP ecosystem , including :

BigQuery (data warehousing)

Pub / Sub (event streaming)

Dataflow / Apache Beam (data processing)

Cloud Storage (data lake)

Composer / Airflow (workflow orchestration)

Strong SQL skills for querying, modeling, and performance optimization of data.

Proficiency in Python for data transformation, automation, and creating custom NiFi processors.

Experience with real-time streaming concepts , such as windowing, late-arriving data, and deduplication.

Strong understanding of ETL / ELT processes , data lakes , and data warehouses .

Familiarity with CI / CD practices and version control using Git .

Understanding of data security , governance , and compliance within cloud environments.