Freelance GCP Data Engineer - Apache NiFi & SQL Expert

Company Description

ThreatXIntel is a startup specializing in cybersecurity, offering tailored and cost-effective solutions for businesses and organizations of all sizes. Our expertise spans cloud security, web and mobile security testing, cloud security assessments, DevSecOps, and more. Driven by a proactive approach, we monitor and test our clients' digital environments to identify vulnerabilities before they can be exploited. Our mission is to deliver reliable and accessible cybersecurity services, empowering businesses to focus on growth while safeguarding their digital assets. Based on a commitment to quality, our team strives to provide peace of mind through customized protection strategies.

Role Description

We are seeking a skilled Freelance GCP Data Engineer with expertise in Apache NiFi, SQL, and API integrations to design, implement, and optimize real-time data pipelines. This role involves working with the Google Cloud Platform (GCP) ecosystem, including BigQuery, Pub / Sub, Dataflow, and Cloud Storage, to ensure high-performance data processing, transformation, and delivery to downstream systems for analytics and reporting.

Key Responsibilities

• Design and implement real-time data ingestion pipelines using Apache NiFi for structured and unstructured data.
• Integrate data pipelines with the GCP ecosystem, including BigQuery, Pub / Sub, Dataflow, Dataproc, Cloud Storage, and Composer / Airflow.
• Ensure low-latency, high-throughput data delivery to downstream systems for analytics and reporting.
• Develop and optimize SQL queries for BigQuery and relational databases to ensure data transformation and analysis.
• Write Python scripts for data transformation, automation, and custom NiFi processors when required.
• Implement data quality, validation, and error-handling mechanisms within ingestion pipelines to ensure reliability and accuracy.
• Collaborate with data analysts, data scientists, and platform engineers to deliver consistent and high-quality datasets.
• Ensure data security, governance, and compliance with GCP IAM, VPC Service Controls, and encryption standards.
• Monitor, troubleshoot, and optimize data pipeline performance to ensure smooth operations.
• Document data flows, architecture, and best practices to maintain consistency across teams.

Required Skills & Qualifications