Freelance Azure Data Engineer - Vectr & Cribl Expert

Company Description

ThreatXIntel is a startup cyber security company focused on empowering businesses and organizations to defend against evolving cyber threats. Offering a comprehensive suite of services, including cloud security, web and mobile security testing, DevSecOps, and cloud security assessments, we tailor affordable solutions to suit the needs of startups and small businesses. With a proactive approach to security, we continuously monitor and assess our clients' digital environments to prevent vulnerabilities. At ThreatXIntel, our mission is to deliver exceptional cyber security services so our clients can safeguard their digital assets while concentrating on their growth.

Role Description

We are seeking an experienced Freelance Azure Data Engineer with strong expertise in Vectr and Cribl to design and develop ETL / ELT pipelines , optimize data workflows, and ensure robust data observability and streaming . This role is perfect for an engineer with extensive experience in Azure cloud services and tools for high-performance data processing.

Key Responsibilities

Design, develop, and maintain ETL / ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark) for efficient data transformation and integration.

Build and manage API integrations for seamless data exchange between systems.

Write and optimize SQL queries and stored procedures for data transformation, analysis, and performance tuning.

Implement and manage Azure cloud services , including Azure Data Lake , Azure SQL Database , ensuring scalability and high performance.

Develop Python-based data processing solutions for automation, orchestration, and transformation tasks.

Integrate and configure Vectr for log management and real-time data streaming , optimizing data pipelines and ensuring data flow integrity.

Leverage Cribl for data observability , transforming and routing data efficiently while filtering out unnecessary data.

Ensure data security , governance , and compliance with industry standards and best practices.

Monitor, troubleshoot , and optimize the performance of data workflows, ensuring reliability and scalability.

Write unit tests and collaborate with the testing team to ensure automated testing is fully integrated.

Collaborate with cross-functional teams to resolve production issues and provide timely data-related solutions.

Required Skills & Qualifications

8+ years of experience as a Data Engineer , with a focus on Azure cloud services and data pipeline design .

Mandatory expertise in Vectr for log management and real-time data streaming .

Strong experience with Cribl for data routing , observability , and transformation .

Extensive knowledge in SQL Server , with strong SQL query writing, optimization, and data transformation capabilities.

Hands-on experience with Azure Data Factory (ADF) , Databricks (PySpark) , Azure Data Lake , and Azure SQL Database .

Proficiency in Python for data processing automation and orchestration.

Experience in unit testing and integrating tests into CI / CD pipelines .

Solid understanding of data security , governance , and compliance practices in cloud environments.

Excellent problem-solving , multi-tasking , and collaboration skills to work across teams and handle complex data tasks.