Kafka Consulting & Architecture : Consult with inquiring teams on how to effectively leverage Kafka within their data pipelines, providing expert guidance and solutions.
Cluster Management (IaC) : Architect, build, and provide ongoing support for existing and new Kafka clusters, strictly adhering to Infrastructure-as-Code (IaC) best practices.
Traffic Routing & Integration : Partner with Splunk teams to strategically route traffic through Kafka, utilizing open-source agents and collectors deployed via Chef.
Health & Performance Remediation : Proactively identify and remediate any health issues within Kafka clusters to ensure optimal performance and stability.
Operational Automation : Identify opportunities to automate operational processes within the team, developing and implementing solutions where possible.
Monitoring & Alerting : Create new and / or update existing monitoring dashboards and alerts as needed to ensure comprehensive visibility into Kafka health.
CI / CD Pipeline Management : Manage and optimize a continuous improvement / continuous development (CI / CD) pipeline for Kafka-related deployments.
Innovation & PoC : Perform Proof-of-Concepts (PoCs) on new components and technologies to expand and enhance the teams Kafka offerings.
Documentation : Maintain thorough documentation of Kafka configurations, procedures, and architectural Skills & Qualifications :
Overall Experience : 7 to 10 years of professional experience in IT infrastructure or software development.
Kafka Administration : Minimum 2+ years of hands-on experience with Kafka clustering & administration.
Infrastructure-as-Code (IaC) : Minimum 2+ years of experience building, deploying, and supporting multiple Kafka clusters using IaC best practices.
Automation : Proven experience developing automated processes within and around Kafka to supplement service operations.
Data Pipeline Architecture : Experience working with multiple teams to build and architect data pipeline solutions where Kafka will be a key component.
Operating Systems : Proficient experience with Linux / Unix and system management.
IaC Tools : Hands-on IaC experience with Virtual / Physical Servers using tools like Chef, Ansible, Jenkins, Artifactory, etc.
CI / CD : Strong understanding of Git workflows and continuous improvement / continuous development (CI / CD) concepts.
Communication : Strong verbal and written communication skills.
Analytical & Technical : Strong technical acumen and analytical skills.
Methodology : Experience working in Agile / Lean Qualifications :
Knowledge and experience with Splunk, Elastic, Kibana, and Grafana for monitoring and visualization.
Knowledge and experience with log collection agents such as Open-Telemetry, Fluent Bit, FluentD, Beats, and LogStash.
Knowledge and experience with Kubernetes / Docker for container orchestration.
Knowledge and experience with Kafka-Connect.
Knowledge and experience with cloud platforms like AWS or Azure.
Knowledge and experience with Streaming Analytics.