StreamSets Developer - Data Pipeline

Role : StreamSets Developer

Job Description :

• Designing, developing, and deploying data pipelines using StreamSets.
• Configuring data sources, transformations, and destinations within pipelines.
• Monitoring and troubleshooting pipeline performance and errors.
• Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.
• Implementing best practices for data governance, security, and compliance.
• Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and consumer : Development Monitoring and Maintenance :
• Designing and developing data pipelines using StreamSets based on business requirements.
• Implementing data transformations, validations, and enrichments within pipelines.
• Monitoring pipeline performance metrics such as throughput, latency, and error and Compliance :
• Ensuring that data handling practices within StreamSets comply with organizational security policies and regulatory requirements.
• Implementing encryption, access controls, and auditing mechanisms to protect sensitive and Knowledge Sharing :
• Documenting pipeline configurations, dependencies, and operational procedures.
• Sharing knowledge and best practices with team members through training sessions, documentation, and collaboration Optimization :
• Analyzing pipeline performance bottlenecks and optimizing configurations for improved efficiency.
• Scaling pipelines to handle increasing data volumes and processing and Communication :
• Collaborating with cross-functional teams including developers, data scientists, and business analysts to understand data requirements and ensure alignment with business objectives.
• Communicating updates, issues, and resolutions effectively with stakeholders.

ref : hirist.tech)