Senior Data Engineer

This role involves collaborating with cross-functional teams to ensure data reliability, scalability, and performance. The candidate will work closely with data scientists, analysts and software engineers to ensure efficient data flow and storage, enabling data-driven decision-making across the organisation.

Responsibilities

• Software Engineering Excellence : Write clean, efficient, and maintainable code using JavaScript or Python while adhering to best practices and design patterns.
• Design, Build, and Maintain Systems : Develop robust software solutions and implement RESTful APIs that handle high volumes of data in real-time, leveraging message queues (Google Cloud Pub / Sub, Kafka, RabbitMQ) and event-driven architectures.
• Data Pipeline Development : Design, develop and maintain data pipelines (ETL / ELT) to process structured and unstructured data from various sources.
• Data Storage & Warehousing : Build and optimize databases, data lakes and data warehouses (e.g. Snowflake) for high-performance querying.
• Data Integration : Work with APIs, batch and streaming data sources to ingest and transform data.
• Performance Optimization : Optimize queries, indexing and partitioning for efficient data retrieval.
• Collaboration : Work with data analysts, data scientists, software developers and product teams to understand requirements and deliver scalable solutions.
• Monitoring & Debugging : Set up logging, monitoring, and alerting to ensure data pipelines run reliably.
• Ownership & Problem-Solving : Proactively identify issues or bottlenecks and propose innovative solutions to address them.

Qualifications