Spigot Software - Data Engineer - Python/SQL/ETL

Role and Responsibilities :

- Strong proficiency in Python for data processing and ETL

- Advanced SQL skills, including query optimization, indexing, joins, and analytical functions

- Hands-on experience with ClickHouse, MongoDB, Redis, and ElasticSearch

- Proficiency in Apache Spark/PySpark and working with data lakes

- Experience with ETL and data ingestion tools such as Apache NiFi

- Familiarity with messaging and streaming platforms like Kafka, RabbitMQ, and ActiveMQ

- Experience with workflow orchestration frameworks such as Apache Airflow

- Exposure to cloud platforms (AWS, GCP, or Azure) and their data services (S3, Redshift, BigQuery, Dataproc, etc.)

- Understanding of data warehousing, data modeling, and performance optimization :

- SQL : Advanced ANSI SQL, stored procedures, functions.

- Python : For scripting, automation, and using Snowpark for in-database processing.

- ETL/ELT : Building data pipelines using tools like Snowpipe.

- Data Warehousing : Concepts, data modeling.

- Cloud Platforms : Experience with Azure and AWS (storage, networking).

- Performance Tuning : Caching, clustering keys, virtual warehouses.

- Data Governance : Role-Based Access Control (RBAC), data sharing, and encryption.

- Virtual Warehouses : Sizing, scaling, and multi-cluster warehouses.

- Streams and Tasks : Change data capture and scheduling.

- Time Travel and Fail-Safe : Data recovery and historical access.

- Data Sharing : Secure data exchange.

(ref:hirist.tech)