Sr. Data Engineer

Exigo Tech is a Sydney-based Technology Solutions Provider that is focused on providing solutions on three major verticals; Infrastructure, Cloud, and Application to businesses across Australia. We help companies reach operational efficiencies by empowering them with technology solutions that drive their business processes.

Exigo is looking for Full-time Sr. Data Engineer

We are ISO 27001 : 2022 certified organization

Visit our website : for more details….

Click Here to know more : LIFE AT EXIGO TECH

Roles and Responsibilities

• Install, configure, and manage Apache Spark (open-source) clusters on Ubuntu, including Spark master / worker nodes and Spark environment files.
• Configure and manage Spark UI and Spark History Server for monitoring jobs, analyzing DAGs, stages, tasks, and troubleshooting performance.
• Develop, optimize, and deploy PySpark ETL / ELT pipelines using DataFrame API, UDFs, window functions, caching, partitioning, and broadcasting.
• Deploy PySpark jobs using spark-submit in client / cluster mode with proper logging and error handling.
• Install, configure, and manage Apache Airflow including UI, scheduler, webserver, connections, and variables.
• Create, schedule, and monitor Airflow DAGs for PySpark jobs using SparkSubmitOperator, BashOperator, or PythonOperator.
• Configure and manage cron jobs for scheduling data processing tasks where needed.
• Install, configure, and optimize Trino (PrestoSQL) coordinator and worker nodes; configure catalogs such as S3, MySQL, or PostgreSQL.
• Maintain Linux / Ubuntu servers including services, logs, environment variables, memory usage, and port conflict resolution.
• Design and implement scalable data architectures using Azure Data Services including ADF, Synapse, ADLS, Azure SQL, and Databricks.
• Develop, manage, and automate ETL / ELT pipelines using Azure Data Factory (Pipelines, Mapping Dataflows, Dataflows).
• Monitor, troubleshoot, and optimize data pipelines across Spark, Airflow, Trino, and Azure platforms.
• Work with structured, semi-structured, and unstructured data across multiple data sources and formats.
• Implement data analytics, transformation, backup, and recovery solutions.
• Perform data migration, upgrade, and modernization using Azure and database tools.
• Implement CI / CD pipelines for data solutions using Azure DevOps and Git.
• Ensure data quality, governance, lineage, metadata management, and security compliance across cloud and big data environments.
• Design and optimize data models using star and snowflake schemas; build data warehouses, Delta Lake, and Lakehouse systems.
• Develop and rebuild reports / dashboards using Power BI, Tableau, or similar tools.
• Collaborate with internal teams, clients, and business users to gather requirements and deliver high-quality data solutions.
• Provide documentation, runbooks, and operational guidance.

Technical Skills :

2. Apache Airflow & Job Orchestration - Must

3. Trino (PrestoSQL) - Must

4. Azure Data Services (nice to have)

5. Microsoft Fabric ( nice to have)

6. Programming & Querying

7. Data Modeling & Warehousing

8. DevOps & CI / CD

9. BI Tools (Nice to have)

10. Linux / Ubuntu Server Knowledge

Soft Skills :

Education :

Work Location : Vadodara, Gujarat, India