To bring domain expertise in data engineering to the team, including the ETL process using tools.
1.Data engineering best practices
You will contribute to the data team&aposs ability to adhere to data engineering best practices across pipeline design, data quality monitoring, storage, versioning, security, testing, documentation, cost, and error handling.
2. Data transformation in dbt
Ensure that the daily dbt build is successful, including full test coverage of existing models.
Create new data models in collaboration with the data analysts.
Add new tests to enhance data quality.
Incorporate new data sources
3.Data extraction
Develop and maintain our data pipelines in Stitch, Fivetran, Segment and Apache Airflow. Evaluate when it&aposs appropriate to using tools, versus building data pipelines in Apache Airflow.
Ensure that data extraction jobs run successfully daily. Collaborate with engineers from MarTrust to add new data sets to our data extraction jobs.
4. Data warehousing in BigQuery
Ensure that the data in our data warehouse is kept secure and that daily jobs in BigQuery run successfully.
5. Maritime domain expertise
You will be expected to quickly develop a deep understanding of MarTrust&aposs data domain in order to support the organisation with accurate data models and tests.
Requirements
Benefits
Show more
Show less
Skills Required
Apache Airflow, BigQuery, dbt, Fivetran, Etl
Senior Data Engineer • Mumbai, India