Responsibilities :
Data Pipeline Development and Maintenance :
- Build and optimize scalable ETL / ELT pipelines to ingest data from diverse sources such as APIs, cloud platforms, and databases.
- Ensure pipelines are robust, efficient, and capable of handling large volumes of data.
Data Integration and Harmonization :
Implement data transformation and enrichment processes to support analytics and reporting needs.Data Quality and Monitoring :
Troubleshoot and resolve issues related to data quality, latency, or performance.Collaboration with Stakeholders :
Provide technical support and guidance on data-related issues or projects.Tooling and Automation :
Leverage cloud-based solutions and frameworks (e.g., AWS) to streamline processes and enhance automation.Maintain and optimize existing workflows while continuously identifying opportunities for improvement.Documentation and Best Practices :
Document pipeline architecture, data workflows, and processes for both technical and non-technical audiences.Follow industry best practices for version control, security, and data governance.Continuous Learning and Innovation :
Stay current with industry trends, tools, and technologies in data engineering and marketing analytics.Recommend and implement innovative solutions to improve the scalability and efficiency of data systems.What you need to succeed (minimum qualifications) :
Bachelor of Science degree in Computer Science or equivalentAt least 2+ years of experience as a data engineer developing and maintaining data pipelinesStrong experience with databases and data platforms (AWS preferred)Proficiency in Python, SQL, PySparkExperience in Data Quality, Data Modeling, Data Analytics / BI, Data Enrichment.Understanding of concepts such as normalization, SCD (Slowly changing dimensions) and CDC (Change data capture)Experience in working on streaming event platforms such as Kafka / KinesisKnowledge of non-relational databasesPreferable experience in DBT for data transformation and modeling.Good understanding of data warehouses, ETL / ELT, AWS architecture (using Glue, SQS, SNS, S3, step functions etc.,)Understanding of orchestration tools such as AirflowAbility to create clean, well-designed code and systemsStrong attention to detail and a commitment to data accuracy.Proven ability to learn new data models quickly and apply them effectively in a fast-paced environment.Ability to work collaboratively in a team environment.What will give you a competitive edge (preferred qualifications) :
Experience working with AWS to develop data pipelinesAWS certifications : Solution Architect or Developer AssociateExperience migrating data pipelines and systems to modern cloud-based solutionsAirline experience is a plusID : DELRM02