Data Platform Architect

About Creditsafe : With the largest owned database supported by a partner network, Creditsafe provides data on 320 million companies in over 100 countries to support its customers in this regard. Creditsafe is scaling business and currently going through a significant growth stage. Being a data company, Creditsafe collects and processes large amounts of complex data, received from multiple channels with the only purpose of serving our clients with opportunities to gain actionable insights. Our holistic data aggregation helps our clients to fill the knowledge gaps, influence decision making, and find hidden patterns.

About Opportunity : This opportunity is to join a team of highly technically skilled engineers who are redesigning Creditsafe’s data platform with high throughput and scalability as the primary goal. The data delivery platform is being built upon AWS Redshift and S3 cloud storage. The platform is expected to manage over a billion objects along with daily increments of more than 10 million objects while handling addition, deletion, and correction of our data and indexes in an auditable manner. Our data processing application is entirely based on Python and designed to efficiently transform incoming raw data volume into API consumable schema. The team is also building highly available and low latency APIs to enable our clients with faster data delivery.

What you will be doing?

• You will actively contribute to the codebase and participate in peer reviews.
• Design and build metadata driven, event based distributed data processing platform using technologies such as Python, DBT, Airflow, Redshift, DynamoDB, AWS Glue, S3.
• As an experienced Engineer, you will play a critical role in the design, development, and deployment of our business-critical system.
• You will be building and scaling Creditsafe’s data pipeline primarily moving data between S3, Redshift and Aurora.
• Execute practices such as continuous integration and deployment using test-driven development where applicable to enable the rapid delivery of working pipelines.
• Understanding company & domain data to make recommendations to improve the existing product.

What are we looking for?

Why Creditsafe?

Our Technology Stack : Python, Linux, Airflow, AWS DynamoDB, S3, Glue, Athena, Redshift, lambda, API Gateway, Terraform, CI / CD