Purpose of the Role :
LUMI is companys largest Big Data Platform, ideally suited for computationally and / or data intensive processing applications. Whether the data needs to be processed in batch, online, or streaming manner, Lumi provides robust capabilities to handle such workloads effectively, in a cost-efficient manner.
A hub of very hardworking Big Data engineers and most exciting & upcoming technologies. Cornerstone platform offers an environment where Engineers are challenged every day to build world class products.
As we embark on the journey to move to public cloud - GCP you will be part of a fast-paced Agile team, design, develop, test, troubleshoot & optimize solutions created to simplify access to the Amexs Big Data Platform.
Focus :
- Designs, develops, solves problems, debugs, evaluates, modifies, deploys, and documents software and systems that meet the needs of customer-facing applications, business applications, and / or internal end user Context :
- Member of an engineering or delivery and integration team reporting to an engineer manager or Engineering Director
Responsibilities :
Implemented enterprise grade robust data migration solutions using Java and Python, facilitating seamless data transfer from on-premises environments to GCP (including Cloud Storage and Big Query), leveraging Apache Airflow and Google Cloud Composer.Build secured, and optimized data architectures on GCP by integrating services such as Cloud Storage, Pub / Sub, Dataproc.Implement automated solutions for data delivery, monitoring, and troubleshooting.Monitor system performance and proactively optimize data pipelines for efficiencyTroubleshoot and resolve issue.Create and maintain comprehensive documentation for tools, architecture, processes, and solutions.Data Pipeline Development :
Build, test, and deploy data pipelines to move, transform, and process data from various sources to GCPEnsure the reliability, scalability, and performance of data pipelines.Utilize GCP's big data technologies such as Big Query, Dataflow, Dataprep, and Pub / Sub to implement effective data processing solutionsMinimum Qualifications :
Overall, 3 - 5 years of developer experience.Proficiency in Java, Python, and shell scripting.Strong SQL knowledgeProficiency in Google Cloud Platform services especially Cloud Storage, Big query, Dataproc and Pub / Sub.Proficiency in RDBMS like Oracle, Postgres or MySQL and good exposure in at least one NoSQL DB like Cassandra.Expertise in Git & CICD processes.Experience of working in agile application development environmentTechnical support to applications on trouble shooting Environment, software and application-level issuesWrite, test programs using Unix Shell scripting.Preferred Qualifications :
Hands on or exposure to DevOps best practices and implementation.Hands-on or exposure to platform engineering including networking and firewall.Hands-on or exposure to GenAI integrations including LLMs and RAG.(ref : hirist.tech)
private%20limited){kind=logo}