Automation Engineer

Job Summary :

• . The candidate should be a data enthusiast who has a zeal to work with data, analyze, understand and improve data processing techniques. The primary focus on this testing is to verify the accuracy of this data as various business rules are applied mainly within Azure Databricks and partly inside Azure Data Factory. This individual will be expected to understand the data flow, validate the content, understand and test the business rules involved in merging consumers and build an automation framework that supports a number of data quality checks.

Essential Job Functions :

Ø Create and execute scenarios to test various API. Preparing request block and analyzing the responses in JSON / XML formats.

Ø Validate the flow of data from disparate sources ingested into multiple databases inside Databricks, post which data is transformed by pipelines and workflows built within Azure Databricks and Azure Data Factory (ETL process).

Ø Thoroughly test the ETL rules built for data transformation & complex business rule built to merge consumers.

Ø Strong in SQL skills. Should possess ability to write complex queries.

Ø Test different source and target tables available in Azure Databricks that are sourced, cleansed, transformed, joined, aggregated and final data sent to downstream applications.

Ø Execute tests using SQL or Python or PySpark as per the user stories to validate the data inside various databases within the Databricks environment.

Ø Automate recurring QA processes through the use of advanced languages such as Python or PySpark or Java as needed.

Ø Design and build out an automation framework using PyTest to validate different scenarios and its data. This includes both automating new tests and / or updating existing scripts.

Ø Previously should have exposure to code repository tools, creating branches, pull requests and perform code merge activities.

Ø Previously should have exposure to SonarQube and main code quality, fix code smells etc.

Ø Experience building basic workflows or data pipelines inside Azure Data Factory would be a plus. This would be a great add as certain repeated testing of data flows will need to be automated.

Ø Create and execute detailed manual test cases from time to time using functional requirements and technical specifications within Jira to ensure quality and accuracy.

Ø Log appropriate defects within Jira when product does not conform to specifications.

Ø Participate in daily stand-ups with project team as part of the agile methodology.

Ø Coordinate with development team members regarding defect validation and assist development team members with re-creating defects.

Ø Create appropriate test cases within TestRail Test Management tool.

Ø Update tasks information in Jira as appropriate to communicate progress with onshore test lead.

Other Responsibilities :

Ø Maintain technology expertise, keeping current with evolving testing tools, techniques, and strategies to improve the overall testing efficiency, processes, and best practices.

Ø Maintain a focus on customer-service, efficiency, quality, and growth.

Ø Safeguard the company's assets.

Ø Adhere to the company's compliance program.

Ø Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.

Minimum Qualifications and Job Requirements :

Ø 3+ years working with Azure Databricks

Ø 1+ years working with Azure Data Factory

Ø 3+ years strong experience in writing complex SQL queries.

Ø 3+ years of experience in Python programming to deal with data or PySpark experience is a must.

Ø 3+ years of experience in building test automation for data processing within data intensive projects.

Ø Experience with code repository tools, creating branch, pull requests and perform code merges.

Ø Experience building basic workflows or data pipelines inside Azure Data Factory would be a plus.

Ø Good understanding of file formats including JSON, Parquet, Avro, and others

Ø Ability to learn new technologies quickly

Ø Excellent problem-solving skills

Ø Basic understanding of software development principles.

Ø Knowledge of Jira

Ø Ability to handle multiple tasks / projects concurrently and meet deadlines.

Ø Ability to work in a fast-paced team environment. Expectations include a high level of initiative and a strong commitment to job knowledge, productivity, and attention to detail

Ø Solid software engineering skills - participated in full lifecycle development on large projects.