Job Title :
Data Quality Automation Specialist
Join a dynamic team of top developers in leading data quality automation efforts. With 4+ years of experience, you will work on designing and developing automated test frameworks for data pipelines and ETL workflows.
You will implement data validation and quality checks using Great Expectations and custom Python scripts, collaborating with Data Engineers to ensure data consistency, accuracy, and performance across Databricks and Snowflake environments.
- Main Responsibilities :
- Design, develop, and maintain automated test frameworks for data pipelines and ETL workflows.
- Implement data validation and quality checks using Great Expectations and custom Python scripts.
- Collaborate with Data Engineers to ensure data consistency, accuracy, and performance across Databricks and Snowflake environments.
- Develop and execute PySpark-based test cases to validate data transformations and processing logic.
- Integrate automated tests into CI / CD pipelines (e.g., GitHub Actions, Jenkins, Azure DevOps, or GitLab CI).
- Monitor, troubleshoot, and improve data validation processes to ensure end-to-end data quality.
- Work closely with cross-functional teams to define data testing strategies, validation metrics, and reporting dashboards.
Required Skills & Qualifications :
Proficient in Python (for scripting, data processing, and automation).Hands-on experience with PySpark for large-scale data validation and transformation testing.Strong understanding of Databricks workflows and Snowflake data warehousing.Practical experience with Great Expectations for data quality checks.Familiarity with CI / CD tools (GitHub Actions, Jenkins, GitLab, or Azure DevOps).Good understanding of ETL processes, data pipelines, and data modeling.Experience in Automation Testing and test-driven data development practices.Excellent debugging, documentation, and communication skills.Preferred Skills :
Experience with Airflow or other orchestration tools.Exposure to AWS, Azure, or GCP data ecosystems.Knowledge of SQL optimization and data performance tuning.Familiarity with containerization tools (Docker, Kubernetes) is a plus.Benefits :
35 hours of work per week from home.Paid monthly.Minimum 4 years of experience required.Currently working full-time remotely required.