| Job Purpose | Responsible for analyzing transforming and deriving insights from large-scale data using Databricks and modern data stack technologies. Designs and executes data pipelines using SQL and PySpark on Databricks leveraging Unity Catalog for data governance and management. Collaborates with cross-functional teams to build data models perform exploratory analysis and deliver actionable insights. Optimizes queries on SQL Warehouses integrates external APIs where needed and ensures data quality and reliability across workflows |
| Duties and Responsibilities | Develop maintain and optimize scalable data pipelines using Databricks (PySpark and SQL). Perform data extraction transformation and loading (ETL/ELT) for large and complex datasets. Analyse data to generate actionable insights supporting business decision-making. Design and manage data models and curated datasets within Unity Catalog. Ensure data quality consistency and reliability through validation and monitoring frameworks. Optimize query performance on SQL Warehouses for faster analytics and reporting. Collaborate with business stakeholders to understand requirements and translate them into data solutions. |
| Key Decisions / Dimensions | Selection of appropriate data models pipeline design and transformation logic to meet business requirements efficiently. Optimization strategies for query performance compute usage and cost on Databricks SQL Warehouses and Spark workloads. Choice of data sources integration approaches and data quality rules to ensure reliable and governed datasets |
| Major Challenges | Handling large-scale distributed datasets in Databricks while maintaining performance and cost efficiency of pipelines and queries. Ensuring data quality and consistency across multiple sources evolving schemas and governed environments like Unity Catalog. Translating ambiguous business requirements into scalable data models and actionable insights within tight timelines |
| Required Qualifications and Experience | Educational Qualifications: Graduate or PostGraduate in Computer Science Information Technology or Data Science/Technologies. Work Experience: 0.62 years of handson data engineering/analyst experience. Technical Expertise / Skills Keywords: Data Platforms & Tools: Databricks Delta Lake Unity Catalog SQL Warehouses Programming & Querying: Python PySpark SQL Spark SQL Data Engineering & Processing: ETL/ELT Pipelines Data Modeling Data Warehousing Data Transformation Big Data Processing Apache Spark Distributed Computing Cloud & Integration: Azure Data Services Azure Data Lake Storage (ADLS) Azure Key Vault REST API Integration DevOps & Workflow: Git Azure DevOps CI/CD Git Workflow Orchestration Analytics & Visualization: Exploratory Data Analysis (EDA) Dashboarding & Reporting (Power BI/Tableau) |
Required Experience:
Senior IC
Senior Data Analyst • Pune, Maharashtra, India
At Parable Associates, we specialize in transforming complex data into clear, actionable insights for our clients across various industries.We’re not just another business intelligence company; we’... Show more
Partner closely with Valuation, Assumptions, Accounting Policy, Pricing and Risk Management teams in the development of sustainable annuity projection models, compliant with emerging accounting and... Show more
