Vsquare Systems - Data Engineer - Databricks

Job Title : Data Engineering - PySpark, ETL & Data Modeling

Experience : 6-8 Years

Location : Bangalore (Preferred) / Remote

Job Summary :

We are seeking an experienced Data Engineering with expertise in PySpark, ETL Development, Data Modeling, Databricks, and Snowflake to lead the design, development, and optimization of enterprise-scale data platforms. The ideal candidate will have a strong background in building scalable data pipelines, designing robust data models, and delivering high-performance data engineering solutions on modern cloud platforms.

This is a hands-on technical leadership role that requires expertise in Data Engineering, Data Modeling, Big Data technologies, and Cloud Data Platforms, along with the ability to mentor teams and drive end-to-end project delivery.

Key Responsibilities :

- Lead end-to-end data engineering initiatives, including requirements gathering, solution design, development, testing, deployment, and production support.

- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark, Apache Spark, and Spark SQL.

- Build high-performance data ingestion, transformation, integration, and loading pipelines for enterprise data platforms.

- Design and implement Conceptual, Logical, and Physical Data Models to support transactional and analytical workloads.

- Develop and optimize database schemas, tables, views, indexes, and other data structures.

- Build reusable ETL frameworks and standardized data engineering components.

- Extract data from multiple sources, including databases, APIs, cloud storage, files, data lakes, and streaming platforms.

- Process and transform large-scale structured and unstructured datasets using distributed computing principles.

- Develop batch and real-time/streaming data pipelines to support enterprise analytics.

- Optimize Spark jobs, SQL queries, and database performance to improve scalability and resource utilization.

- Implement data quality frameworks, validation rules, monitoring, and exception handling within ETL workflows.

- Maintain data lineage, metadata, documentation, and technical specifications to support enterprise data governance.

- Implement logging, monitoring, and troubleshooting mechanisms for ETL jobs and production pipelines.

- Collaborate with business stakeholders, architects, and cross-functional teams to deliver scalable data solutions.

- Provide technical leadership, mentor development teams, and establish data engineering best practices.

Required Skills :

- 6-8 years of experience in Data Engineering, ETL/ELT Development, and Data Modeling.

- Strong expertise in :

i. PySpark

ii. Apache Spark

iii. Spark SQL

iv. Python

v. Databricks

vi. Snowflake

- Strong understanding of ETL architecture, distributed computing, and large-scale data processing.

- Experience designing :

i. Conceptual Data Models

ii. Logical Data Models

iii. Physical Data Models

iv. ER Models

v. Dimensional Models

vi. Data Vault Models

- Hands-on experience with Data Lakes, Data Warehouses, and Cloud Data Platforms.

- Strong SQL skills with expertise in query optimization and performance tuning.

- Experience with relational databases such as Oracle, SQL Server, PostgreSQL, or Snowflake.

- Experience with orchestration and data integration tools such as Apache Airflow, Kafka, Apache NiFi, or similar.

- Knowledge of data governance, metadata management, data lineage, and data quality frameworks.

- Experience developing both batch and streaming data pipelines.

- Strong analytical, debugging, and problem-solving Skills :

- Experience with cloud platforms such as AWS or Microsoft Azure.

- Familiarity with Hadoop ecosystem and modern Big Data technologies.

- Experience with CI/CD pipelines, Git, and Agile development methodologies.

- Knowledge of API integrations and enterprise data ingestion frameworks.

- Experience working on enterprise-scale data modernization and cloud migration Qualification :

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

(ref:hirist.tech)