Data Engineer - Python / Spark

VariteBangalore

30+ days ago

Job description

About The Job :

Develops technical tools and programming to cleanse, organize and transform data and to maintain, protect and update data structures and integrity on an automated basis.
Applies data extraction, transformation, and loading techniques in order to tie together large data sets from a variety of sources.
Partners with both internal & external sources to design, build and oversee the deployment and operation of technology architecture, solutions and software.
Designs, develops and programs methods, processes and systems to capture, manage, store and utilize structured and unstructured data to generate actionable insights and solutions.
Responsible for the maintenance, improvement, cleaning, and manipulation of data in the business client's operational and analytics databases.
Proactively analyzes and evaluates the business client's databases in order to identify and recommend improvements and optimization.

Essential Job Functions :

Uses knowledge of existing and emerging data science engineering principles, theories, and techniques to inform business decisions; and produce accurate business insights.

Completes projects and assignments of moderate scope and complexity under normal supervision to ensure customer and business needs are met.

Applies discretion and independent judgement to interpret data trends and summarize data insights.

Assists in the preliminary data exploration, data preparation for accurate model development.

Establishes working relationships with others outside area of Data Science Engineering expertise.

Prepares presentations of project outputs for external customers with assistance.

Design, develop, and maintain scalable data pipelines and systems for data processing.

Utilize Data Lakehouse, Spark on Kubernetes and related technologies to manage large-scale data processing.

Perform data ingestion from various sources like API's, RDBMS, NoSQL DB's, Kafka, Middleware & Files using Spark and process data into Lakehouse platform.

Develop and maintain py-spark scripts for automation of data processing tasks.

Implement full and incremental data loading strategies to ensure data consistency and availability.

Orchestrate and monitor workflows using Apache Airflow.

Ensure code quality and version control using GIT.

Troubleshoot and resolve data-related issues in a timely manner.

Stay up-to-date with the latest industry trends and technologies to continuously improve our data infrastructure.

Design, develop, and maintain scalable data pipelines and systems for data processing.

Utilize Data Lakehouse, Spark on Kubernetes and related technologies to manage large-scale data processing.

Perform data ingestion from various sources like API's, RDBMS, NoSQL DB's, Kafka, Middleware &

Files using Spark and process data into Lakehouse platform.

Develop and maintain py-spark scripts for automation of data processing tasks.

Implement full and incremental data loading strategies to ensure data consistency and availability.

Orchestrate and monitor workflows using Apache Airflow.

Ensure code quality and version control using GIT.

Troubleshoot and resolve data-related issues in a timely manner.

Stay up-to-date with the latest industry trends and technologies to continuously improve our data :

Proven experience as a Data Engineer (ETL, data warehousing, data Lakehouse).

Strong knowledge of Spark on Kubernetes, S3 and Docker Images.

Proficiency in Data engineering techniques with Py-spark.

Strong experience in Data warehousing techniques like data mining, data analysis, data profiling.

Experience with Python scripting for automation.

Expertise in full and incremental data loading techniques.

Excellent problem-solving skills and attention to detail.

Ability to work collaboratively in a team environment and communicate effectively with stakeholders.

Good to have :

Understanding of streaming data applications using.

Hands-on experience with Apache Airflow for workflow orchestration.

Proficiency with GIT for version control

Understanding of data engineering integration with LLMs or GEN-AI applications and Vector DB.

Knowledge on Shell scripting Postgres SQL or SQL server or MSBI.

(ref : hirist.tech)

Create a job alert for this search

Data Engineer • Bangalore

Related jobs

Promoted

Data Engineer – Databricks & PySpark

Capgemini EngineeringBengaluru, Karnataka, India

Data Engineer – Databricks & PySpark.Choosing Capgemini means choosing a place where you’ll be empowered to shape your career, supported by a collaborative global community, and inspired to reimagi...Show moreLast updated: 16 days ago

Promoted

AWS Data Engineer

TerraGiGBengaluru, IN

Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR. Writing reusable, testable, and efficient code.Integration of data storage ...Show moreLast updated: 5 days ago

Promoted

Snowflake Data Engineer (Snowflake+DBT+SQL+Python) Only Immediate Joiners- PAN India

Mount Talent Consulting Pvt Ltd.hosur, tamil nadu, in

As a Snowflake Lead with experience, and additional expertise in ETL tools and DBT, youwill be entrusted with a pivotal role in designing, implementing, and managing Snowflake solutions to driveeff...Show moreLast updated: 24 days ago

Promoted

Data Engineer(Azure / AWS, Python / Pyspark, SQL)

Sail AnalyticsBengaluru, Karnataka, India

Architect, develop, test and maintain scalable data warehouses and data pipelines.Expertise in SQL, PySpark / Python and Azure(ADB, ADF) or AWS(Glue, Lambda, Redshift). Bachelor's degree or equivalent...Show moreLast updated: 30+ days ago

Promoted

Senior Python / AI Engineer — Core Team @ Zumlo

Zumlohosur, tamil nadu, in

Zumlo is an always-on well-being companion—one place for immediate help, gentle structure, and progress you can see.We unify mind, body, emotions, and relationships through timely support, a caring...Show moreLast updated: 2 days ago

Promoted

Software Engineer - Backend Python / AI

JuiceLabs AIhosur, tamil nadu, in

Where creative engineering meets applied AI.At JuiceLabs, we’re building vertical AI-native tools that unlock fresh insights and creative superpowers for our clients in advertising, ecommerce, and ...Show moreLast updated: 1 day ago

Promoted

Senior Data Platform Engineer

Black Dog Labshosur, tamil nadu, in

Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps. We’re looking for a hands-on Senior Data Platform...Show moreLast updated: 30+ days ago

Promoted

Python AWS Data Engineer

Digitrix Software LLPBangalore, IN

Mandatory skills - python , pyspark, who can write codes , any cloud exp - aws / gcp / azure.Python, AWS Python (core language skill) Backend, Pandas, PySpark (DataFrame API), interacting with AW...Show moreLast updated: 30+ days ago

Promoted

AWS Data Engineer

Tata Consultancy Serviceshosur, tamil nadu, in

Aws data engineer having strong experience of Python.Technical / Behavioral Competency.Proficient in Python, with experience in deploying Python packages and OOP, Experience in ingesting data from di...Show moreLast updated: 3 days ago

Promoted

Sr. GenAI / Python Developer

BCI~ITGreater Bengaluru Area, India

BCI is looking for GenAI / Python Developers to join an ongoing project for our direct client in the USA.You will join an offshore team that is growing and there is a lot of new and exciting work t...Show moreLast updated: 16 days ago

Promoted

Data Engineer

VAANTECHhosur, tamil nadu, in

Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

AAA Globalhosur, tamil nadu, in

Proprietary Trading / Financial Markets.We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and efficien...Show moreLast updated: 16 days ago

Promoted

Data Engineer

Jaipur Rugshosur, tamil nadu, in

Jaipur Rugs is a social enterprise that connects rural craftsmanship with global markets through its luxurious handmade carpets. It is a family-run business that offers an exclusive range of hand-kn...Show moreLast updated: 12 days ago

Promoted

Senior Data Engineer

USEReadyhosur, tamil nadu, in

Job Title : Senior Databricks Engineer.As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakeho...Show moreLast updated: 26 days ago

Promoted

Sr Data Engineer

Mitchell Martin Inc.hosur, tamil nadu, in

Job Title : Senior Data Engineer.We are looking for a Senior Data Engineer to design, build, and optimize data pipelines and systems that power our analytics, reporting, and data-driven decision-mak...Show moreLast updated: 7 days ago

Promoted

Data Engineer

RAOS INFOSOFT JOIN PRIVATE LIMITEDBengaluru, Karnataka, India

Looking to hire a data engineer with the following : .Born programmer, Strong in Python frameworks.Strong in SQL and RDBMS concepts. Should join within 1 week from the time interview is scheduled.Idea...Show moreLast updated: 5 days ago

Promoted

Data Engineer

SynechronBangalore Urban, Karnataka, India

We have immediate opportunity for.Data Engineer (Python, Spark / Scala).Data Engineer (Python, Spark / Scala).Notice Period : Immediate joiner only. At Synechron, we believe in the power of digital to tr...Show moreLast updated: 30+ days ago

Promoted

Software Engineer

Alp Consulting Ltd.Bangalore Rural, Karnataka, India

Years of in Big Data & Data related technology experience.Expert level understanding of distributed computing principles. Expert level knowledge and experience in Apache Spark.Hands on programming w...Show moreLast updated: 7 days ago