Senior Big Data Solutions EngineerInfoCepts • Pune, Republic Of India, IN
Senior Big Data Solutions Engineer
InfoCepts • Pune, Republic Of India, IN
16 days ago
Job description
Key Result Areas and Activities :
Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.
Should be able to design complex Big Data use cases using AWS Services and Databricks, under guidance of Architect
Should be able to assist the architect in reasoning about various architecture choices in terms of cost, performance and durability.
Should be able to suggest optimizations in existing designs.
Project documentation adheres to quality guidelines & schedules.
Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.
Articulate design decisions very clearly with stakeholders.
Perform Proof-of-Concepts and document all observations before proposing a new solution.
Conduct design review sessions with other teams and identify scope of improvement.
Suggests automation to improve existing processes.
Work and Technical Experience :
Must-Have :
In-depth knowledge of the following AWS services : S3, EC2, EMR, Athena, AWS Glue, Lambda
Experience with at least one MPP database : AWS Redshift, Snowflake, SingleStore
Proficiency in Big Data technologies : Apache Spark, Databricks
Must have strong programming skills in Python
Responsible for building data pipelines in AWS And Databricks
Experience with Big Data table formats, such as Delta Lake (open source)
Must have very strong SQL skills
Experience with orchestration tools like Apache Airflow
Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
Strong understanding of data warehousing concepts
Good to have :
Cloud Databases – Snowflake, AWS Aurora
Big Data – Hadoop, Hive
Cloud Databases– AWS Aurora
Associate Level or Professional Level AWS Certification or Databricks Certification
Qualifications :
Overall, 7+ years of IT Exp
5+ years of relevant experience in AWS related project
Bachelor’s degree in computer science, engineering, or related field (master's degree is a plus)
Demonstrated continued learning through one or more technical certifications or related methods.
Qualities :
Hold strong technical knowledge and experience
Should have the capability to deep dive and research in various technical related fields
Self-motivated and focused on delivering outcomes for a fast-growing team and firm
Able to communicate persuasively through speaking, writing, and client presentations
Able to consult, write, and present persuasively
Able to work in a self-organized and cross-functional team
Able to iterate based on new information, peer reviews, and feedback
Prior experience of working in a large media company would be added advantage
Create a job alert for this search
Senior Data Engineer • Pune, Republic Of India, IN
Related jobs
Senior Data Engineer
Intellias • Nagpur, IN
Apache Flink / Apache Spark (Streaming).Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines.
Kafka + Flink / Spark Streaming).Python for data engin...Show more
Last updated: 16 days ago • Promoted
(Senior) Azure Data Engineer - Full remote - contractor in USD
All European Careers • Nagpur, IN
Remote
For an international project in Chennai, we are urgently looking for a Full Remote Senior Azure Data Engineer, who will build data pipeline for enterprise search applications using ADF and Databric...Show more
Last updated: 1 day ago • Promoted
Senior Data Engineer
Adaptive Technology Insights • India
We are looking for an experienced.You will be responsible for designing, building, and optimizing data pipelines and architecture to support analytics, reporting, and data-driven decision-making ac...Show more
Last updated: 1 hour ago • Promoted • New!
Data Engineer | Bangalore | Leading Big 4 Consulting Firm
Acme Services • Nagpur, IN
Design, develop, and maintain scalable and robust data pipelines (ETL / ELT) using PySpark on the Databricks platform.Implement data ingestion strategies to load data from various sources (e.APIs, da...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Data Engineer
American Inference • India
Please apply directly through for this position.We kindly request that candidates refrain from contacting company officials via email or messages regarding this role.
We are an AI and Data Consulti...Show more
Last updated: 13 days ago • Promoted
Senior AWS Data Engineer
CYAN360 • India, India
Position : Senior AWS Data Engineer.Work Timings : 2 : 30 PM to 11 : 30 PM IST.Need someone who can join immediately or in 15 days • • •.
Design, develop, and deploy end-to-end data pipelines on AWS cloud in...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineer - Data Acquisition
InfoBeans • Nagpur, IN
We are seeking a highly skilled.Senior Data Engineer – Data Acquisition (ODS).The ideal candidate will have extensive hands-on experience in building and optimizing data ingestion and transformatio...Show more
Last updated: 8 days ago • Promoted
Senior Oracle Data Engineer
Strattek Consulting • Nagpur, IN
This role centers on architecting robust data pipelines across on-prem and cloud Oracle ecosystems, automating ingestion from varied sources including APIs.
Exposure to Oracle FDI (Fusion Data Intel...Show more
Last updated: 1 day ago • Promoted
Data Engineer
IntraEdge • Nagpur, IN
We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team.
You will be responsible for building scalable and reli...Show more
Last updated: 30+ days ago • Promoted
Senior Data Engineer
Mindlance • India
We’re looking for a strong, hands-on Sr Data Engineer who can independently drive client conversations, collaborate with SMEs, and deliver high-quality data solutions in a cloud-native environment....Show more
Last updated: 1 hour ago • Promoted • New!
Senior DataOps Engineer (AWS)
MSBC Group • Nagpur, IN
Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services.Lead the E-Comms data pipeline within Compass’s Application Simplification workstream : design, ...Show more
Last updated: 1 hour ago • Promoted • New!
Data Engineer
Alp Consulting Ltd. • Nagpur, IN
Architect and maintain our Amazon Redshift Serverless data warehouse.Design and implement ETL pipelines from operational Redshift to staging (DSA), landing (DLA), and TDW layers.Model data using st...Show more
Last updated: 30+ days ago • Promoted
Principal Data Engineer
KPI Partners • Nagpur, IN
We are seeking a highly skilled Principal Data Engineer to join our dynamic team at KPI Partners.Lead data engineering initiatives and projects from conception to delivery.Design and implement scal...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Data Architect- Snowflake
USEReady • Nagpur, IN
USEReady is a data and analytics firm that provides the strategies, tools, capability, and capacity that businesses need to turn their data into a competitive advantage.
USEReady partners with cloud...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Data Engineer
Donyati • Nagpur, IN
We are seeking a highly skilled Senior Data Engineer to join our team in building a modern data platform on AWS.You will play a key role in transitioning from legacy systems to a scalable, cloud-na...Show more
Last updated: 1 day ago • Promoted
Senior Data Engineer
TEKsystems Global Services in India • India
Develop and manage data pipelines using ADF and Azure data bricks.Expert knowledge in data warehousing, data modeling and ETL.
Good to have datastage exposure.Expertise in on-prem to azure cloud mig...Show more
Last updated: 1 hour ago • Promoted • New!
Senior Data Engineer
SAIVA AI • Nagpur, IN
We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show more
Last updated: 30+ days ago • Promoted
Senior Data Platform Engineer
Black Dog Labs • Nagpur, IN
Remote (collaboration across time zones), India or LATAM preferred.Proficient English communication.Data Engineering / Backend Engineering / DevOps.
We’re looking for a hands-on Senior Data Platform...Show more