Talent.com
Principal Data Pipeline Engineer

Principal Data Pipeline Engineer

Newscape ConsultingPune, Republic Of India, IN
5 days ago
Job description

About the Role

We are seeking a Senior Databricks Engineer to design, develop, and optimize scalable data pipelines and analytics solutions on the Databricks platform. The ideal candidate will be a hands-on expert in data engineering, cloud infrastructure, and distributed data processing, with a deep understanding of how to build reliable and performant data ecosystems to power business insights and AI / ML initiatives.

Key Responsibilities

Design &

  • Development :
  • Build and maintain scalable ETL / ELT data pipelines using Apache Spark and Databricks.
  • Develop Delta Lake architectures and optimize data storage for performance and cost.
  • Automate data ingestion, transformation, and orchestration processes across multiple data sources.
  • Build different stages of medallion architecture such as Bronze, Silver and Gold layer

Data Architecture & Modeling :

  • Collaborate with data architects to design data lakehouse and warehouse solutions.
  • Implement data quality frameworks, versioning, and governance standards (e.G., Unity Catalog).
  • Optimize Spark jobs, cluster configurations, and query performance.
  • Cloud & Infrastructure :
  • Work with Azure Databricks or AWS to configure and manage cloud-based data platforms.
  • Develop and manage CI / CD pipelines for Databricks notebooks, workflows, and jobs.
  • Implement security best practices for data access, encryption, and compliance.

    • Collaboration & Leadership :

  • Partner with data scientists, analysts, and product teams to enable data-driven decisions.
  • Provide technical mentorship to junior engineers.
  • Contribute to strategic planning and architecture reviews.

    • Required Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
  • 6+ years of experience in data engineering or software development.
  • 4+ years hands-on experience with Databricks and Apache Spark.
  • Strong proficiency in Python, SQL, and PySpark.
  • Experience with Delta Lake, Data Lakehouse, and Data Warehousing concepts.
  • Solid understanding of data modeling, ETL design, medallion architecture and distributed data processing.
  • Familiarity with CI / CD tools (Azure DevOps, GitHub Actions, Jenkins, etc.).
  • Cloud platform expertise in Azure, AWS, or GCP.

    • Preferred Qualifications

  • Experience with machine learning pipelines in Databricks (MLflow).
  • Knowledge of data governance and compliance frameworks.
  • Experience integrating Databricks with BI tools like Power BI, Tableau, or Looker.
  • Certifications such as :
  • Databricks Certified Data Engineer Associate or Professional
  • Azure Data Engineer Associate
  • AWS Big Data Specialty

    • Soft Skills

  • Strong problem-solving and analytical thinking.
  • Excellent communication and collaboration abilities.
  • Proactive mindset with ownership of deliverables.
  • Passion for learning emerging data technologies.

    • Why Join Us

  • Work with cutting-edge data technologies on large-scale projects.
  • Opportunity to shape the organization’s data strategy and architecture.
  • Collaborative and innovative culture focused on continuous growth.
    • Create a job alert for this search

    Data Pipeline Engineer • Pune, Republic Of India, IN

    Related jobs
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    IntelliasNagpur, IN
    Apache Flink / Apache Spark (Streaming).Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines. Kafka + Flink / Spark Streaming).Python for data engin...Show moreLast updated: 5 days ago
    • Promoted
    Founding Data Engineer

    Founding Data Engineer

    InSiteVerseNagpur, IN
    We are building a next-generation .Generative AI and advanced ML models.Founding Data Pipeline Engineer.You will work closely with the . You will also architect and optimize .Azure OLAP services suc...Show moreLast updated: 30+ days ago
    • Promoted
    Principal Engineer - Data Science

    Principal Engineer - Data Science

    Pull LogicNagpur, IN
    Pull Logic is transforming the retail supply chain with a revolutionary availability-oriented paradigm that directly links customer preferences with real-time inventory. Backed by cutting-edge resea...Show moreLast updated: 2 days ago
    • Promoted
    Data Engineer

    Data Engineer

    DigitalzoneNagpur, IN
    As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics. Develop and maintain scalable ETL / ELT pipelines using Py...Show moreLast updated: 5 days ago
    • Promoted
    • New!
    Data Engineer

    Data Engineer

    Pro5.aiNagpur, IN
    You’ll play a key role in ensuring reliable data flows, integrations, and preprocessing frameworks that power advanced GenAI systems. This is a high-impact opportunity within an.Design, build, and o...Show moreLast updated: 3 hours ago
    • Promoted
    Data Engineer

    Data Engineer

    DraconXNagpur, IN
    DraconX specializes in creating intelligent, scalable digital solutions that drive growth and innovation for startups and enterprises. As pioneers in AI business automation and SaaS platforms, we ex...Show moreLast updated: 1 day ago
    • Promoted
    • New!
    Data Engineer – ETL & Pipeline Development (5 to 10 yrs)

    Data Engineer – ETL & Pipeline Development (5 to 10 yrs)

    AIMLEAPNagpur, IN
    Data Engineer – ETL & Pipeline Development.Remote (Work from Home) / Bangalore / India.Tech / MCA / Computer Science / IT. IT / Data / AI / LegalTech / Enterprise Solutions.ETL workflows — not just ...Show moreLast updated: 17 hours ago
    • Promoted
    Principal Data Engineer

    Principal Data Engineer

    CodeMyMobileNagpur, IN
    Experience Required - 7 to 10 Years.Are you a Data Engineer who cares about clean engineering, autonomy, and solving real data challenges? If this sounds like you, we’d love to connect!.Email your ...Show moreLast updated: 5 days ago
    • Promoted
    Backend and Data Pipeline Engineer

    Backend and Data Pipeline Engineer

    JRD SystemsNagpur, IN
    Job Role : Backend and Data Pipeline Engineer - Python.We’re investing in technology to develop new products that help our customers drive their growth and transformation agenda.These include new da...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    IntraEdgeNagpur, IN
    We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team. You will be responsible for building scalable and reli...Show moreLast updated: 30+ days ago
    • Promoted
    Sr Data Engineer

    Sr Data Engineer

    Mitchell Martin Inc.Nagpur, IN
    Job Title : Senior Data Engineer.We are looking for a Senior Data Engineer to design, build, and optimize data pipelines and systems that power our analytics, reporting, and data-driven decision-mak...Show moreLast updated: 19 days ago
    • Promoted
    Data Engineer

    Data Engineer

    RecroNagpur, IN
    Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago
    • Promoted
    AWS Data Engineer

    AWS Data Engineer

    TerraGiGNagpur, IN
    Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR. Writing reusable, testable, and efficient code.Integration of data storage ...Show moreLast updated: 16 days ago
    • Promoted
    Python Data Engineer

    Python Data Engineer

    iVoyantNagpur, IN
    One of our clients is looking for an experienced Python Data Engineer to join their team.Strong Python Experience (web services, background jobs. we use Fast API).Data processing and reporting usin...Show moreLast updated: 16 days ago
    • Promoted
    Data Engineer

    Data Engineer

    VeraxionNagpur, IN
    Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 16 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    SAIVA AINagpur, IN
    We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 30+ days ago
    • Promoted
    GCP Data Engineer

    GCP Data Engineer

    Xsell ResourcesNagpur, IN
    Seeking a GCP Certified Data Engineer to work remotely from India for our Fortune 5 healthcare client in the US.No notice periods more than 15 days. GCP Google Cloud Professional Data Engineer Certi...Show moreLast updated: 4 days ago
    • Promoted
    Freelance Data Engineer

    Freelance Data Engineer

    upGradNagpur, IN
    We are seeking a highly skilled and motivated.The ideal candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and data warehouse solutions, utilizing a ...Show moreLast updated: 25 days ago