Talent.com
Senior Data Platform Engineer

Senior Data Platform Engineer

Black Dog Labsvijayapura, India
13 days ago
Job description

Position : Senior Data Platform Engineer

Location : Remote (collaboration across time zones), India or LATAM preferred

Engagement Type : Contract (12+ months)

Language : Proficient English communication

Experience : 5+ years in Data Engineering / Backend Engineering / DevOps

About the Role

We’re looking for a hands-on Senior Data Platform Engineer to build and operate production-grade data pipelines and services. You’ll work across ingestion, transformation, orchestration, and APIs; automate environments with Terraform; instrument observability; and write tests to keep things reliable. Azure experience is preferred; strong AWS experience is also acceptable. You should communicate clearly (async chat and live sessions), learn fast, and use AI coding tools (e.g., Claude Code) effectively without relying on them exclusively.

Key Responsibilities

  • Data pipelines : Design, build, and operate reliable batch and change-driven pipelines from multiple sources; schedule / orchestrate jobs; handle schema evolution and failures gracefully.
  • Transformations & modeling : Implement clean, tested merge and transformation pipelines and produce models that are easy to consume for products and analytics; tune SQL for performance.
  • APIs & integration : Build or collaborate on APIs for data access and updates; design request / response contracts; handle auth (OAuth2 / OIDC / JWT), idempotency, validation, and auditing.
  • Infrastructure & DevOps : Provision and manage cloud resources with Terraform; automate build / test / deploy with CI / CD; write solid shell scripts for glue / ops tasks.
  • Observability & reliability : Instrument structured logs, metrics, and traces; define alert policies and on-call runbooks; track SLOs and drive root-cause prevention.
  • Security basics : Apply least-privilege access, secrets management, and encryption in transit / at rest; collaborate with the team on compliance requirements.
  • Quality & tests : Write unit, integration, and data-quality tests (including SQL tests) to keep pipelines and services correct and maintainable.
  • Collaboration & docs : Communicate clearly in writing and in person; produce concise docs (data contracts, mappings, runbooks); work closely with product / engineering / analytics.

Required Skills & Qualifications

  • Strong SQL (query tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL).
  • Python for data work and services; comfortable with shell scripting (bash preferred, powershell proficiency if not bash).
  • Terraform and IaC fundamentals; CI / CD experience (version control workflows, automated tests, environment promotion).
  • Cloud : Azure preferred (e.g., storage, compute, identity, monitoring). If not Azure, very strong AWS background.
  • Data integration : Experience with ingestion (batch + change-driven), orchestration, schema / versioning, and resilient retries / replays.
  • Workflow orchestration & scheduling : Prefect preferred (flows / deployments, retries / backoff, scheduling, observability); Airflow, Dagster, or similar also acceptable.
  • APIs & auth : Practical experience with REST patterns, pagination, validation, rate limiting, and OAuth2 / OIDC / JWT.
  • Observability : Familiar with logs / metrics / traces and setting actionable alerts and dashboards.
  • Testing mindset : Habit of writing tests for code and SQL; comfortable with fixtures / test data and CI.
  • Communication : Clear, concise writing and verbal comms; comfortable in async chat and live sessions.
  • AI tooling : Ready to use Claude Code (and similar assistants) from day one to accelerate work—while exercising judgment and writing your own code / tests.
  • Preferred Qualifications (Nice to Have)

  • Regulatory / compliance awareness (e.g., data protection or public-sector standards) and how it impacts design / operations
  • Analytics experience (dim / fact modeling, BI consumption patterns)
  • Full-stack exposure (can prototype simple UI / ops views when needed)
  • Data platform architecture exposure (storage / layout choices, catalog / lineage, governance concepts)
  • Familiarity with event streaming / message queues, job schedulers / orchestrators, and columnar formats (e.g., Parquet)
  • Experience with cloud monitoring stacks (Azure Monitor / Application Insights, CloudWatch, Datadog, New Relic, etc.)
  • Engagement Details

  • Duration : 12+ months (extension / full-time possible)
  • Schedule : Flexible, with overlap for key meetings across time zones
  • Compensation : Competitive, based on experience
  • Mode : Remote; available for virtual collaboration and on-call windows as agreed
  • How to Apply

    Please share :

  • Resume / CV highlighting data engineering, DevOps / IaC, and / or API / auth work
  • Brief note (one paragraph) about a recent pipeline / service : requirements, your role, tech, validation / tests, deployment / ops, outcome
  • Links to code samples or design docs (if available and shareable)
  • Apply via LinkedIn or by sending an email to

    Create a job alert for this search

    Senior Data Engineer • vijayapura, India

    Related jobs
    • Promoted
    DataOps Engineer

    DataOps Engineer

    Landmark GroupBengaluru, Karnataka, India
    Responsible for the governance ensuring data availability, accuracy, and performance across environments as an expert Azure Data Ops with 6-8 years of experience. Orchestrate, monitor, and automate ...Show moreLast updated: 10 days ago
    • Promoted
    Big Data Engineer

    Big Data Engineer

    Tata Consultancy ServicesBengaluru, Karnataka, India
    Job Title : - Data Engineer - Pyspark.Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL. Good Spark, Pyspark,Big Data experience.Spark UI / Optimization / debugging ...Show moreLast updated: 30+ days ago
    • Promoted
    Python Squish Developer

    Python Squish Developer

    L&T Technology ServicesGreater Bengaluru Area, India
    Must have strong background in implementing and extending exiting automation using python with Squish.Exposure in squish test frameworks. Able to detail out requirements and write and maintain test ...Show moreLast updated: 20 days ago
    • Promoted
    Datastage Developer

    Datastage Developer

    Tata Consultancy ServicesGreater Bengaluru Area, India
    Bengaluru / Chennai / Hyderabad / Indore / Ahmedabad / Mumbai.Required Technical Skill Set : -.Datastage, UNIX Shellscripts, CA Autosys. Responsibility of / Expectations from the Role.Associate responsible for...Show moreLast updated: 20 days ago
    • Promoted
    snowflake Cloud Database Administrator

    snowflake Cloud Database Administrator

    Leading automobile GCCBengaluru, Karnataka, India
    Cloud Platform Expertise : Familiarity with leading cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, and Snowflake is essential. Understanding their database offerings, such as AWS...Show moreLast updated: 30+ days ago
    • Promoted
    Power BI

    Power BI

    Relevance LabGreater Bengaluru Area, India
    Power BI using DAX, Power Query (M), and data modeling best practices.API integrations, and custom analytics not natively supported by Power BI. SharePoint, Dataverse, SQL Server, Excel, APIs, etc.C...Show moreLast updated: 18 days ago
    • Promoted
    Python Developer ( only Bangalore, Hyderabad Locations) - 5-8 Years experience only, Immediate - 15 Days notice

    Python Developer ( only Bangalore, Hyderabad Locations) - 5-8 Years experience only, Immediate - 15 Days notice

    Tata Consultancy ServicesGreater Bengaluru Area, India
    Skill – Python Backend developer.Location : Bangalore / Hyderabad.Design, develop, and maintain backend services using FastAPI or Spring Boot. Build and maintain RESTful APIs for integration with fron...Show moreLast updated: 20 days ago
    • Promoted
    Snowflake Cortex Developer

    Snowflake Cortex Developer

    Tata Consultancy ServicesGreater Bengaluru Area, India
    Job Title : Snowflake Cortex Developer.Required Skillset : Snowflake Cortex, SQL, Snowflake.Location : Delhi / Bangalore / Hyderabad / Pune / Mumbai. Strong data engineer ( snowflake + SQL) with good under...Show moreLast updated: 12 days ago
    • Promoted
    SAP Consultant

    SAP Consultant

    YASH TechnologiesBangalore Rural, Karnataka, India
    Bangalore Rural, Karnataka, India (Hybrid).Dear Connection Yash Technologies is looking for Senior SAP FSCM Consultant.Our SAP Service Line is currently looking for industry-leading seasoned SAP FS...Show moreLast updated: 11 days ago
    • Promoted
    Senior Python Full-Stack Developer (Python+React)

    Senior Python Full-Stack Developer (Python+React)

    LuxoftGreater Bengaluru Area, India
    We have an ambitious goal to migrate a legacy system written in HLASM (High-Level Assembler) from the mainframe to a cloud-based Java environment for one of the largest banks in the USA.We are look...Show moreLast updated: 18 days ago
    • Promoted
    DevSecOps Engineer

    DevSecOps Engineer

    Best Infosystems Ltd.Greater Bengaluru Area, India
    DevSecOps Engineer_Full-Time_Bangalore, Pune.DevSecOps Engineering with 7 plus years of experience, he / she will use their experience and knowledge to design, implement, and support complex DevSecOp...Show moreLast updated: 30+ days ago
    • Promoted
    Snowflake Developer

    Snowflake Developer

    Tata Consultancy ServicesGreater Bengaluru Area, India
    Job Title : Snowflake Developer.Required Skillset : Snowflake architecture, Snowpipe, data cloning, and time travel, SQL, Data Modeling. Deep understanding of cloud data warehousing and expertise in d...Show moreLast updated: 30+ days ago
    • Promoted
    Software Engineer, AI Agents

    Software Engineer, AI Agents

    Asha Health (YC F24)Greater Bengaluru Area, India
    Asha Health is a New York based seed stage startup that helps medical practices launch their own AI clinics.We're backed by Y Combinator, General Catalyst, 186 Ventures, Reach Capital and many more...Show moreLast updated: 20 days ago
    • Promoted
    User Interface (UI) Intern - Qt / Python / MATLAB / LabVIEW

    User Interface (UI) Intern - Qt / Python / MATLAB / LabVIEW

    Ci4 - Autonomous DefenseGreater Bengaluru Area, India
    User Interface (UI) Intern - Qt / Python / MATLAB / LabVIEW.We are looking for creative and technically-oriented UI Interns to assist in designing, developing, and testing interactive graphical use...Show moreLast updated: 13 days ago
    • Promoted
    Databricks Engineer

    Databricks Engineer

    Tata Consultancy ServicesBengaluru, Karnataka, India
    Skills : Azure Databricks, Pyspark, ADLS, ADF.Responsibility of / Expectations from the Role : .Designing and building Databricks data pipelines is required. Experience in transforming the data using A...Show moreLast updated: 28 days ago
    • Promoted
    GCP Dataflow Engineer

    GCP Dataflow Engineer

    Webologix Ltd / INCBengaluru, Karnataka, India
    Job Position : GCP Dataflow Engineer.GCP Storage, GCP BigQuery, GCP Data Proc, GCP Cloud Composer, GCP DMS, Apache airflow, Java, Python, Scala, GCP Data stream, Google Analytics Hub, GCP Workflows,...Show moreLast updated: 30+ days ago
    • Promoted
    Machine Learning Specialist

    Machine Learning Specialist

    upGrad RekrutBangalore Rural, Karnataka, India
    Coding : Write clean, efficient, and well-documented Python code adhering to OOP principles (encapsulation, inheritance, polymorphism, abstraction). Experience with Python and related libraries (e.Te...Show moreLast updated: 18 days ago
    • Promoted
    OIC Solution Architect

    OIC Solution Architect

    OracleGreater Bengaluru Area, India
    Location : India (Bangalore, Hyderabad, Noida).About Oracle CSS Apps Delivery Team.Oracle’s Customer Success Services (CSS) Apps Delivery Team partners with clients to drive transformation and innov...Show moreLast updated: 10 days ago