Talent.com
Data Pipeline Architect

Data Pipeline Architect

EXLChennai, Republic Of India, IN
30+ days ago
Job description

Responsibilities :

  • Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
  • Create the data integration and data diagram documentation.
  • Lead the data validation, UAT and regression test for new data asset creation.
  • Create and maintain data models, including schema design and optimization.
  • Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills :

  • Strong knowledge on Python and Pyspark
  • Expectation is to have ability to write Pyspark scripts for developing data workflows.
  • Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
  • Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
  • Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
  • Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
  • Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
  • Expectation is to have strong problem-solving and troubleshooting skills.
  • Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
  • Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
  • 3-7 years of experience in Data Engineer.
  • Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
  • Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
  • Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.
    • Create a job alert for this search

    Data Pipeline Architect • Chennai, Republic Of India, IN

    Related jobs
    • Promoted
    Solution & Data Architect - Azure Applications

    Solution & Data Architect - Azure Applications

    InfotechtionNagpur, IN
    Solution & Data Architect - Azure Applications .Location : India (Remote) | Employment Type : Full-time .Senior - 8-11+ Years of experience. We are seeking an accomplished Solution and Data Architect ...Show moreLast updated: 3 days ago
    • Promoted
    Principal / Senior Data Architect

    Principal / Senior Data Architect

    AaysNagpur, IN
    Position : Principal / Senior Data Architect.You will act as a key member of the consulting team helping Clients to re-invent their corporate finance function by leveraging advanced analytics.You wil...Show moreLast updated: 30+ days ago
    • Promoted
    GCP Data Architect

    GCP Data Architect

    Tata Consultancy Servicesnagpur, maharashtra, in
    Job Title : - GCP Data Architect.Design and Implement Data Architectures : Architect and build scalable, end-to-end data solutions on GCP, encompassing data ingestion, transformation, storage, and co...Show moreLast updated: 25 days ago
    • Promoted
    Senior Architect - Data

    Senior Architect - Data

    Quantiphinagpur, maharashtra, in
    Quantiphi is an award-winning Data Science and Machine Learning Software and Services Company focused on helping organizations translate the big promise of Machine Learning technologies into quanti...Show moreLast updated: 30+ days ago
    • Promoted
    Data Pipeline Architect

    Data Pipeline Architect

    Philodesign Technologies IncRepublic Of India, IN
    We’re Hiring : Enterprise Data Engineer.Are you passionate about building large-scale, high-performance data solutions? We’re looking for an experienced. Implement and orchestrate data workflows with...Show moreLast updated: 1 day ago
    • Promoted
    Enterprise Data Architect

    Enterprise Data Architect

    DBiz.aiNagpur, IN
    You will ensure that data is modeled, structured and integrated across the organization to support analytics and AI / ML initiatives at scale. You will work closely with business stakeholders and tech...Show moreLast updated: 14 days ago
    • Promoted
    Data Pipeline Architect

    Data Pipeline Architect

    IntraEdgeRepublic Of India, IN
    We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team. You will be responsible for building scalable and reli...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Lead Data Pipeline Architect

    Lead Data Pipeline Architect

    Lear CorporationPune, Republic Of India, IN
    Lear, a global automotive technology leader in Seating and E-Systems, is Making every drive better by delivering intelligent in-vehicle experiences for customers around the world.With over 100 year...Show moreLast updated: 12 hours ago
    • Promoted
    Data Architect

    Data Architect

    CareerUS SolutionsIndia, India
    The Data Architect is responsible for.This includes ensuring that data is.The role focuses on developing.Collaborate with software engineers, analysts, and stakeholders to understand data needs and...Show moreLast updated: 1 day ago
    • Promoted
    Data Architect

    Data Architect

    RecroNagpur, IN
    Central Data Office – Data Architect.SMEs, engineers, and developers to design and govern enterprise-level data architectures, ensuring alignment with corporate data principles and standards.This p...Show moreLast updated: 3 days ago
    • Promoted
    Data Architect

    Data Architect

    Tech Mahindranagpur, maharashtra, in
    Strong experience architecting and delivering.Proficiency in Azure services including.Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Storage, Azure Databricks.Python, Spark, or relate...Show moreLast updated: 3 days ago
    • Promoted
    AI Data Architect

    AI Data Architect

    Tata Consultancy Servicesnagpur, maharashtra, in
    We are seeking an inventive Data Architect for AI with 8–15 years of experience to lead the strategic design and implementation of enterprise-scale AI solutions. ETL / ELT, Big Data cloud data solutio...Show moreLast updated: 20 days ago
    • Promoted
    Azure Data Lead / Architect

    Azure Data Lead / Architect

    TEKsystems Global Services in Indianagpur, maharashtra, in
    Notice Period : Immediate to15 Days.Please find the below Azure Data Lead / Architect JD.Azure AI skills, Azure Open AI, AI Foundry. Excellent communication and interpersonal skills, with the ability t...Show moreLast updated: 3 days ago
    • Promoted
    • New!
    Data Pipeline Architect

    Data Pipeline Architect

    thinkbridgePune, Republic Of India, IN
    We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 12 hours ago
    • Promoted
    Databricks Data Architect

    Databricks Data Architect

    Tata Consultancy Servicesnagpur, maharashtra, in
    Experience with machine learning frameworks and pipelines, particularly in Databricks.Familiarity with containerized environments (e. Docker, Kubernetes) for deploying data workflows.Should be able ...Show moreLast updated: 3 days ago
    • Promoted
    • New!
    Azure Data Architect

    Azure Data Architect

    Programmers.ionagpur, maharashtra, in
    We offer a vibrant and collaborative work environment, cutting-edge tools and technologies, and ample opportunities for professional growth. Job Title : Azure Data Architect.Experience required : 15+ ...Show moreLast updated: 6 hours ago
    • Promoted
    Lead Data Pipeline Architect

    Lead Data Pipeline Architect

    CareerXperts ConsultingPune, Republic Of India, IN
    As Director of Engineering, you will lead the technical direction and execution of our engineering organization.You will be directly involved in system design, architecture.You will work with senio...Show moreLast updated: 1 day ago
    • Promoted
    Data Architect

    Data Architect

    Tata Consultancy Servicesnagpur, maharashtra, in
    Role- Snowflake Data Architect.Cloud data warehouse technologies on Snowflake and AWS, Azure or GCP.At least one End-to-end Snowflake implementation is a must covering all aspects including archite...Show moreLast updated: 30+ days ago