Talent.com
This job offer is not available in your country.
Data Engineer - Big Data Technologies

Data Engineer - Big Data Technologies

Zyoin GroupChennai
17 days ago
Job description

The core responsibilities for the job include the following :

Data Pipeline Development :

  • Design, develop, and maintain data pipelines to ingest, process, and transform data from various sources into usable formats.
  • Implement data integration solutions that connect disparate data systems, including databases, APIs, and third-party data sources.

Data Storage and Warehousing :

  • Create and manage data storage solutions, such as data lakes, data warehouses, and NoSQL databases.
  • Optimize data storage for performance, scalability, and cost-efficiency.
  • Data Quality and Governance :

  • Establish data quality standards and implement data validation and cleansing processes.
  • Collaborate with data analysts and data scientists to ensure data consistency and accuracy.
  • ETL (Extract, Transform, Load) :

  • Develop ETL processes to transform raw data into a structured and usable format.
  • Monitor and troubleshoot ETL jobs to ensure data flows smoothly.
  • Data Security and Compliance :

  • Implement data security measures and access controls to protect sensitive data.
  • Ensure compliance with data privacy regulations and industry standards (e. g., GDPR, HIPAA).
  • Performance Tuning :

  • Optimize data pipelines and queries for improved performance and efficiency.
  • Identify and resolve bottlenecks in data processing.
  • Data Documentation :

  • Maintain comprehensive documentation for data pipelines, schemas, and data dictionaries.
  • Create and update data lineage and metadata documentation.
  • Scalability and Reliability :

  • Design data infrastructure to scale with growing data volumes and business requirements.
  • Implement data recovery and backup strategies to ensure data availability and resilience.
  • Collaboration :

  • Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver data solutions.
  • Required Technical Skills :

  • Programming Languages : Proficiency in Python is essential. Experience with SQL for database querying and manipulation is also required. Knowledge of Java or Scala is a plus.
  • Big Data Technologies : Hands-on experience with big data frameworks such as Apache Spark, Hadoop, or Flink.
  • Cloud Platforms : Strong experience with at least one major cloud provider's data services (AWS, GCP, or Azure). This includes services like AWS S3, Redshift, Glue; GCP BigQuery, Dataflow, Cloud Storage; or Azure Synapse, Data Factory, Blob Storage.
  • Databases and Warehousing : Deep expertise in relational databases (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Cassandra). Experience with modern data warehouses (Snowflake, Redshift, BigQuery) is highly preferred.
  • ETL / ELT Tools : Experience with ETL / ELT tools like Apache Airflow for workflow orchestration, or other similar tools.
  • Data Governance and Security : Understanding of data governance principles and experience with implementing security measures like encryption, access controls, and data masking.
  • Version Control : Proficiency with Git for version control.
  • Containerization : Experience with Docker and Kubernetes is a plus.
  • (ref : hirist.tech)

    Create a job alert for this search

    Big Data Engineer • Chennai

    Related jobs
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Eucloid Data SolutionsChennai, Tamil Nadu, India
    Eucloid is looking for a skilled Data Engineer to join our team and contribute to the design, development, and optimization of data frameworks pipelines using Python, Elasticsearch, SQL DB and Queu...Show moreLast updated: 7 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DeltacubesChennai, IN
    Build and maintain scalable ETL / ELT pipelines.Work with Snowflake and BigQuery for data storage.Implement orchestration with Airflow or Prefect. Integrate data workflows with Python.Optimize data pi...Show moreLast updated: 15 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    InfogainChennai, IN
    Big Data Engineer (Lead) : As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers. You will work closely with clients and team members to understand their req...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data SolutionsChennai, IN
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 7 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Innodata Inc.Chennai, IN
    CI / CD practices, Databricks (Spark), Python, Github and SQL.The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and...Show moreLast updated: 25 days ago
    • Promoted
    Senior Big Data Engineer

    Senior Big Data Engineer

    VeltrisChennai, IN
    Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries. We specialize in delivering next-gene...Show moreLast updated: 14 days ago
    • Promoted
    Data Engineer

    Data Engineer

    HISH IT SERVICESChennai, IN
    Location : Remote(Banglore,Chennai,Pune).Pay : 14LPA - 18 LPA(Based on Experience).Timings : A couple of hours overlap with EST, as the client is Canada-based (till 12AM IST).Start Date : 20th Octob...Show moreLast updated: 6 days ago
    • Promoted
    Data Engineer

    Data Engineer

    RevXChennai, IN
    RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    ACL Digitalchennai, tamil nadu, in
    Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.Design, develop, and optimize AWS pipeline as applicable. Implement and manage GitHub asset bun...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    BrightEdgeChennai, IN
    BrightEdge About Us BrightEdge is a leading SEO and content performance marketing platform that transforms online content into tangible business results. Our platform processes massive amounts of da...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    AceolutionChennai, IN
    We are looking for a freelancer to engage with us for 20-40 hours per week.Kindly find the JD below for your reference.Design, develop, and maintain scalable data pipelines and workflows.Work exten...Show moreLast updated: 7 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Insight GlobalChennai, IN
    Notice Period : This is a ASAP hire - only apply if you can start 2 weeks after offer is extended!.Priority scheduling for candidates who message on with resume. Design, develop, and maintain robust...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    ImpetusChennai, Tamil Nadu, India
    Bigdata, Pyspark, Python ,Hadoop / HDFS; Spark;.Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity. Collaborates with analytics and business...Show moreLast updated: 23 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Kumaran SystemsChennai, IN
    Strong hands-on experience in Databricks.Proven expertise in building and managing data ingestion pipelines.Exposure to Databricks data ingestion jobs along with incident management frameworks.Expe...Show moreLast updated: 7 days ago
    • Promoted
    Data Engineer

    Data Engineer

    EveriseChennai, IN
    Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 6 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Bahwan CyberTekChennai, IN
    Job Title : Data Engineer – Google Cloud Platform (GCP).We are seeking a skilled and motivated Data Engineer with hands-on experience in building scalable data pipelines and cloud-native data soluti...Show moreLast updated: 17 days ago
    • Promoted
    • New!
    Data Engineer

    Data Engineer

    Kline + CompanyChennai, IN
    Here at Kline our data capabilities have grown exponentially over the last four years.Having gone through a rapid digitization process and becoming a cloud-native corporation, we are looking for to...Show moreLast updated: 7 hours ago
    • Promoted
    Data Engineer

    Data Engineer

    IntraEdgeChennai, IN
    Snowflake, AWS (Lambda, Glue), DBT, and SQL.The ideal candidate will be responsible for enabling seamless data integration, transformation, and analytics to support business intelligence and advanc...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    CYGNVSChennai, Tamil Nadu, India
    We’re looking for Data Engineers on our Analytics Engineering team - you’ll collaborate with teams across the globe to build pipelines and systems to deliver data insights critical to our customers...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Kanerika Incchennai, tamil nadu, in
    Databricks, Microsoft Fabric, or Snowflake.You will also play a key role in enabling effective data visualization and reporting for business stakeholders. Databricks, Fabric, or Snowflake.Power BI d...Show moreLast updated: 19 days ago