Talent.com
This job offer is not available in your country.
Big Data Engineer - Python / PySpark

Big Data Engineer - Python / PySpark

DexianChennai
16 days ago
Job description

Job Description :

We are intending to hire Data engineer to handle day-to-day activities involving data ingestion from multiple source locations, help identify data sources, to troubleshoot issues, and engage with a third-party vendor to meet stakeholders needs.

Work Location : Chennai or Hyderabad or Pune / WFO.

Shift hours : 2.00pm to 11.00pm IST.

  • Required Immediate Joiners.
    • Required Skills :
  • Python
  • Processing of large quantities of text documents
  • Extraction of text from Office and PDF documents
  • Input json to an API, output json to an API
  • Nifi (or similar technology compatible with current EMIT practices)
  • Basic understanding of AI / ML concepts
  • Database / Search engine / SOLR skills
  • SQL build queries to analyze, create and update databases
  • Understands the basics of hybrid search
  • Experience working with terabytes (TB) of data
  • Basic OpenML / Python / Azure knowledge
  • Scripting knowledge / experience in an Azure environment to automate
  • Cloud systems experience related to search and :
  • DataBricks
  • Snowflake
  • ESRI ArcGIS / SDE
  • New GenAI app being developed

Scope of work :

  • Ingest TB of data from multiple sources identified by the Ingestion Lead
  • Optimize data pipelines to improve on data processing, speed, and data availability
  • Make data available for end users from several hundred LAN and SharePoint areas
  • Monitor data pipelines daily and fix issues related to scripts, platforms, and ingestion
  • Work closely with the Ingestion Lead & Vendor on issues related to data Skills demonstrated :
  • SOLR Backend database
  • Nifi Data movement
  • Pyspark Data Processing
  • Hive & Oozie For jobs monitoring
  • Querying SQL, HQl and SOLR querying
  • Python

    • Behavioral Skills demonstrated :

  • Excellent communication skills
  • Ability to receive direction from a Lead and implement
  • Prior experience working in an Agile setup, preferred
  • Experience troubleshooting technical issues and quality control checking of work
  • Experience working with a globally distributed team in different

    • ref : hirist.tech)

    Create a job alert for this search

    Big Data Engineer • Chennai