Talent.com
This job offer is not available in your country.
Data Engineer - Sr. Consultant level (Hadoop, Spark, Python, GenAI, 11-15 years exp)

Data Engineer - Sr. Consultant level (Hadoop, Spark, Python, GenAI, 11-15 years exp)

Visabangalore, India
23 hours ago
Job description

Job Description

This is a great opportunity to work with Engineering and AI team to scale and structure large scale data engineering and AI / ML that drives significant revenue for Visa. As a member of the Post Purchase Solutions your role will involve developing and implementing practices that will allow deployment of machine learning models in large data science projects.

You must be a hands-on expert able to navigate both data engineering and data science disciplines to build effective engineering solutions that support AIML models. You will partner closely with global stakeholders across products in VERA including Data Science to help create and prioritize our strategic roadmap.  You will then leverage your expert technical knowledge of data engineering, tools and data architecture in the design and creation of the solutions on our roadmap.

Essential Functions :

  • You’ll be the face of the team representing us in various technical forums.
  • You’ll help our customers achieve their business goals by understanding their requirements. You will build deep partnerships with product management to make sure that the products we are building deliver real value.
  • You’ll analyze business requirements and architect highly secure, robust, and scalable solutions.
  • You’ll drive internal proof of concept initiatives. When needed, quickly design and implement a prototype of a system or component with a proper architecture, and then lead or hand over to engineering team to implement.
  • You’ll champion efforts to architect, design and implement various components of our global transaction processing systems.
  • Your DNA will include following and creating software best practices and processes.
  • You’ll coach and mentor your team members and help them grow professionally. You will also create an atmosphere of mutual accountability.
  • You’ll play key role in meetings and discussions with cross-functional and non-technical teams.
  • This is a hybrid position. Expectation of days in the office will be confirmed by your Hiring Manager.

Qualifications

Basic Qualifications :

  • 11+ years of relevant work experience with a Bachelor’s Degree or with an Advanced degree
  • Solid experience in Big Data engineering, with experience in Hadoop, Apache Spark, Python, and SQL.
  • Proficiency in creating and managing large-scale data pipelines, handling complex, high-volume, multi-dimensional data, and machine learning models.
  • Expertise in developing ETL processes, maintaining Spark pipelines.
  • Expertise in Python and experience in REST APIs, container-based technologies, and implementing containerization strategies using Docker and Kubernetes.
  • Proficient in popular technologies and cloud services, such as Kafka, Hazlecast, AWS services, and containerized environments (Docker, K8s).
  • Skilled in Unix / Shell or Python scripting, using scheduling tools like Airflow and Control-M, and maintaining OPS tools.
  • Familiarity with Agile development incorporating TDD, CI / CD, and relational and non-relational databases.
  • Proven track record of building applications adhering to industry best practices for reliability, availability, scalability, and operability.
  • Ability to manage component security analysis and collaborate with security teams.
  • Strong work ethic, focus on immediate goals, and proven experience as a technical leader.
  • Passion for mentoring juniors and helping them reach new heights.
  • Demonstrated proficiency with programming languages, scripting languages, code quality initiatives, and operating systems.
  • Excellent communication and interpersonal skills, and a strong team player.
  • Gen AI expertise :
  • Have experience in building and enhancing RAG pipelines.
  • Have expertise in architecting, implementing and optimizing AI agents capable of planning, reasoning an acting.
  • Have a working expertise in building MCP tools and clients.
  • Have expertise in frameworks like Lang chain, Lang graph etc.
  • Expertly craft prompts for various LLMs and contributes to fine tuning of models for specific tasks to enhance performance and alignment.
  • Additional Information

    Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

    Create a job alert for this search

    Sr Data Engineer • bangalore, India

    Related jobs
    • Promoted
    Data Engineer(Azure / AWS, Python / Pyspark, SQL)

    Data Engineer(Azure / AWS, Python / Pyspark, SQL)

    Sail AnalyticsBengaluru, Karnataka, India
    Architect, develop, test and maintain scalable data warehouses and data pipelines.Expertise in SQL, PySpark / Python and Azure(ADB, ADF) or AWS(Glue, Lambda, Redshift). Bachelor's degree or equivalent...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Aceolutionhosur, tamil nadu, in
    We are looking for a freelancer to engage with us for 20-40 hours per week.Kindly find the JD below for your reference.Design, develop, and maintain scalable data pipelines and workflows.Work exten...Show moreLast updated: 9 days ago
    • Promoted
    Data Engineer

    Data Engineer

    ACL DigitalBengaluru, Karnataka, India
    Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.Design, develop, and optimize AWS pipeline as applicable. Implement and manage GitHub asset bun...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    EXLhosur, tamil nadu, in
    We are seeking a highly skilled.You will work closely with cross-functional teams to provide thought leadership deliver robust, secure, and high-performance data solutions that empower analytics an...Show moreLast updated: 27 days ago
    • Promoted
    Sr Data Engineer

    Sr Data Engineer

    Accion LabsBengaluru, Karnataka, India
    Data Engineer / Data Analyst (Hybrid Role).Data Analysis : Python, Pandas, NumPy, Strong SQL, EDA techniques, Data Visualization Tools, Statistical Knowledge. Data Engineering : SQL, ETL pipelines, Ai...Show moreLast updated: 5 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Bahwan CyberTekBengaluru, Karnataka, India
    We are seeking a skilled and motivated Data Engineer with 5–8 years of experience in building scalable data pipelines using Python, PySpark, and AWS services. The ideal candidate will have hands-on ...Show moreLast updated: 2 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Otomeyt AIhosur, tamil nadu, in
    We are seeking a highly skilled 7+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 19 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Manuh Technologieshosur, tamil nadu, in
    Strong proficiency in Python programming (Pandas, NumPy, PySpark, or similar).Hands-on experience with Dask for large-scale distributed data processing. Proven expertise as a Data Modeler (conceptua...Show moreLast updated: 22 days ago
    • Promoted
    Azure Data Engineer

    Azure Data Engineer

    Tata Consultancy Serviceshosur, tamil nadu, in
    Job Title : Azure Data Engineer.Required Skillset : Azure Databricks(Strong),scala(strong), Pyspark(Good to have).Azure Databricks using Scala, PySpark. Software Development Life Cycle experience.Work...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Deltacubeshosur, tamil nadu, in
    Build and maintain scalable ETL / ELT pipelines.Work with Snowflake and BigQuery for data storage.Implement orchestration with Airflow or Prefect. Integrate data workflows with Python.Optimize data pi...Show moreLast updated: 16 days ago
    • Promoted
    Lead Azure Data Engineer

    Lead Azure Data Engineer

    RandomTreeshosur, tamil nadu, in
    We’re a leading software company specializing in Artificial Intelligence, Machine Learning, Data Analytics, Innovative data solutions, Cloud-based technologies. If you're passionate about building r...Show moreLast updated: 27 days ago
    • Promoted
    Data Engineer

    Data Engineer

    TechVeritohosur, tamil nadu, in
    You will play a critical role in designing, building, and optimizing data workflows that enable scalable analytics and real-time insights. The ideal candidate is hands-on, detail-oriented, and passi...Show moreLast updated: 1 day ago
    • Promoted
    Data Engineer

    Data Engineer

    INFEC Serviceshosur, tamil nadu, in
    Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 6 days ago
    • Promoted
    Sr. Data Engineer

    Sr. Data Engineer

    Cognologix Technologiesbangalore district, karnataka, in
    Bangalore / Pune / Hyderabad / Remote.We are looking for self driven Data Professional to be a key member of our Data Practice. We help many of our clients make sense of their large investments in data – ...Show moreLast updated: 9 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    SAIVA AIhosur, tamil nadu, in
    We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 9 days ago
    • Promoted
    Data Engineer

    Data Engineer

    VAANTECHhosur, tamil nadu, in
    Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Kanerika Inchosur, tamil nadu, in
    Following are high level responsibilities that you will play but not limited to : .Analyze the Data Model and do GAP analysis with Business Requirements and Power BI. Design and Model Power BI schema....Show moreLast updated: 20 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Innodata Inc.hosur, tamil nadu, in
    CI / CD practices, Databricks (Spark), Python, Github and SQL.The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and...Show moreLast updated: 27 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data Solutionshosur, tamil nadu, in
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 9 days ago
    • Promoted
    Senior Azure Data Engineer

    Senior Azure Data Engineer

    Programmers.iohosur, tamil nadu, in
    Job Title : Senior Azure Data Engineer.Skills Required : Azure Synapse, Azure Data Factory, Azure Databricks, and ADLS Gen2, PySpark, SQL,Microsoft Fabrics. We are seeking a highly skilled and experie...Show moreLast updated: 30+ days ago