Python Systems Engineer – LLM EvaluationSupercoder • Vellore, IN

No longer accepting applications

Python Systems Engineer – LLM Evaluation

Supercoder • Vellore, IN

6 days ago

Job description

Greetings and thank you for visiting our job post.

Supercoder is an AI-powered career development platform connecting developers worldwide to remote job opportunities with competitive payment.

Type of work : 100% Remote

Overview :

The client is hiring Python / Linux Engineers to design complex system-level evaluation tasks for LLMs. Design advanced benchmark tasks that evaluate the capabilities of modern Large Language Models (LLMs) such as ChatGPT, Claude, and other AI systems.

This role focuses on building realistic, technically challenging engineering scenarios that test model reasoning, debugging, and problem-solving abilities.

What You Will Do

Design complex, realistic engineering tasks to evaluate LLM reasoning, coding, debugging, and system understanding.

Build Python- and Linux-based workflows, pipelines, and multi-step scenarios.

Create reproducible environments using Python, Shell, and CLI tools.

Develop tasks that measure code comprehension, debugging, refactoring, and optimization.

Write clear technical documentation : problem statements, constraints, expected outputs, and detailed edge cases.

Use LLM tools (ChatGPT, Claude, etc.) to validate tasks and analyze model performance.

Must-Have Qualifications

5+ years of professional software development experience.

Strong Python : modular code design, debugging complex programs, structured codebases.

Proficiency with Linux, Shell scripting, Bash, and command-line tools.

Solid technical English writing ability.

Strong reasoning, analytical thinking, and problem-solving skills.

Ability to design logical multi-step engineering scenarios.

Nice-to-Have Skills

Experience creating benchmark datasets, online judge problems, coding tests, or technical challenges.

Background with ICPC, Codeforces, Kaggle, or competitive programming.

Familiarity with Docker, Git, and CI / CD pipelines.

Experience with ML / AI or data-intensive engineering environments.

Who Will Excel in This Role

Engineers who enjoy designing difficult problems rather than simple feature development.

Developers who are strong at debugging, identifying subtle issues, and understanding complex system interactions.

Engineers who work well independently and can define their own approach.

Individuals interested in LLM evaluation, AI reliability, and technical task design.

Create a job alert for this search

Engineer Llm • Vellore, IN

Related jobs

DLP Engineer

Team1 Consulting • Vellore, IN

Team1 Consulting is a leading System Integrator specializing in IT infrastructure, software, cyber security, cloud solutions, and generative AI domains. We deliver cutting edge IT solutions tailored...Show more

Last updated: 17 hours ago • Promoted • New!

System Engineer

Karix • Vellore, IN

Systems Engineer responsibilities include deploying product updates, identifying production issues and implementing integrations that meet customer needs. The position works closely with internal te...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

Aceolution • Vellore, IN

Data Engineer – Python Expert(Freelance Role).We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) developmen...Show more

Last updated: 30+ days ago • Promoted

Generative AI Python Senior Specialist

Coforge • vellore, tamil nadu, in

Generative AI Python Senior Specialist.Python Senior Specialist to design and implement AI / ML and GenAI solutions.The ideal candidate will have strong programming and statistical skills, with hands...Show more

Last updated: 30+ days ago • Promoted

Databricks Data Engineer Lead – Sustainability Project

Blue Cloud Softech Solutions Limited • Vellore, IN

BCSS is seeking a Databricks Data Engineer to support its enterprise-wide Sustainability initiative.The engineer will be responsible for building data pipelines and models to support product-level ...Show more

Last updated: 17 hours ago • Promoted • New!

AI / ML Engineer

Innodata Inc. • Vellore, IN

Our AI-driven platforms and expert teams empower clients in healthcare, life insurance, and other industries to identify risks, improve efficiency, and make smarter decisions.By combining proprieta...Show more

Last updated: 13 days ago • Promoted

Freelance AI / ML Engineer (ETL + MLOps + AWS)

ThreatXIntel • Vellore, IN

ThreatXIntel is a dedicated cybersecurity startup specializing in protecting businesses and organizations from cyber threats. With services such as cloud security, web and mobile security testing, c...Show more

Last updated: 18 hours ago • Promoted • New!

Senior Data Engineer

Tata Consultancy Services • Vellore, IN

Job Title : Senior Data Engineer.Required Skillset : Python, Spark, Databricks, AWS (S3, Glue, AirFlow, Cloudwatch, Lambda). Master Databricks tools (job creation, cluster, notebook) and be able to qu...Show more

Last updated: 30+ days ago • Promoted

Senior Python Engineer (LLMs & Django)

LeoRix • vellore, tamil nadu, in

Senior Python Engineer (LLMs & Django) — LeoRix (AI Enterprise Hub).LeoRix is an AI-driven Enterprise Hub designed to unify and automate end-to-end business workflows. Our vision is to create an int...Show more

Last updated: 17 hours ago • Promoted • New!

Prompt Engineer

Innodata Inc. • Vellore, IN

Demonstrated experience programmatically using LLMs to automate data labeling, classification, localization and annotation tasks. Strong expertise in Python for NLU, for data processing & transforma...Show more

Last updated: 14 days ago • Promoted

AI / ML Engineer

Reqpedia • Vellore, IN

Proficiency in enterprise-approved AI tools as part of their day-to-day responsibilities.This includes, but is not limited to : . Consistent Use : Maintain a minimum of 90% weekly usage of AI tools suc...Show more

Last updated: 17 hours ago • Promoted • New!

ML Ops

EXL • Vellore, IN

Deploy, monitor, and scale ML models on.GCP (Vertex AI, GKE, Cloud Functions).GitHub Actions / Jenkins / cloud-native tools. Containerize and orchestrate workloads with.MLflow, Feast, Prometheus / Gra...Show more

Last updated: 30+ days ago • Promoted

Platform Engineer – Python / Databricks / Notebooks / Kubernetes

Syntasa • Vellore, IN

Platform Engineer – Python / Databricks / Notebooks / Kubernetes.Syntasa is seeking a high-caliber and dedicated.Syntasa Technologies India Private Limited. This position offers an exciting opportuni...Show more

Last updated: 24 days ago • Promoted

Lead Engineer

Hyqoo • Vellore, IN

Design, deploy, and manage AWS cloud infrastructure, including EC2 instances, S3 buckets, VPCs, RDS databases, and Lambda functions. Assist in the design, implementation, and maintenance of backup, ...Show more

Last updated: 9 days ago • Promoted

Engineer : Senior LLM Optimization (LLMO) / GEO Expert – Google Vertex

Proso.ai • vellore, tamil nadu, in

Generative Engine Optimization (GEO).You’ll help make sa products and services.AI-driven assistants and LLMs like ChatGPT, Copilot, and Gemini. Optimize LLM pipelines on Google Vertex AI : .Design RAG...Show more

Last updated: 12 days ago • Promoted

LLM Scientist

Insight Global • vellore, tamil nadu, in

Familiarity with LangSmith, LangFuse, or similar LLM monitoring tools.Show more

Last updated: 1 day ago • Promoted

Senior AI / ML Engineer

Luxoft • Vellore, IN

Our client, one of the leading Agriculture Companies, is modernising their landscape and adopting AI and innovations in their process. We are seeking a highly skilled and innovative Python / AI Engine...Show more

Last updated: 11 days ago • Promoted

Search Engineer

YourTribe • Vellore, IN

Design & implement search solutions.Architect and develop advanced search features using.OpenSearch / Elasticsearch, including custom analysers, tokenisers, and scoring algorithms.Create and maintain...Show more

Last updated: 30+ days ago • Promoted