Python Systems Engineer – LLM EvaluationSupercoder • Kolhapur, IN

No longer accepting applications

Python Systems Engineer – LLM Evaluation

Supercoder • Kolhapur, IN

5 days ago

Job description

Greetings and thank you for visiting our job post.

Supercoder is an AI-powered career development platform connecting developers worldwide to remote job opportunities with competitive payment.

Type of work : 100% Remote

Overview :

The client is hiring Python / Linux Engineers to design complex system-level evaluation tasks for LLMs. Design advanced benchmark tasks that evaluate the capabilities of modern Large Language Models (LLMs) such as ChatGPT, Claude, and other AI systems.

This role focuses on building realistic, technically challenging engineering scenarios that test model reasoning, debugging, and problem-solving abilities.

What You Will Do

Design complex, realistic engineering tasks to evaluate LLM reasoning, coding, debugging, and system understanding.

Build Python- and Linux-based workflows, pipelines, and multi-step scenarios.

Create reproducible environments using Python, Shell, and CLI tools.

Develop tasks that measure code comprehension, debugging, refactoring, and optimization.

Write clear technical documentation : problem statements, constraints, expected outputs, and detailed edge cases.

Use LLM tools (ChatGPT, Claude, etc.) to validate tasks and analyze model performance.

Must-Have Qualifications

5+ years of professional software development experience.

Strong Python : modular code design, debugging complex programs, structured codebases.

Proficiency with Linux, Shell scripting, Bash, and command-line tools.

Solid technical English writing ability.

Strong reasoning, analytical thinking, and problem-solving skills.

Ability to design logical multi-step engineering scenarios.

Nice-to-Have Skills

Experience creating benchmark datasets, online judge problems, coding tests, or technical challenges.

Background with ICPC, Codeforces, Kaggle, or competitive programming.

Familiarity with Docker, Git, and CI / CD pipelines.

Experience with ML / AI or data-intensive engineering environments.

Who Will Excel in This Role

Engineers who enjoy designing difficult problems rather than simple feature development.

Developers who are strong at debugging, identifying subtle issues, and understanding complex system interactions.

Engineers who work well independently and can define their own approach.

Individuals interested in LLM evaluation, AI reliability, and technical task design.

Create a job alert for this search

Engineer Llm • Kolhapur, IN

Related jobs

ML Ops

EXL • Kolhapur, IN

Deploy, monitor, and scale ML models on.GCP (Vertex AI, GKE, Cloud Functions).GitHub Actions / Jenkins / cloud-native tools. Containerize and orchestrate workloads with.MLflow, Feast, Prometheus / Gra...Show more

Last updated: 30+ days ago • Promoted

AIML Engineer

Tata Consultancy Services • sangli, maharashtra, in

Competencies (Technical / Behavioral Competency).AI / ML, Azure ML Studio, AI / ML On Databricks, Python & CICD Devops.Supervised and unsupervised ML and Predictive Analytics using Python • Feature gener...Show more

Last updated: 13 days ago • Promoted

Lead Software Statistician (SDLC, R / Python / Julia and "code repository tools; Jira, Confluence, BitBucket, TestRail")

Cytel • Kolhapur, IN

Lead the development and testing of our innovative and market-leading statistical software.Assist team members on the implementation of complex and innovative statistical algorithms in software pro...Show more

Last updated: 5 days ago • Promoted

Search Engineer

YourTribe • Kolhapur, IN

Design & implement search solutions.Architect and develop advanced search features using.OpenSearch / Elasticsearch, including custom analysers, tokenisers, and scoring algorithms.Create and maintain...Show more

Last updated: 30+ days ago • Promoted

AI / ML Engineer

Innodata Inc. • Kolhapur, IN

Our AI-driven platforms and expert teams empower clients in healthcare, life insurance, and other industries to identify risks, improve efficiency, and make smarter decisions.By combining proprieta...Show more

Last updated: 12 days ago • Promoted

Lead AI Engineer

Genisys Group • sangli, maharashtra, in

As an AI / ML Engineer at Genisys Group, you will be instrumental in developing and.AI solutions, with a strong focus on Large Language Models. LLMs) and Retrieval-Augmented Generation (RAG) technique...Show more

Last updated: 13 days ago • Promoted

Founding MLOps Engineer

Vectorial AI • Kolhapur, IN

Vectorial is a simulation engine platform powered by millions of synthetic users—state-of-the-art models that capture real human behavior—to deliver instant, nuanced validation across the entire pr...Show more

Last updated: 7 days ago • Promoted

Prompt Engineer

Innodata Inc. • Kolhapur, IN

Demonstrated experience programmatically using LLMs to automate data labeling, classification, localization and annotation tasks. Strong expertise in Python for NLU, for data processing & transforma...Show more

Last updated: 13 days ago • Promoted

Data Engineer

Aceolution • Kolhapur, IN

Data Engineer – Python Expert(Freelance Role).We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) developmen...Show more

Last updated: 30+ days ago • Promoted

MLOps Engineer

Capgemini • kolhapur, maharashtra, in

Experience in developing MLOps framework cutting ML lifecycle : model development, training, evaluation, deployment, monitoring including Model Governance. Expert in Azure Databricks, Azure ML, Unity...Show more

Last updated: 11 days ago • Promoted

LLM Engineer

HireAlpha • sangli, maharashtra, in

Experience working with LLMs and Generative AI.Experience building conversational bots.Experience using RAG base approaches. Understanding of financial concepts and investing would be a big plus but...Show more

Last updated: 11 hours ago • Promoted • New!

Senior AI / ML Engineer

Luxoft • Kolhapur, IN

Our client, one of the leading Agriculture Companies, is modernising their landscape and adopting AI and innovations in their process. We are seeking a highly skilled and innovative Python / AI Engine...Show more

Last updated: 10 days ago • Promoted

Platform Engineer – Python / Databricks / Notebooks / Kubernetes

Syntasa • Kolhapur, IN

Platform Engineer – Python / Databricks / Notebooks / Kubernetes.Syntasa is seeking a high-caliber and dedicated.Syntasa Technologies India Private Limited. This position offers an exciting opportuni...Show more

Last updated: 23 days ago • Promoted

Applied AI Engineer

Strategic Talent Partner • Kolhapur, IN

Design and deploy advanced multi-agent pipelines for credit analysis.Optimize inference and prompt chains using frameworks like DSPy, GEPA, and LangChain. Implement reasoning techniques (CoT, ToT, G...Show more

Last updated: 13 days ago • Promoted

Integration Engineer | 3–6 YoE | Join a Fast-Growing HealthTech Startup in Pune

Geektrust • Kolhapur, IN

Are you passionate about building systems that make a.HL7, FHIR, and DICOM standards.Epic, Cerner, NextGen, eClinicalWorks. Implementing and configuring integration engines like.Collaborating with c...Show more

Last updated: 13 days ago • Promoted

Oracle ETL (ODI / OIC) Systems Analyst

Digile • sangli, maharashtra, in

We’re Hiring : Oracle ETL (ODI / OIC) Systems Analyst – Lead Role (8+ Years).This is a high-ownership role covering the full delivery lifecycle — from assessment and design to build, testing, deployme...Show more

Last updated: 11 hours ago • Promoted • New!

LLM Scientist

Insight Global • Kolhapur, IN

Familiarity with LangSmith, LangFuse, or similar LLM monitoring tools.Show more

Last updated: 16 hours ago • Promoted • New!

Engineer : Senior LLM Optimization (LLMO) / GEO Expert – Google Vertex

Proso.ai • kolhapur, maharashtra, in

Generative Engine Optimization (GEO).You’ll help make sa products and services.AI-driven assistants and LLMs like ChatGPT, Copilot, and Gemini. Optimize LLM pipelines on Google Vertex AI : .Design RAG...Show more

Last updated: 11 days ago • Promoted