Role Overview:
We are hiring for one of our clients, seeking a Senior Software Engineer – Ruby (LLM Evaluation & Repository Validation) to work on a Full-Time basis. The projects focus on building LLM evaluation and training datasets that address realistic software engineering problems. One approach involves creating verifiable software engineering tasks from public repository histories using a synthetic method with human‑in‑the‑loop, while expanding coverage across programming languages and difficulty levels.
Key Responsibilities:
• Analyze and triage GitHub issues across trending open‑source libraries to identify suitable evaluation scenarios.
• Set up and configure code repositories, including Docker containerization and environment preparation for reproducible testing.
• Evaluate unit test coverage and quality metrics to assess baseline code health and test suite robustness.
• Modify and run Ruby codebases locally to measure LLM performance in bug‑fixing and code‑completion tasks.
• Collaborate with AI researchers to design and select repositories and issues that present challenging cases for LLMs.
Required Skills & Qualifications:
• Minimum three years of professional software engineering experience with strong proficiency in Ruby and Ruby on Rails.
• Demonstrated expertise with Git, Docker, and basic software pipeline setup for reproducible testing.
• Ability to navigate and understand complex, real‑world codebases, and to run, modify, and test them locally.
• Experience with Ruby testing frameworks (RSpec, Minitest), gem management (Bundler), and code quality tooling.
• Prior contribution to or evaluation of open‑source projects is considered a plus.
More About the Opportunity:
This role provides a unique opportunity to work with a global leader in the artificial intelligence industry, contributing to how large language models interact with real code. The work directly influences the development of AI‑assisted software development tools and benchmarks.
Equal Opportunity Employer:
We hire based on skills and expertise. All qualified candidates are welcome regardless of background, experience, or prior employment history. Applications are reviewed solely on demonstrated technical ability and qualifications.
Apply Now!
Software Engineer – Ruby (Remote) • dindigul, India
Administer and configure Xurrent ITSM platform; manage roles, workflows, SLAs, incident/problem/change modules; optimize automation, dashboards, and service catalog; troubleshoot issues, support in... Show more
Aelum Consulting is a ServiceNow Partner helping enterprises accelerate digital transformation through workflow automation, enterprise service management, AI-driven solutions, and consulting servic... Show more
Minimum 3 years Revit experience !!.Only experienced Revit architects.This post is not for engineers.Make sure you meet the requirments before you apply.An amazing work from home opportunity for ex... Show more
The Account Manager (AM) will be the primary owner of commercial health, revenue retention, and account expansion at TestGrid.This role bridges the gap between customer business goals and company r... Show more
We are looking for a Video Editor with 3–5 years of hands-on experience in creating high-quality content for digital platforms.You will work closely with the marketing and design teams to deliver p... Show more
MAX Security Solutions is a global risk management and intelligence company providing monitoring, analysis, and operational support services to clients worldwide across the corporate, travel, and s... Show more
About Aelum Consulting Aelum Consulting is a ServiceNow Partner helping enterprises accelerate digital transformation through workflow automation, enterprise service management, AI-driven solution... Show more
Statusbrew is a social media management platform used by teams and agencies worldwide to schedule content, manage engagement across 20+ channels, run analytics, and moderate conversations, all from... Show more
We’re hiring a full-time Cost Accountant to join Babco Foods.This is not a routine accounting role.We are looking for someone who can deeply understand pricing, margins, profitability, promotions, ... Show more
Minimum 3 years Revit experience !!.Only experienced Revit architects.This post is not for engineers.Make sure you meet the requirments before you apply.An amazing work from home opportunity for ex... Show more
Accredian is a fast growing edtech company that was started in 2018 by ISB MBAs.The company has so far served 18,000+ professionals working in almost 3000 companies.Accredian arms working professio... Show more
The Profit Pixel is a performance marketing agency led by Nishant Agrawal, a founder with 12+ years of experience in paid acquisition.We help D2C and other growth-focused businesses improve adverti... Show more
Job Title: Customer Success Manager Role Summary : The CSM will play a critical role in ensuring customer satisfaction, product adoption, and long-term success.This role requires a strong blend o... Show more
First Choice Business Debt Solutions is dedicated to helping small businesses, particularly Construction, Trade, and Trucking companies, overcome financial hardships caused by Merchant Cash Advance... Show more
