SWE Bench Engineer (C#) Code Evaluation

The work

You will evaluate open-source software repositories to help measure how well large language models handle real bug-fixing tasks. The work combines software engineering, test review, environment setup, and hands-on code evaluation.

• Analyze and triage issues across open-source libraries.
• Configure repositories and development environments, including Docker-based setups.
• Review unit-test coverage and assess overall test quality.
• Run and modify real C# or other codebases locally.
• Evaluate how well large language models solve software bugs.
• Work with researchers to identify repositories and issues that are difficult for AI models.

What it pays and takes

This is a remote contractor assignment for professionals located in Latin America. The contract lasts four weeks and requires at least 20 hours per week, including four hours of overlap with Pacific Time.

• Pay: $25 per label.
• Location: Open to candidates in listed Latin American countries.
• Schedule: At least 20 hours per week.
• Contract length: Four weeks.
• Time overlap: Four hours with Pacific Time.
• Experience: At least three years of software engineering experience.
• Technical skills: Strong C# or C++ experience, plus Git, Docker, and basic software pipeline setup.
• Codebase skills: Ability to navigate, run, modify, and test complex real-world projects locally.
• Language: Professional fluency in English.
• Helpful background: Open-source contribution or evaluation experience.
• Additional advantages: LLM research or evaluation experience, or experience with developer tools and automation agents.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training work uses human judgment to improve how artificial intelligence systems perform. In this role, your software engineering experience helps assess whether AI models can understand, modify, and test real code.