AI Coding Agent Model Evaluator (DevOps, English)

The work

You will evaluate three AI coding agents that received the same real software engineering request. Review each complete run, including commands, changed files, checks, and the final response.

Score each run from 1 to 5, rank the three results, and explain your ranking with evidence. Your review covers instruction-following, functionality, efficiency, safety, honesty, communication, and other quality areas.

• Review commands, changed files, checks, and final messages from AI coding agents.
• Compare implementation quality, correctness, safety, and practical usefulness.
• Give each run an overall score from 1 to 5 and rank all three results.
• Support your ranking with clear evidence from the agent runs.

What it pays and takes

This is a one-month remote contractor assignment for people who can run, modify, and test real software projects locally. The work requires careful technical judgment when comparing AI-generated implementations.

• Pay: $25 per approved task.
• Location: Open to candidates in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, and Laos.
• Schedule: At least 40 hours per week, including four hours of overlap with Pacific Time.
• Language: Professional fluency in English.
• Experience: At least three years of overall professional experience.
• Programming: Strong experience with Python, JavaScript, TypeScript, Java, Go, PHP, or a similar language.
• Tools: Proficiency with shell commands, Git, Docker, and basic software pipeline setup.
• Work skills: Ability to run, modify, and test real-world projects locally.

How it works

Apply on OpenTrain with your resume and then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including reviewing code and rating AI responses. People with software engineering and DevOps experience help assess whether AI-generated work is correct, safe, efficient, and useful.