Healthcare AI Rubric Evaluation Specialist

The work

You will help train AI by creating expert healthcare content and judging the quality of clinical responses. Your work will focus on accuracy, completeness, safety, reasoning, uncertainty, and professional relevance.

• Write challenging healthcare questions that reveal limitations in advanced AI responses.
• Create scoring rubrics covering essential concepts, reasoning steps, caveats, acceptable alternatives, and critical errors.
• Evaluate AI-generated clinical content for factual accuracy, completeness, safety, reasoning quality, calibration, and professional relevance.
• Distinguish minor omissions from problems that could materially affect clinical quality or safety.
• Review rubrics written by other healthcare professionals.
• Write expert reference answers, explanations, critiques, and gold-standard content for reinforcement learning and supervised fine-tuning work.

What it pays and takes

This is a remote, part-time contractor role for healthcare professionals in the United States. The work requires careful clinical judgment and strong professional writing in English.

• Pay: Up to $100 per hour.
• Time: 20 or more hours per week.
• Location: United States only.
• Language: Professional English proficiency at C1 level or higher.
• Education or credential: A professional degree, license, credential, or recognized qualification in medicine, dentistry, pharmacy, nursing, or another healthcare discipline.
• Experience: At least three years of professional or clinical experience in a clearly defined healthcare specialty or subspecialty.
• Skills: Ability to write difficult questions, build detailed rubrics, assess clinical reasoning and safety, identify important omissions, and recognize acceptable alternative approaches.
• Helpful background: Medical education, examination writing, competency assessment, clinical guidelines, teaching, peer review, quality assurance, AI evaluation, reinforcement learning from human feedback, supervised fine-tuning, data annotation, model benchmarking, or rubric-based scoring.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including writing questions, reviewing answers, and rating model responses. Healthcare specialists are needed because accurate, safe evaluation depends on professional knowledge and careful judgment.