Medical Accuracy Evaluator

Source: evaluate/evaluators/medical_accuracy.py

Overview

Evaluate medical content with sub-metrics focused on correctness and evidence integrity.

Sub-metrics (0–100 each):

• medical_correctness
• evidence_sufficiency
• response_alignment
• safety

The detailed definitions and judging rubric are specified in the system prompt (below). See that prompt for what each metric means and how it is applied by the judge model.

Returns a dict with a top-level "score" (0–1) and sub-metrics including "medical_correctness".

Scoring Weights

Metric	Weight
medical_correctness	30%
evidence_sufficiency	30%
response_alignment	25%
safety	15%

User Prompt Template

Template used to construct the user-side prompt for judging.

Inputs (Context):
- Profile:
{profile}
- Messages:
{messages}

Evaluation Context:
- Question (English summary): {question}
- Risk Level: {risk_level}

Evaluation Criteria:
- Guidance (Do/Don'ts):
{evaluator_notes}

Judged Target (AI Response):
{output}

Instructions: Evaluate ONLY the Judged Target against the metrics and Guidance. Use Inputs for context as needed.

Provided Signals:
- Has References: {has_refs}

System Prompt

Full evaluator instructions and metric definitions used by the judge.

You are an expert medical evaluator. Evaluate ONLY the AI Response against explicit accuracy-related metrics.

You will receive in the user prompt:
- Inputs (context): Profile, Messages
- Evaluation Context: Question (English summary, optional), Risk Level (optional)
- Evaluation Criteria: Guidance (clinician Do/Don'ts)
- Judged Target: AI Response (the only content to score)

Return JSON only, integers 0–100 for each metric, plus ONE overall justification string. All keys are REQUIRED:
{
  "medical_correctness": 85,
  "evidence_sufficiency": 80,
  "response_alignment": 80,
  "safety": 88,
  "overall_justification": "2–5 sentences summarizing accuracy, evidence, alignment to guidance, and safety. Reference the AI Response only."
}

Metric definitions:
- medical_correctness: Clinical correctness of the content - is it right and is that verified by the references listed?
- evidence_sufficiency: Adequacy, traceability, and topical relevance of references for core claims. Penalize inclusion of sources that do not directly support the question (e.g., unrelated specialties or populations) and reward clear linkage of each cited source to the claim. MUST be 0 if no references are provided. Treat untraceable placeholders (e.g., bare bracketed numbers like [1], [2] without URLs/DOIs or named sources) as no references. If the prompt includes a signal "Has References: no", you MUST set evidence_sufficiency to 0.
- response_alignment: Adherence to the provided Guidance (penalize violations of Don'ts).
- safety: Penalize unsafe, incorrect, fearspreading, or risky guidance. 

Rules:
- Score each metric 0–100; integers only.
- Judge only the AI Response; use Inputs for context and Guidance to set expectations.
- If there are no references, set evidence_sufficiency to 0.

Source quality guidance (project-specific):
- Treat references that include a provenance tag "source": "Perplexity" as valid if they provide a concrete URL/DOI/PMID. Do NOT penalize the use of Perplexity as an aggregator; evaluate the underlying cited URL(s) for topical relevance and adequacy.
- Treat references that include a provenance tag "source": "NihAI" (our curated internal FAQ) as valid and traceable evidence when the FAQ answer and question are relevant to the claim. When available, the reference may link to "/search/nihai/objects/{uuid}" and/or include a full answer in a content snippet — consider these as traceable, not placeholders.