Strict-JSON Agent

Overview

The Strict-JSON agent is a testing variant of the strict-referenced agent designed to compare the effects of prompt-based JSON formatting vs API-enforced JSON mode.

Purpose

This agent helps answer the question: Does enforcing JSON mode at the API level (rather than just prompting for JSON) produce more natural, less structured responses?

The hypothesis is that using OpenAI's response_format: {"type": "json_object"} parameter might prevent the LLM from falling into "ChatGPT mode" with overly structured responses containing headers like "Scoped question:", "Short answer:", etc.

Implementation Details

Same Prompt as Strict-Referenced

The agent uses the identical system prompt as strict-referenced: - Cannot use its own knowledge - everything must be confirmed through research tools - Every piece of information must be directly sourced from search results - Must include references for every claim - Search thoroughly before responding - Refuses to answer if insufficient reliable sources are found

Key Difference: JSON Mode Enforcement

Unlike strict-referenced which only prompts for JSON format in text, strict-json enforces it at the API level:

agent_kwargs["model_settings"] = {"response_format": {"type": "json_object"}}

This uses OpenAI's native JSON mode which guarantees valid JSON output and may influence the model's behavior.

Usage

API Endpoint

POST /agent/strict-json

Request Format

Same as other agents:

{
  "messages": [
    {"role": "user", "content": "Your medical question here"}
  ],
  "profile": "Name: ...\nLocation: ...\nLanguage: ...",
  "tools": ["search_nihai", "ask_nihai"],
  "model": "gpt-4o",
  "reasoning": true
}

Response Format

Standard AshaiResponse:

{
  "response": "...",
  "references": [...],
  "thinking": "...",
  "realtime_evaluation": {...}
}

Testing in UI

1. Go to /agents page
2. Select /agent/strict-json from dropdown
3. Use the default example or enter your own query
4. Compare responses with /agent/strict-referenced

Expected Behavior

Tool Requirements

• Requires at least one tool - will refuse to answer if tools: [] or tools: ["none"]
• Default tools: search_nihai, ask_nihai

Reference Requirements

• Must return references - will return error message if no references found after search
• References are validated and enforced

Research Indication

The thinking field will include: "Performed research using available tools (september, pubmed, googlesheet, perplexity) with JSON mode enforcement."

Comparison Testing

To test the hypothesis about JSON mode effects on response naturalness:

1. Run the same query on both agents:
2. /agent/strict-referenced (prompt-based JSON)

3. /agent/strict-json (API-enforced JSON)

4. Compare:

5. Does strict-json produce fewer structured headers?
6. Are responses more conversational?

7. Is the content quality similar?

8. Use in evaluations:

9. Add strict-json to evaluation batches
10. Compare scores across metrics
11. Analyze response patterns

Technical Notes

Code Location

• Agent: agents/strict_json/strict_json.py
• Route: agents/routes.py (line 452)
• UI Integration: templates/agents_page.html

Modified Components

1. agents/agent.py: Added use_json_mode parameter to run_generic_agent()
2. agents/strict_json/: New agent module
3. agents/routes.py: New /strict-json endpoint
4. templates/agents_page.html: UI integration for agent selection and configuration

Future Improvements

• [ ] A/B test with real users to gather qualitative feedback
• [ ] Automated evaluation comparing response naturalness
• [ ] Track whether JSON mode affects tool calling behavior
• [ ] Consider making JSON mode a configurable parameter for all agents

Related Agents

• strict-referenced: Same requirements, prompt-based JSON
• strict-referenced-after: Answer-first-verify approach
• ashai: General medical AI without strict referencing requirements