45 lines
2.6 KiB
JSON
45 lines
2.6 KiB
JSON
{
|
|
"scope": "Post-run observations on the frozen run, not regrading or replacement scores. Frozen runtime and original verdicts remain unchanged.",
|
|
"observations": [
|
|
{
|
|
"cases": [
|
|
"C001/rag",
|
|
"C001/graphrag"
|
|
],
|
|
"type": "confirmed_false_refusal",
|
|
"observation": "The explicit work/type/year question was rejected at scope/wrong_work_or_meeting. Read-only retrieval diagnosis found the correct thesis, year and heading in raw graph facts, while graph_context kept type/year but discarded the heading required by the quoted-label scope check. The binding needs to preserve trusted work identity; do not waive the scope guard."
|
|
},
|
|
{
|
|
"cases": [
|
|
"C011/rag",
|
|
"C011/graphrag"
|
|
],
|
|
"type": "confirmed_requirement_failure",
|
|
"observation": "The question requests goals, not supposed results. The delivered answer says goals are missing and supplies results. Raw Markdown explicitly contains Predbežné zadanie - úlohy na semester and three assignment bullets. Required repairs concern source section recognition, exclusion/negation in the plan, and availability checks beyond selected evidence."
|
|
},
|
|
{
|
|
"cases": [
|
|
"C004/graphrag"
|
|
],
|
|
"type": "unsupported_judge_verdict",
|
|
"observation": "The unsupported_time proof compares answer Podľa zápisu zo stretnutia 2024-12-12 against source Stretnutie 12.12.2024. These are the same explicit record date. This proof does not establish an invented date. Original score is preserved."
|
|
},
|
|
{
|
|
"cases": [
|
|
"C010/rag",
|
|
"C010/graphrag"
|
|
],
|
|
"type": "unsupported_judge_proof",
|
|
"observation": "The unsupported_time proof quotes Bakalárska práca 2024 from both answer and source. An identical source work-year label does not itself demonstrate an unsupported time claim. This is not a full manual regrade of the answers."
|
|
},
|
|
{
|
|
"cases": [
|
|
"C003/rag"
|
|
],
|
|
"type": "disputed_judge_proof",
|
|
"observation": "The wrong_phase proof compares a stated Ciel with the same goal rendered under Zadanie a ciele as zadanie. That quoted pair alone does not show a plan upgraded into an achieved result. The answer also contains unrequested results, which still needs separate fulfilment review."
|
|
}
|
|
],
|
|
"conclusion": "Do not infer true general accuracy, zero critical errors or readiness from this automatic judge. Calibration on eight synthetic scenarios did not establish reliability for this diverse TEST. Report raw conservative pass counts and false refusals, retain uncertain grades, and require further source-based review."
|
|
}
|