Reading model evaluations

KRXY compares models by the answers they give, not by parameter count, speed or the fact that they run. Part 5 of the journal describes the first stored evaluations.

A run sends the same cases to each selected deployment, stores the returned text, and records pass, fail or review for each answer. The output limit and generation settings are saved with the run. When scoring rules are corrected, the stored answers are not overwritten.

Several terms are kept separate:

• Catalogue entry: a model the platform knows by name.
• Installed artifact: a specific file on a node, such as a compiled HEF or a GGUF build.
• Available, or ready: a runtime that can answer now.
• Runtime compatibility: a node can run the model a request names.
• Inference performance: how long the work takes on that node.
• Answer quality: whether a reply is correct and complete for the task.
• Research evaluation: a recorded comparison under rules fixed before the answers are read.
• Recommended default: a decision taken from that evaluation, never assumed from size.

A job that completes is not necessarily a complete answer. Part 5 records a reply that stopped at a 256-token output limit.

The evaluations used synthetic tasks on one Raspberry Pi and automated verdicts. They are not a general ranking of models, and they are not evidence about distributed learning. See Part 5 on Hashnode.