What A Useful Tiny Model Eval Report Should Include
A practical guide to the review notes, test results, limits, and owner decisions that make a small local model safe to use after delivery.
Start with a report a real owner can read
A tiny model eval report should not feel like a research paper that only an engineer understands. It should help an owner decide whether the model is ready for real work, whether it still needs human review, and what the next improvement should be. The model may be small, private, and affordable, but the decision to trust it still needs evidence.
For Tiny Model Generator, the report is part of the product. The model artifact matters, the prompt or runtime files matter, and the dataset matters. Yet the report is what turns those pieces into an operating decision. It explains what was tested, what passed, what failed, what was left outside the first version, and how a person should use the model without pretending it is magic.
A good report should be short enough to read in one sitting and detailed enough to answer the first hard questions. Can this model handle normal inputs. What happens when the input is incomplete. Does it return the expected format. Where does it need a human. What should the owner watch during the first week.
Describe the job in plain language
The first section should restate the one job the model was built to do. This sounds simple, but it prevents confusion. A tiny model is not supposed to answer every question, replace every workflow, or act like a general assistant. It is supposed to repeat one focused behavior.
For example, the report might say that the model reads a customer support message and returns a department, urgency level, and short summary. Another report might say that the model reads an intake note and returns missing fields, a readiness score, and a suggested next step. Another might say that the model rewrites an internal note into a standard customer reply while preserving the facts.
This section should also say what the model is not for. If it should not give legal advice, diagnose a medical condition, invent prices, approve refunds, or make final hiring decisions, the report should say so clearly. Boundaries make the model easier to use safely.
Show the test set shape
An eval report does not need to expose private data, but it should explain the shape of the test set. The owner should know whether the test examples looked like real work or only clean demos.
Useful notes include the number of examples tested, the main input types, the expected output format, and the range of difficulty. The report can say that the test set included short messages, long messages, incomplete forms, repeated details, unclear wording, and edge cases where the right answer was to ask for more information. If sensitive records were replaced with safe placeholders, say that too.
This helps the owner understand the meaning of the score. Ten perfect demo examples are not the same as fifty realistic examples with messy language. A tiny model should be judged against the kind of work it will actually see.
Score the behavior that matters
The report should score the model against the business behavior, not against vague excitement. If the job is classification, measure correct labels. If the job is extraction, measure whether required fields were present and accurate. If the job is structured output, measure valid format. If the job is rewriting, measure whether facts were preserved and tone matched the standard.
It is better to use a small set of clear checks than a large set of confusing metrics. A practical report might track format validity, correct routing, missing information detection, unsafe answer avoidance, and reviewer acceptance. Each check should have a plain explanation.
The best reports separate harmless mistakes from costly mistakes. A slightly awkward summary may be acceptable. Sending a billing complaint to the wrong team may not be. Missing a required field may be recoverable if the workflow asks a human to review it. Inventing a fact should be treated much more seriously.
Include examples of wins and misses
Numbers are useful, but examples make the report real. Include a few anonymized cases where the model performed well and a few where it struggled. The goal is not to embarrass the model. The goal is to show the owner what success and failure look like.
A good win example might show a messy message that the model routed correctly, summarized cleanly, and returned in valid JSON. A good miss example might show an unclear input where the model guessed instead of asking for more detail. Another might show a case where the output was valid but the urgency score was too low.
These examples help the owner train their own judgment. They also help the next improvement cycle. If most misses come from short inputs, the next dataset needs more short inputs. If most misses come from mixed language messages, the next dataset needs that pattern. If most misses come from unusual policy exceptions, the model may need clearer rules or a human escalation path.
State the safe use recommendation
Every eval report should end with an operating recommendation. This is where the report becomes useful.
A model may be ready for full automation on low risk work. It may be ready only as a draft assistant. It may be ready for internal triage but not customer facing replies. It may need more examples before real use. The report should choose one of those positions and explain why.
For many first versions, the best answer is supervised use. The model can prepare a suggested output, but a person reviews it before action is taken. This still saves time because the human starts from a structured draft instead of a blank page. It also lets the owner collect corrections for the next version.
The recommendation should name the review points. For example, review all high urgency cases, review any output with missing fields, review any customer facing message, or review ten percent of routine cases during the first month. These rules make adoption calmer.
List deployment checks
A tiny model is only useful if it runs where the work happens. The report should confirm the practical deployment details. Can it run on CPU. Was the local runtime tested. Is the expected file format included. Does the owner need Ollama, llama.cpp, a small server wrapper, or a simple command line script. What input shape should the application send.
This section should also include response time notes and memory expectations when available. Owners do not need every benchmark, but they do need to know whether the model feels instant for one user, reasonable for a small team, or better suited for batch processing.
If the model was delivered with a sample script, API wrapper, or schema file, list it. If the model requires a specific prompt template or output parser, include that too. A model without operating notes can become shelfware. A model with clear checks can become a working tool.
Turn the report into the next training plan
The final section should describe the next improvement. Tiny models improve fastest when the owner collects real corrections. The report can recommend exactly what to save during the first week. Save inputs where the model guessed. Save outputs that required edits. Save cases where the model asked for review and the human agreed. Save cases where the model looked confident but was wrong.
This creates a practical feedback loop. The first model handles the stable center of the workflow. The report shows the edges. The owner gathers better examples from those edges. The next version becomes more reliable.
A useful eval report is not a decoration. It is the bridge between a model file and a safe business process. It tells the owner what the tiny model does well, where it should slow down, and how to make the next version stronger.