Building A Rejection Set For A Tiny Model
A practical guide to collecting examples of what a focused local model should refuse, flag, or send back for human review before launch.
Why rejection examples belong in the first dataset
A tiny model is usually built because one business task repeats often enough to deserve a focused tool. The task may be routing customer messages, scoring intake forms, extracting fields from notes, rewriting private drafts, or returning a clean JSON answer for another system. Most teams collect examples of what the model should do. Fewer teams collect examples of what the model should not do.
That missing set is where many launch problems begin. A small local model can look ready when every test input is clear, complete, and safely inside the task. Real work is not that tidy. A customer writes a message that belongs to another department. A lead form asks for something the business does not offer. A document contains a private detail that should not be repeated. A note includes two conflicting dates. A staff member pastes a long message and expects the model to understand everything.
A rejection set is the group of examples that teach the model and the surrounding workflow when to pause. It shows cases where the safest answer is not a confident category or polished output. The safest answer may be needs human review, missing information, outside scope, unsafe to answer, or no action recommended. For a tiny model, those boundaries are not a weakness. They are what make the narrow lane trustworthy.
Start with the ways the task can fail
Begin by writing the one job statement for the model. The model reads one type of input and returns one type of output. Then ask how that job can fail in normal use. Do not start with dramatic edge cases. Start with ordinary confusion.
For a support routing model, the input might look like billing but actually describe a delivery issue. For a lead scoring model, the person may provide a budget but no timeline. For a document extraction model, the document may include two possible names for the same field. For a private rewriting model, the draft may include a promise the business should not make. For a JSON output model, the source text may not contain enough facts to fill every field.
Each failure type should become a short example. Keep the original input shape realistic. Then write the desired response. The response may still be structured. It can say category unknown, review required true, reason missing timeline, suggested next step ask for date. The point is to show the model that uncertainty has a valid format.
Include outside scope requests
A focused local model should know when the request is outside its lane. If the model is built for routing messages, it should not answer legal questions. If it is built for lead readiness, it should not invent pricing. If it is built for document extraction, it should not summarize unrelated policies. If it is built for private rewriting, it should not add facts that were not provided.
Outside scope examples are especially useful for owner trust. They prevent the first version from pretending to be a general assistant. A tiny model should be allowed to say that the input does not match the trained workflow. That answer protects the business from confident nonsense and protects the model from being judged against work it was never designed to do.
Write these examples in plain language. A non technical owner should be able to read the input and understand why the model should pause. If the boundary only makes sense to the builder, the boundary is not ready yet.
Teach missing information as a normal outcome
Many business records are incomplete. A person may forget a phone number, send only a first name, leave the budget blank, omit a document date, or describe a problem without the account detail needed to act. A tiny model should not be forced to guess just because the output schema has fields.
Add examples where the correct answer names the missing information. This is different from failure. The model can still be useful if it says what is missing and what to ask next. For a lead form, the output might say not ready yet because timeline is absent. For an intake note, it might say human review needed because the request includes symptoms, pricing, or personal data. For a routing model, it might say route to intake review because the message could fit two teams.
These examples help the model support human operators instead of replacing their judgment too early. They also make the first week after launch calmer because staff know what to do with uncertain outputs.
Keep private details out of the training file
A rejection set often comes from real messy work, and messy work may contain sensitive details. Before saving examples, replace names, phone numbers, account numbers, addresses, medical context, candidate notes, private prices, and other personal details with safe placeholders. Preserve the shape of the issue without exposing the person behind it.
For example, replace a real name with customer name. Replace a phone number with phone number. Replace an address with address. The model does not need the real secret to learn the boundary. It needs the pattern that tells it when to pause, ask for review, or avoid adding facts.
This habit also helps the owner share the dataset with a builder, reviewer, or future maintainer without turning the model project into a data handling problem.
Review the rejection set before launch
Before launch, read the normal examples and the rejection examples together. The set should feel balanced. If every example is easy, add more uncertainty. If every example rejects, add more normal cases. A tiny model needs to learn both confidence and restraint.
Ask the owner to approve the rejection labels. The owner should agree that these cases deserve a pause, a missing information response, or a human review flag. If the owner disagrees, fix the rule before training or prompt design. A model cannot follow boundaries that the business has not decided.
After the model is tested, include the rejection set in the eval report. Show how many pause cases passed, where the model still guessed, and what should stay behind a review gate. This gives the launch decision a practical base. The question is not whether the model sounds impressive. The question is whether it knows its own lane well enough to help safely.