A small model makes text generation possible on a phone. A harness turns that capability into a feature that can work with real data, time limits, and checkable answers.
Running a language model on Android is no longer just a technical demo. Models such as Gemma 3 1B Instruction Tuned, quantized for mobile devices, make it possible to build chats that generate text on the phone.
The harder question begins after the first response arrives: how much of what the model wrote may the application treat as true?
A 1B model can produce a convincing explanation from a short context. It can also miss an important condition, connect unrelated facts, or fill a gap with an assumption. These failures are not exclusive to small models. But when we choose a model to fit a phone’s constraints, we have less room to ask it to select data, calculate, interpret exceptions, and write correctly in one operation.
The official Gemma 3 model card reports differences between its instruction-tuned 1B and larger variants on grounding evaluations. Those benchmarks do not predict the results of any particular app. They reinforce an important distinction: the ability to produce natural language is not the ability to verify it.
That is where a harness becomes useful.
Three problems a prompt cannot solve on its own
1. Hallucinations can use correct data
An invented amount is relatively easy to catch. A sentence using two real amounts to assert a false trend is harder.
A model can assign a value to the wrong period, compare records with different coverage, or infer a cause absent from the source data. A validator that checks only for unknown numbers will miss these relational errors.
A prompt can ask the model to be careful. It cannot independently verify each claim against the original source. That responsibility belongs to the application.
2. More context is not necessarily better context
A tempting solution is to send every available record. On a phone, this increases preparation and inference work. It also makes the model decide which records matter, which are incomplete, and how they may be combined.
Useful context should be selected and typed: period, units, origin, coverage limits, and facts relevant to the question. A user-written description or merchant name may be data, but it is not an instruction for the model.
The goal is to reduce the task before generation, rather than expect the model to discover every constraint hidden in a long prompt.
3. Local inference has a real lifecycle
On Android, a model call consumes memory and may cross into native code. Initialization takes time; a response may be slow; a timeout may expire while a JNI operation is still running.
If an app interprets “I stopped waiting” as “the model stopped,” it may launch another inference over an occupied runtime. No change to the prompt fixes that failure.
What the harness does
Here, harness means the runtime orchestration and control infrastructure surrounding the model inside the app. It is more than a test suite, and it does not require another LLM acting as a judge.
A simplified path is:
Question → route → typed facts ───────────────→ direct result
↓ when language helps
model → candidate
↓
validation → answer or fallback
Each stage has a purpose:
- Routing decides whether a question needs a calculation, clarification, or generation.
- Typed facts come from sources the application can verify and retain their scope.
- Bounded generation asks the model to explain or suggest something about selected facts.
- Validation treats output as a candidate and checks for claims beyond the evidence.
- Fallback keeps verified facts available when generation fails or is rejected.
- Runtime control manages initialization, busy operations, timeouts, and native failures.
The harness does not make an LLM deterministic. It makes explicit and testable the decisions the surrounding software can control.
One implementation example
In the chat of WalletMind, an Android personal finance app, query data comes from Room and is prepared in Kotlin before invoking Gemma 3 1B INT4 through LiteRT-LM. An overview can be displayed directly as an app-computed data card, without generation.
For an advice request, Kotlin selects a finding supported by the records and prepares a verifiable observation. Gemma receives a narrower task: write a suggestion related to that finding. Amounts on the card do not come from the model’s prose.
Imagine, using fictional data, records for one closed month and another still in progress. If the model writes, “spending decreased in the second month,” it may have turned an incomplete comparison into a trend. The harness does not need Gemma to audit its own sentence: deterministic checks run before the candidate is shown.
In the current advice path, those checks use typed facts, text normalization, and rules about numbers, categories, and certain causal or trend claims. There is no second LLM judge. That avoids another mobile inference, but it has an honest limitation: rules cannot understand every possible way to phrase a wrong conclusion.
Generated output therefore remains a candidate. If it is rejected, verified data stays available and the app returns a fallback. The text is not published fragment by fragment before validation completes.
Testing the system around the model
A harness can replace inference with controlled output in tests. Feed it an invented amount, a category outside the evidence, or an unsupported trend, then verify what the application does.
Tests can also cover behavior unrelated to language: whether facts reach the UI before an explanation and whether a new request avoids overlapping with native work that remains active after a timeout.
These tests make known failures reproducible. They cannot prove that every future model sentence will be correct. Device review must inspect what the user finally sees, since a sentence can satisfy a narrow rule and still mislead.
Conclusion
A 1B model is well suited to turning carefully selected context into useful language. It is less suited to acting as a database, calculator, analyst, writer, and auditor of its own work at the same time.
The harness separates those jobs. The application owns facts and scope; the model writes within a bounded task; validation determines whether the text may be shown; and the runtime is managed according to Android’s actual constraints.
The quality of an on-device AI feature depends as much on what its model can do as on what the system avoids asking it to do blindly.
Why a 1B On-Device LLM Needs a Harness was originally published in ProAndroidDev on Medium, where people are continuing the conversation by highlighting and responding to this story.
