Putting a language model on a device sounds simple until you look at the memory budget. This post covers what we have found to be realistic when the goal is local explanation rather than cloud round trips.
Start with why you want it local
There are three honest reasons to run a model on the device: the site has poor connectivity, the data should not leave the building, or the response needs to be available when the network is not. If none of those apply, a cloud call is cheaper and better. Local inference is a design constraint you accept, not a feature you add for its own sake.
What fits, and on what
A microcontroller class part cannot host a language model. It has a few hundred kilobytes of RAM, and even an aggressively quantized small model needs orders of magnitude more. What an MCU can do very well is signal processing and small anomaly models, which is where it belongs.
For language you need an applications processor with an NPU. Models in the one to three billion parameter range, quantized to four bits, fit into the memory envelope of that class of hardware and produce usable token rates for short outputs. If your device needs both, you are designing two compute domains into one enclosure, and you should plan for that from the schematic stage rather than discovering it later.
Keep the output short
Latency is dominated by tokens generated. A three-sentence diagnosis is a very different workload from a page of prose. Constrain the output format, give the model structured input rather than raw data, and it will be both faster and more accurate.
Thermal and power reality
Inference is a power spike. In a sealed industrial enclosure that spike becomes heat with nowhere to go. Size your thermal design around the inference duty cycle you actually expect, not the idle draw, and consider whether explanation needs to run continuously or only when a finding is raised. In our design it runs only on findings, which keeps the average draw low.
The boundary rule
Whatever you deploy, keep the model out of the decision path. Let deterministic code decide what happened and let the model describe it. That way a bad generation is a wording problem, not a safety problem.