compact-token-estimate-is-word-count
OUT premise
`estimate_tokens` counts whitespace-separated words, not BPE tokens; the budget parameter throughout the compact module is measured in this unit.
Summary
The token counting used during compaction is just splitting on whitespace and counting words, not doing real BPE tokenization. This means every "token budget" in the compact module is actually a word budget, so the actual token usage will differ from the estimate since BPE tokens and whitespace-delimited words don't map one-to-one.
Dependents
These beliefs depend on this one:
- compact-budget-controls-output-size — The compact module's token budget reliably constrains total output size
Details
| Source | entries/2026/04/23/reasons_lib-compact.md |