deduplication

48 beliefs (48 IN, 0 OUT)

This topic covers two closely related concerns: how the reasons system groups semantically similar beliefs using embedding-based clustering, and how it uses those clusters to detect and eliminate duplicate beliefs while preserving the integrity of the justification network. These capabilities matter because a growing belief network inevitably accumulates redundant claims, and naive deduplication risks severing dependency chains that downstream beliefs rely on. The clustering infrastructure also serves double duty, powering both the derive command's budget-aware belief selection and the semantic contradiction detection pipeline.

The clustering subsystem is built on optional dependencies, sentence-transformers and scikit-learn, gated behind a clean import check that degrades gracefully when they are absent (cluster-deps-are-optional, cluster-deps-optional-with-graceful-skip). When available, the system embeds beliefs in sorted ID order for determinism (cluster-embed-order-is-deterministic), caches embeddings keyed by content hash so edits force re-embedding (cluster-cache-keys-include-content-hash, cluster-cache-no-recompute), and partitions beliefs into clusters where every belief lands in exactly one group (list-clusters-partitions-all-beliefs, cluster-stats-sizes-sum-to-input). Budget management is precise: the auto-k heuristic targets roughly five beliefs per cluster (cluster-auto-k-heuristic), remainder slots go to the largest clusters (cluster-remainder-favors-largest), and the output never exceeds the requested budget (cluster-beliefs-respects-budget, cluster-beliefs-returns-exact-budget). The entire pipeline is deterministic given the same seed (cluster-beliefs-deterministic-with-seed, cluster-selection-is-deterministic-and-budget-exact), and it shortcuts all ML work when beliefs fit within budget (cluster-skips-ml-when-under-budget). This same infrastructure is reused by semantic contradiction detection (semantic-contradiction-reuses-cluster-infrastructure), which clusters beliefs before sending topically related groups to the LLM (semantic-contradictions-cluster-before-llm), skipping singleton clusters where no contradiction is possible (semantic-contradiction-skips-singleton-clusters).

Deduplication itself operates on these clusters with careful attention to network topology. In auto mode, the survivor of each duplicate cluster is the node with the most dependents, with lexicographic tiebreaking (dedup-auto-keeps-most-dependents, dedup-keeps-most-connected-node, dedup-survivor-selection-is-topology-reliable). The dependent count reflects the complete dependency graph including both antecedent and outlist edges (dedup-reflects-complete-dependency-graph). When losers are retracted, all justification references across the network are rewritten to point at the survivor (dedup-rewrites-both-antecedents-and-outlist), preserving topology (dedup-is-topology-preserving-and-auditable). The system also supports human oversight through an editable plan format with KEEP/RETRACT markers (dedup-plan-is-user-editable). On the prevention side, the validate-proposals step uses Jaccard similarity on tokenized belief IDs to block re-derivation of retracted beliefs under variant names (validate-proposals-rejects-jaccard-similar-to-out, jaccard-tokenizer-splits-on-hyphens-and-colons).

A substantial number of beliefs in this topic are themselves flagged as duplicates or as being covered by other existing beliefs. Entries like import-is-idempotent, network-dependents-eagerly-maintained, premise-default-in, and propagation-is-immediate each note which prior belief already captures the same claim. Several others, such as make-nodes-excludes-active-premises, network-tests-are-database-free, and hash-file-import-unused, are marked as test infrastructure details or trivial implementation artifacts rather than meaningful architectural invariants. All of these remain IN rather than retracted, which suggests they were identified as duplicates during a review pass but have not yet been cleaned up through the deduplication pipeline they themselves describe.

IN cluster-and-sample-are-mutually-exclusive
The `--cluster` and `--sample` flags in `cmd_derive` are mutually exclusive, enforced with an explicit check and `sys.exit(1)` — they represent competing strategies for belief subset selection.
IN cluster-auto-k-heuristic
Auto cluster count is computed as `len(beliefs) // 5`, clamped between 2 and `min(budget // 3, 20)`, targeting approximately 5 beliefs per cluster with at least 3 beliefs per cluster given the budget.
IN cluster-beliefs-deterministic-with-seed
Given the same beliefs dict, budget, and seed, `cluster_beliefs` produces identical output across calls.
IN cluster-beliefs-respects-budget
`cluster_beliefs` never returns more IDs than the `budget` parameter; each cluster's allocation is individually capped by `min(alloc, len(members))`.
IN cluster-beliefs-returns-exact-budget
`cluster_beliefs` returns exactly `budget` belief IDs when the input set is larger than the budget, and all items when the input set is smaller.
IN cluster-cache-keys-include-content-hash
`ClusterCache` keys embeddings by `(node_id, sha256_prefix)`, so editing a belief's text with the same ID forces re-embedding rather than serving stale vectors.
IN cluster-cache-no-recompute
`ClusterCache.embed()` does not recompute embeddings for previously cached belief texts; cache size stays constant on repeated calls with the same input and grows by exactly the count of new texts on superset calls.
IN cluster-deps-are-optional
`sentence-transformers` and `scikit-learn` are optional dependencies behind a `HAS_CLUSTER_DEPS` gate; the module degrades to a clear `ImportError` with install instructions when they are absent.
IN cluster-deps-optional-with-graceful-skip
The clustering module (`reasons_lib.cluster`) is behind an optional `[cluster]` install extra; when `sentence-transformers` or `scikit-learn` are missing, `_require_cluster_deps` raises `ImportError` and all dependent tests skip cleanly.
IN cluster-derive-is-semantically-informed-and-deterministic
When using cluster-based belief selection, the derive pipeline achieves semantically-informed budget allocation (embedding-based grouping ensures topical diversity across the prompt) with end-to-end determinism (sorted embedding order, fixed-seed clustering, and exact budget counts feed into reproducible prompt construction with accurate token allocation).
IN cluster-embed-order-is-deterministic
Beliefs are sorted by ID before embedding (`ids = sorted(beliefs.keys())`), making cluster assignments reproducible given the same random seed.
IN cluster-remainder-favors-largest
When the budget doesn't divide evenly across clusters, extra slots are distributed one-per-cluster to the largest clusters first via descending size sort.
IN cluster-selection-is-deterministic-and-budget-exact
Cluster-based belief selection produces identical results given the same seed, returns exactly the requested budget count, and processes beliefs in sorted order — ensuring fully reproducible, precisely-sized belief subsets for derive prompt construction.
IN cluster-skips-ml-when-under-budget
When the number of beliefs is less than or equal to the budget, all ML work (embedding, clustering, sampling) is skipped and every belief is returned directly.
IN cluster-stats-sizes-sum-to-input
The `cluster_sizes` list in the stats dict returned by `cluster_beliefs` always sums to the total number of input beliefs, enforcing that every belief is assigned to exactly one cluster.
IN dedup-auto-keeps-most-dependents
In auto mode, `deduplicate` retains the cluster member with the most dependents and retracts all others
IN dedup-is-topology-preserving-and-auditable
Deduplication preserves network topology (rewrites both antecedent and outlist references to survivors), selects structurally-optimal survivors (most dependents with lexicographic tiebreak), and supports human oversight (KEEP/RETRACT markers in a user-editable plan format).
IN dedup-keeps-most-connected-node
In auto-dedup mode, the node with the most dependents survives each cluster; ties break by lexicographic ID, and losers are retracted after dependents are rewired.
IN dedup-plan-is-user-editable
The dedup plan format uses KEEP/RETRACT markers that users can swap before applying, making deduplication decisions reviewable and overridable
IN dedup-reflects-complete-dependency-graph
Deduplication survivor selection accurately reflects the complete dependency graph — the node with the most dependents survives each cluster, and that dependent count includes both antecedent-based and outlist-based dependency edges, ensuring the structurally most-connected node is always preserved.
IN dedup-rewrites-both-antecedents-and-outlist
When a duplicate is retracted via dedup, all justification references (both antecedent and outlist) across the network are rewritten to point at the kept node
IN dedup-survivor-selection-is-topology-reliable
Deduplication reliably selects the structurally-optimal survivor in each duplicate cluster by choosing the node with the most dependents, and this selection is correct because the dependents index accurately reflects the justification graph.
IN hash-file-import-unused
Trivial/unstable detail about an unused import; not a meaningful architectural invariant.
IN idempotent-reimport-skips-all
Covered by existing `import-skips-existing-sync-is-remote-wins` which captures the idempotent import behavior
IN import-agent-outlist-not-antecedent
Duplicates existing belief `kill-switch-uses-outlist-not-antecedent`.
IN import-beliefs-path-import-unused
Vestigial import detail — too trivial and unstable to track as a belief; it could be cleaned up at any time.
IN import-is-idempotent
Covered by existing `import-skips-existing-sync-is-remote-wins`
IN import-skips-existing-nodes
Duplicates existing belief `import-skips-existing-sync-is-remote-wins`.
IN import-wires-reverse-index
Covered by existing `dependents-is-manual-reverse-index` which captures that the reverse index is manually maintained
IN jaccard-tokenizer-splits-on-hyphens-and-colons
`_tokenize_id` splits belief IDs on hyphens and colons into token sets for Jaccard similarity comparison, so `foo-bar-baz` and `foo-bar-qux` share 2/4 tokens (0.5 similarity)
IN justification-order-preserved-by-rowid
Already exists as `justification-order-preserved-via-rowid`
IN list-clusters-partitions-all-beliefs
`list_clusters` assigns every input belief to exactly one cluster with no drops or duplicates — the union of all cluster members equals the input set.
IN make-nodes-excludes-active-premises
Test helper implementation detail, not a claim about production code behavior
IN make-nodes-omits-active-premises
Test helper implementation detail, not a production code invariant
IN network-any-mode-justification
Duplicates existing belief `node-in-if-any-justification-valid`.
IN network-dependents-eagerly-maintained
Duplicates existing beliefs `dependents-bidirectional-index` and `dependents-is-manual-reverse-index`.
IN network-missing-outlist-passes
Duplicates existing belief `missing-outlist-nodes-pass-validation`.
IN network-retracted-skips-propagation
Duplicates existing belief `retracted-nodes-skipped-in-propagation`.
IN network-tests-are-database-free
Test infrastructure detail, not a production code claim
IN out-beliefs-imported-as-bare-premises
Covered by existing `out-beliefs-imported-without-justifications` which captures the same invariant
IN out-nodes-excluded-from-staleness
Duplicate of existing `check-stale-skips-out-nodes`.
IN parse-beliefs-returns-dicts
Too granular — the dict-return pattern is partially covered by `api-functions-return-dicts`, and the specific field list is an implementation detail likely to evolve
IN premise-default-in
Covered by existing `premise-behavior-emerges-from-absence` which captures this as an emergent property
IN propagation-is-immediate
Covered by existing `mutations-are-atomic-and-safely-propagated` and `propagation-is-safe-and-terminating`
IN semantic-contradiction-reuses-cluster-infrastructure
Semantic contradiction detection delegates embedding and clustering to the existing `list_clusters()` from `reasons_lib/cluster.py` — the same infrastructure that serves deduplication also serves contradiction detection, with no duplicate embedding/clustering implementation.
IN semantic-contradiction-skips-singleton-clusters
Single-belief clusters are skipped during semantic contradiction detection (no contradiction possible within one belief), and clusters exceeding `CONTRADICTION_BATCH_SIZE` are sub-batched within the cluster boundary.
IN semantic-contradictions-cluster-before-llm
The `--semantic` flag on `detect-contradictions` embeds beliefs via sentence-transformers, clusters with KMeans via `list_clusters()`, and sends each cluster to the LLM as a batch — ensuring topically related beliefs are analyzed together instead of being scattered across random batches of 50.
IN validate-proposals-rejects-jaccard-similar-to-out
`validate_proposals` skips any proposal whose tokenized ID has >= 0.5 Jaccard similarity to an existing OUT belief, returning "similar to retracted" in the skip reason — preventing re-derivation of retracted beliefs under variant names