Disaster Recovery

Scope: ohshii_governance and dao_storage derivable-index recovery endpoints.

Audience: backend engineers + DAO operators. End-user-facing description lives in HowItWorksPage and TermsPage (governance section).

---

Why the feature exists

ohshii_governance and dao_storage rely on several secondary indices that are derived from a canonical source-of-truth map. Two examples:

• FOLLOWERS_BY_FOLLOWEE (memory id 72) is derived from FOLLOWS (memory id 71). Without it, listing the followers of a delegate would require iterating the entire FOLLOWS map — a hot-path cost incompatible with a public query.
• ANY_METHOD_VERIFIED_USERS (memory id 70) is the union of VERIFIED_USERS (World ID) and DECIDEID_VERIFIED_USERS (DecideID). It lets the universal vote gate check verification status in O(log N) without consulting two maps.

Secondary indices are written in lockstep with their source-of-truth on every mutator. The lockstep can drift if:

1. A canister upgrade ships a schema change that leaves the secondary in a half-written state.
2. A Storable::from_bytes decode panic surfaces a placeholder row that pollutes the secondary.
3. An operational mistake (e.g. a manual stable-memory write outside the normal lockstep) corrupts the secondary.

Because the secondary is derivable, it can always be rebuilt from the source-of-truth. The challenge is doing it safely: clearing the index and repopulating from scratch must NOT corrupt the data and must NOT brick the canister if the rebuild itself traps.

The Guardian is the role authorised to trigger this recovery. Admin and the DAO are deliberately excluded: the rebuild endpoint must remain reachable even when the proposal pipeline itself is corrupted.

---

Storage-side recovery — admin_rebuild_indexes

Source: src/storage/src/lib.rs.

Endpoint: #[update] async fn admin_rebuild_indexes(which: RebuildableIndex) -> Result<String, String>.

Auth (in order):

1. Reject anonymous.
2. Sync fast path: is_admin() or is_ohshii_governance_caller(&caller) (admin principal allowlist + governance self-call).
3. Async fallback: check_is_guardian_via_governance(&caller).await — ICC to ohshii_governance.is_guardian with bounded_wait(...).change_timeout(10). Fails closed on any ICC error.

Concurrency: RebuildGuard (RAII Drop) acquired before the async guardian check. The Drop impl releases the lock even if the callback traps in cleanup context, so a poisoned ICC can never stick the lock on true.

Rebuildable targets (enum RebuildableIndex):

Variant	Source-of-truth	Notes
All	every supported SOT	Sequentially rebuilds every supported index.
CampaignState	CAMPAIGNS.campaign_state	Sync repopulate, ~10k entries.
SponsorUsers	USER_SPONSORS	Sync repopulate.
ContributorsByCampaign	CAMPAIGNS.contributions[].user	Sync repopulate.
SponsorCaseInsensitive	SPONSOR_INFO keys (lowercased)	Sync repopulate.
PrincipalSponsor	SPONSOR_INFO.principal_id	Sync repopulate.

Every rebuild branch runs clear_new() on the target (which DOES wipe corrupt bytes) and re-inserts from the source-of-truth, skipping placeholder rows.

---

Governance-side recovery — guardian_rebuild_indexes

Source: src/ohshii_governance/src/disaster_recovery.rs + src/ohshii_governance/src/memory.rs.

Endpoints:

guardian_rebuild_indexes : (GovRebuildableIndex) -> (variant { Ok: text; Err: text });
guardian_abort_rebuild   : () -> (variant { Ok: text; Err: text });
get_rebuild_status       : () -> (RebuildStatus) query;

Auth: guardian only (sync memory::is_guardian against the local GUARDIAN_PRINCIPAL cell). Admin and governance-self are deliberately rejected — the endpoint is the emergency lifeline for when the proposal pipeline is unusable.

Rate limit: per-principal cooldown of 10 minutes + a MIN_CYCLES_RESERVE = 50B floor check. Refuses the call if the canister is already below the floor.

Targets

pub enum GovRebuildableIndex {
    All,
    FollowersByFollowee,        // SOT: FOLLOWS (71)
    FollowersCount,             // SOT: FOLLOWERS_BY_FOLLOWEE
    VotersByProposal,           // SOT: PROPOSAL_VOTES (47)
    RpKeysCache,                // single-message clear; lazy refill via tECDSA
    JwksCache,                  // single-message clear; lazy refill via HTTPS outcall
    AnyMethodVerifiedUsers,     // SOT: VERIFIED_USERS(53) ∪ DECIDEID_VERIFIED_USERS(63)
}

Hard wall — never rebuildable

The enum is the structural allowlist. Indices NOT on the enum cannot be passed; the following are explicitly enumerated in HARD_WALLED_LABELS for review clarity:

• USED_NULLIFIERS (54) — anti-replay World ID
• USED_DECIDEID_PRESENTATIONS (60) — anti-replay DecideID
• PROPOSALS (43), EXECUTED_PROPOSALS (44) — primary governance state
• VOTING_PARAMS_CONFIG (51) — singleton, DAO-voted
• GUARDIAN_PRINCIPAL (13) — auth root of this very endpoint
• VERIFIED_USERS (53), DECIDEID_VERIFIED_USERS (63) — single-SOT verification maps
• FOLLOWS (71) — primary delegation state
• PENDING_DELEGATIONS (73), PENDING_DELEGATIONS_INDEX (78) — in-flight queue applied by the cascade timer

Rebuilding any of the above would either erase user-visible state (proposals, votes) or weaken security primitives (anti-replay, auth). They MUST stay primary-source.

Chunked execution

For derivable indices that may contain thousands of entries, the rebuild is timer-driven with chunks of 500 entries / 5 seconds:

Safety primitives

Primitive	Role
TickGuard (RAII)	Heap flag REBUILD_TICK_IN_FLIGHT. Drop releases on trap. Without this, a poisoned chunk would leave the flag stuck on true and brick every future tick.
Wall-clock watchdog	REBUILD_MAX_DURATION_NS = 4h. Auto-aborts a cascade that has run too long, regardless of progress. Operator always has a fresh slate within a workday.
Failure counter (probe-then-reset)	consecutive_failures: Option<u32> in stable state. Bumped BEFORE the chunk work; reset to 0 on successful flush. After MAX_TICK_FAILURES = 5 consecutive ticks without flush, cascade auto-aborts. Protects against poisoned-chunk infinite trap loops.
Write-freeze (assert_index_writable)	Every mutator of a rebuildable index calls this as its first instruction. If a cascade is active against the target, the call fails-loud with maintenance_in_progress: index <X> rebuild active, ETA ~Ns rather than silently corrupting the rebuilt index.
guardian_abort_rebuild	Operator escape hatch. Resets state.active = false (stable) AND REBUILD_TICK_IN_FLIGHT = false (heap), cancels the timer. Partial rebuild left as-is — operator may invoke a fresh guardian_rebuild_indexes to restart.

Mutators currently gated by assert_index_writable

Mutator	Source	Freeze targets
follow(followee)	lib.rs:~6741	FollowersByFollowee + FollowersCount
unfollow()	lib.rs:~6891	FollowersByFollowee + FollowersCount
set_accepts_followers(accept)	lib.rs:~6948	FollowersByFollowee + FollowersCount
dismiss_follower(follower)	lib.rs:~7024	FollowersByFollowee + FollowersCount
vote(proposal_id, decision, voting_token)	lib.rs:~2336	VotersByProposal — except guardian-removal votes (freeze-exempt, see below)
register_world_id_verification(...)	lib.rs:~6155	AnyMethodVerifiedUsers
register_decideid_verification(...)	lib.rs:~5136	AnyMethodVerifiedUsers
register_decideid_oidc(...)	lib.rs:~5645	AnyMethodVerifiedUsers
admin_clear_world_id_verified(principal)	lib.rs:~4671	AnyMethodVerifiedUsers
admin_clear_decideid_verified(principal)	lib.rs:~4982	AnyMethodVerifiedUsers
admin_test_add_votes(proposal_id, votes)	lib.rs:~2667	VotersByProposal (test endpoint)
process_pending_delegations(now) (timer)	timer.rs:~335	VotersByProposal — early-return BEFORE take_pending_delegations so the queue is preserved

The discipline is: every helper that mutates a rebuildable index must be reachable only through a call site that invokes assert_index_writable first. This guards against the Hidden Mutator Bypass: an index mutator reachable through a path that skips the write-freeze check, silently corrupting an index mid-rebuild. The audit grep is: enumerate every clear_new() / insert( / remove( on a rebuildable map and confirm each call site is dominated by an assert_index_writable guard.

Sanctioned exception — do NOT “repair” it. memory::store_proposal_vote_unchecked performs an intentionally ungated PROPOSAL_VOTES insert: it is the guardian-removal freeze exemption (see “Freeze exemption” below, enforced by scripts/check_removal_vote_unchecked.sh). It writes only the source-of-truth (PROPOSAL_VOTES), never a rebuildable index, so the Hidden-Mutator-Bypass grep — which targets writes to rebuildable maps — correctly excludes it. Do not restore an assert_index_writable guard on this path: doing so re-freezes guardian-removal votes and reopens the exact self-removal-block the exemption exists to prevent.

Freeze exemption — guardian-removal votes

The vote(...) row above carries one deliberate exception. The VotersByProposal write-freeze gates vote-record writes while that index is being rebuilt except for a vote cast on a guardian-removal proposal, which is freeze-exempt.

The reason is a self-protection invariant: a guardian must not be able to block its own removal. Triggering a VotersByProposal rebuild is a guardian-only capability, and a rebuild freezes the normal vote() write path. Without the exemption, a captured guardian could keep that rebuild active to indefinitely obstruct the very SetGuardian proposal that removes it.

How the exemption is bounded:

• Target-only classification. At the start of vote() the call is classified — before any rebuild can begin — as a removal vote when its guardian-proposal kind is RemoveGuardian and the target proposal is in its Open round or its veto-override round (Vetoed), so the override votes that complete a contested removal (one the guardian vetoed) are freeze-exempt too — otherwise a captured guardian could veto its own removal, then spam index rebuilds to freeze the override that would remove it. Every other vote (and every non-removal SetGuardian proposal) stays fully gated by the write-freeze. The classification is computed once, up front, so it cannot flip mid-call if a rebuild starts during the voting-power lookup.
• Writes the source-of-truth only. A freeze-exempt removal vote is committed through an unchecked writer that touches only the canonical vote-record map (PROPOSAL_VOTES) — the source-of-truth from which a VotersByProposal rebuild re-derives the per-voter VOTER_HISTORY index. Writing the source-of-truth during a rebuild cannot corrupt it; it is the same class of mutation that vote-removal (already ungated) performs during a rebuild. The vote tally lives on the proposal record (in the hard-walled PROPOSALS map, which is never rebuilt) and is updated through the ungated proposal-update path, so the tally and finalization are unaffected.
• Residual is cosmetic and self-healing. The only collateral is the derived VOTER_HISTORY (“your votes”) index, which stays gated during a rebuild. For a removal vote, its update is soft-skipped if it reports maintenance — the already-committed vote never fails because of it. The next VotersByProposal rebuild re-derives VOTER_HISTORY from the source-of-truth, restoring the entry. The append is idempotent (it only adds a (voter, proposal) pair not already present), in both the live path and the rebuild path, so the exemption can never produce a duplicate entry.
• The window is bounded regardless. A rebuild is not an indefinite state: each cascade is bounded by the same primitives documented above — a 10-minute per-principal cooldown between triggers, a 4-hour wall-clock watchdog that auto-aborts a long-running cascade, and a circuit-breaker that auto-aborts after 5 consecutive failed ticks. So even setting the exemption aside, no guardian can hold the freeze open without limit.

Net effect: the disaster-recovery write-freeze still protects every derivable index against mid-rebuild corruption, while the one flow that must never be blockable — removing a misbehaving guardian — remains reachable, with no lasting effect on any rebuilt index.

Recovery limits that scale with the DAO

Two recovery mechanisms degrade as protected state grows — disclosed so operators size around them rather than discover them mid-incident (the pre-auth cache has a sibling limit; see the Shield Pre-Auth Cache coverage-horizon note).

• The VotersByProposal rebuild has a ~1.44M-vote ceiling, and it restarts (it does not resume). A rebuild processes the vote source-of-truth in 500-row ticks every 5 s (≈100 rows/s); the 4-hour watchdog therefore caps a single run at ≈1.44M (proposal, voter) rows. A re-trigger after an abort restarts from cursor zero — it clears and re-derives the index rather than resuming from a saved cursor — so a rebuild that legitimately needs more than 4 hours never completes. Vote rows accumulate monotonically (finalized proposals and their votes are retained), so a very high-traffic ecosystem DAO eventually approaches this asymptote. It is a recovery limit, not a liveness one: it only bites if the derived VOTER_HISTORY index has drifted (a rare decode/upgrade fault) and needs a full rebuild. The structural fix is a resumable cascade (persist and continue from the saved cursor across an abort); pruning/archiving votes of long-finalized proposals would push the asymptote out but not remove it.
• Campaign DAOs (SONS) expose no rebuild endpoint at all. This means the ceiling above does not exist there — but the converse does: a campaign DAO’s derived indices (per-voter history, lock indices, follower indices) have no in-canister rebuild path, so a drift there is recoverable only by a code upgrade carrying a one-shot reindex. Their bounded per-campaign scale keeps this niche. Note the write-freeze gate is wired on those canisters but inert (it never fires, because no rebuild ever runs) — do not assume it protects index writes there.

---

Operator runbook

Status — read-only check

dfx canister --network ic call ohshii_governance get_rebuild_status

Response fields:

• active: bool — cascade in flight?
• target: opt GovRebuildableIndex — which index is being rebuilt
• started_at_ns: nat64 — IC timestamp of cascade start
• total_processed: nat64 — entries flushed to the rebuilt index so far
• total_pending: nat64 — best-effort estimate at cascade start
• estimated_remaining_seconds: nat64 — ETA, derived from chunk size and tick rate
• consecutive_failures: nat32 — 0 on healthy progression; 5 triggers auto-abort

Trigger a rebuild — guardian only

# Single derivable index
dfx canister --network ic call ohshii_governance guardian_rebuild_indexes '(variant { FollowersByFollowee })'
# Cache flush (single-message, lazy refill)
dfx canister --network ic call ohshii_governance guardian_rebuild_indexes '(variant { JwksCache })'
# Nuclear option — cascade every derivable index
dfx canister --network ic call ohshii_governance guardian_rebuild_indexes '(variant { All })'

While a cascade is active, the gated mutators (see table above) return Err("maintenance_in_progress: ..."). Frontend surfaces this as a maintenance banner via the ShieldMaintenanceError class — see src/frontend/src/utils/shieldedCall.js and the classifyShieldError regex \bmaintenance_in_progress\b.

Abort a stuck rebuild

dfx canister --network ic call ohshii_governance guardian_abort_rebuild

Effect:

• state.active flipped to false in stable memory.
• Heap REBUILD_TICK_IN_FLIGHT force-reset.
• Timer disarmed.
• Partial state on the rebuilt index left as-is — the operator should re-trigger a full rebuild before re-opening the gated mutators to real traffic.

Storage side

# Auth: admin OR governance self OR guardian (async ICC, bounded_wait 10s)
dfx canister --network ic call dao_storage admin_rebuild_indexes '(variant { SponsorUsers })'

Same RebuildableIndex variants as documented in src/storage/src/lib.rs. No chunked timer here — every rebuild branch in storage is sync (CAMPAIGNS is bounded; per-call instruction budget accommodates ~10k entries).

When to use what

Scenario	Endpoint
One specific secondary index shows stale data after an upgrade	guardian_rebuild_indexes(<Specific>)
Public counts (followers, voters) don’t match the source maps	guardian_rebuild_indexes(All)
World ID / DecideID verification check returns false negatives	guardian_rebuild_indexes(AnyMethodVerifiedUsers)
HTTPS-outcalled JWKS or tECDSA-derived pubkey is stale after a rotation	guardian_rebuild_indexes(JwksCache) or (RpKeysCache)
Cascade is stuck — consecutive_failures near 5 or no progress	guardian_abort_rebuild + investigate logs
dao_storage placeholder rows surfacing in get_sponsors_paginated	admin_rebuild_indexes(SponsorCaseInsensitive) etc.

The two stuck-state anti-patterns this design is hardened against:

• Immortal Timer Lock — a poisoned chunk leaves the heap flag REBUILD_TICK_IN_FLIGHT stuck true, bricking every future tick. Mitigated by TickGuard’s RAII Drop (releases even when the callback traps in cleanup context) plus the consecutive-failure auto-abort.
• Hidden Mutator Bypass — an index mutator reachable through a path that skips assert_index_writable, corrupting the index mid-rebuild. Mitigated by gating every mutator on the write-freeze check (see the mutator table above).