Disaster Recovery

Scope: the derivable-index recovery endpoints of ohshii_governance and dao_storage.

Why the feature exists

ohshii_governance and dao_storage keep several secondary indices that are derived from a canonical source-of-truth map. Two examples:

FOLLOWERS_BY_FOLLOWEE is derived from FOLLOWS. Without it, listing the followers of a delegate would require iterating the entire FOLLOWS map — a cost incompatible with a public query.
ANY_METHOD_VERIFIED_USERS is the union of the four per-provider verified sets (World ID, DecideID, Telegram, Google). It lets the universal vote gate check verification status in O(log N) without consulting four maps.

Every mutator writes a secondary index together with its source-of-truth. The two can still drift apart if, for example, an upgrade leaves a secondary half-written or a stored row fails to decode.

Because a secondary index is derivable, it can always be rebuilt from its source-of-truth. The recovery endpoints do that safely: clearing and repopulating an index never touches the source-of-truth, and a rebuild that fails cannot leave the canister stuck.

The Guardian is the role authorised to trigger this recovery on ohshii_governance. The DAO proposal pipeline is deliberately not the trigger path: the rebuild endpoint must remain reachable even when the proposal pipeline itself is the thing that needs repair.

Storage-side recovery — `admin_rebuild_indexes`

admin_rebuild_indexes : (RebuildableIndex) -> (variant { Ok: text; Err: text });

Callers: ohshii_governance, the bootstrap principals of the transition phase, or the governance-set Guardian (checked by asking ohshii_governance.is_guardian, failing closed if that call fails). Anonymous callers are rejected. A concurrency guard ensures only one rebuild runs at a time.

Rebuildable targets (RebuildableIndex):

Variant	Source-of-truth
`All`	every supported index, rebuilt in sequence
`CampaignState`	each campaign’s state
`SponsorUsers`	the user → sponsor map
`ContributorsByCampaign`	each campaign’s contribution records
`SponsorCaseInsensitive`	sponsor codes (lowercased)
`PrincipalSponsor`	the sponsor principal of each sponsor record

Every storage-side rebuild runs in a single call: it clears the target index and re-inserts from the source-of-truth, skipping rows that fail to decode.

Governance-side recovery — `guardian_rebuild_indexes`

guardian_rebuild_indexes : (GovRebuildableIndex) -> (variant { Ok: text; Err: text });
guardian_abort_rebuild   : () -> (variant { Ok: text; Err: text });
get_rebuild_status       : () -> (RebuildStatus) query;

Callers: the Guardian only. A cooldown separates two triggers, and the call is refused if the canister’s cycle balance is below a reserve floor.

Targets

Variant	Source-of-truth / effect
`All`	every derivable index below
`FollowersByFollowee`	rebuilt from `FOLLOWS`
`FollowersCount`	rebuilt from `FOLLOWERS_BY_FOLLOWEE`
`VotersByProposal`	the per-voter vote history, rebuilt from the vote records
`RpKeysCache`	cache cleared; refilled on demand via threshold ECDSA
`JwksCache`	cache cleared; refilled on demand via HTTPS outcall
`AnyMethodVerifiedUsers`	rebuilt from the four per-provider verified sets

What can never be rebuilt

The variant list is the allowlist: an index that is not a member cannot be passed. Primary state and security primitives are deliberately absent from it, so no recovery call can erase or regenerate them:

the anti-replay sets of every verification provider, and the canister-generated salt behind the Telegram and Google anti-replay digests;
proposals, executed proposals, and the DAO-voted voting parameters;
the Guardian principal itself (the authorisation root of this endpoint);
the four per-provider verified sets;
the delegation records and the queue of pending delegations.

The verification salt in particular is readable by no query and has no source to be rebuilt from. Losing or replacing it would not remove anyone’s verification (the verified sets are separate), but it would stop the canister from recognising accounts that are already bound to a wallet — which is why no recovery path touches it.

Chunked execution

Indices that may hold many entries are rebuilt on a timer, in bounded chunks, so no single message runs out of instructions:

The AnyMethodVerifiedUsers rebuild walks the four providers in a fixed order (World ID, DecideID, Telegram, Google), and its progress figure counts all four sets.

Safety properties

Property	Guarantee
Re-entrancy guard	Released even if a tick traps, so a failing chunk cannot block later ticks.
Wall-clock watchdog	A cascade that runs longer than 4 hours is aborted automatically.
Circuit breaker	After 5 consecutive ticks that fail to make progress, the cascade aborts automatically.
Write-freeze	While an index is being rebuilt, every call that would write to it is refused with `maintenance_in_progress: …` instead of corrupting the index mid-rebuild. The frontend shows this as a maintenance notice.
`guardian_abort_rebuild`	Stops an active cascade and disarms its timer. The partially rebuilt index is left as-is; a fresh `guardian_rebuild_indexes` restarts it from the beginning.

Calls refused during a rebuild include following / unfollowing a delegate, changing follower settings, voting (for the VotersByProposal index), and registering or clearing a verification (for AnyMethodVerifiedUsers). The delegation cascade timer pauses without consuming its queue.

Guardian-removal votes are never frozen

A guardian must not be able to block its own removal. Triggering a VotersByProposal rebuild is a guardian-only capability and freezes the normal vote path, so without an exception a captured guardian could keep rebuilds running to obstruct the proposal that removes it.

Votes on a guardian-removal proposal are therefore exempt from the write-freeze, both in its open voting round and in its veto-override round (so a guardian cannot veto its own removal and then freeze the override vote). The classification is made once, at the start of the vote call. Such a vote writes only the source-of-truth vote record and the tally on the proposal, which no rebuild touches; the only index it may skip is the voter’s own “your votes” history, which the next VotersByProposal rebuild restores. Every other vote stays subject to the write-freeze. For the same reason, the fresh verification attestation that a veto-override ballot requires is never frozen either.

Independently of this exemption, no rebuild can hold the freeze open indefinitely: the cooldown between triggers, the 4-hour watchdog and the circuit breaker all bound it.

Campaign DAOs

Campaign DAOs (sons_governance) expose no rebuild endpoint. Their derived indices are bounded by the size of one campaign, and a correction there ships as a canister upgrade approved through that campaign’s governance.

Reading the rebuild status

get_rebuild_status is a public query:

Field	Meaning
`active`	a cascade is in flight
`target`	which index is being rebuilt
`started_at_ns`	IC timestamp of the cascade start
`total_processed`	entries written to the rebuilt index so far
`total_pending`	best-effort estimate taken at the cascade start
`estimated_remaining_seconds`	ETA derived from the chunk size and tick rate
`consecutive_failures`	`0` while progressing; the cascade aborts itself at `5`

Example, read anonymously:

icp canister call xnsdw-yaaaa-aaaal-qsqua-cai get_rebuild_status '()' -n ic --query --identity anonymous -o candid

Who can do what

Action	Caller
Read the rebuild status	anyone
Rebuild a governance index or flush a governance cache	the Guardian
Abort a governance rebuild	the Guardian
Rebuild a `dao_storage` index	`ohshii_governance`, the Guardian, or a transition-phase bootstrap principal