Ecommerce operational runbooks

Runbooks define safe operator actions. They do not authorize bypassing tenant isolation, accounting approval, idempotency, capability gates, or Odoo readback.

Incident roles and evidence

Role	Responsibility
Incident commander	Severity, coordination, timeline, communications, closure
Operations lead	Queues, workers, leases, retries, runbooks, service restoration
Integration lead	Source/Airbyte/Odoo diagnosis and capability evidence
Accounting lead	Posting/reconciliation impact and financial containment
Security/privacy lead	Isolation, credentials, restricted data, notification decision
Communications owner	Approved tenant/internal updates without sensitive detail

Every incident records incident ID, tenant scope, first/last known impact, affected contracts/versions, correlation/operation/run IDs, containment, evidence, decisions, customer impact, recovery validation, and follow-up owners.

Severity

Severity	Examples	Initial response target
SEV-1	Suspected cross-tenant exposure, credential compromise, duplicate legal posting at scale	Immediate paging and containment
SEV-2	Unknown Odoo mutations aging, settlement close blocked, sustained ingestion loss	Page on-call within 15 minutes
SEV-3	Single-tenant mapping/schema issue, stale mart with correct stale marker	Work queue within business SLO
SEV-4	Documentation/low-impact defect, no incorrect state	Planned remediation

Ingestion lag or missing data

Trigger: two missed hourly runs, webhook/canonical p95 breach, cursor stall, or daily control mismatch.

1. Identify tenant, source connection, stream, connector/catalog version, last good cursor, and ingestion run.
2. Disable affected downstream posting profile when completeness-sensitive records may be missing.
3. Check source authentication/scopes, provider status/rate limits, Airbyte job, destination writes, inbox quarantine, and canonicalizer backlog.
4. Do not advance cursors manually. Preserve failed job/checkpoint evidence.
5. Repair configuration or dependency; run a bounded lookback/backfill under a new ingestion run.
6. Compare source IDs/counts/amounts to accepted/canonical records by date and currency.
7. Rebuild affected mart partitions and verify control hashes.
8. Re-enable only after capability/completeness evidence returns supported.

Exit: cursor advances, backfill/control differences are zero or explicitly resolved, downstream posting impact is reconciled, and marts are fresh.

Odoo unknown mutation outcome

Trigger: timeout, connection loss, ambiguous HTTP result, or worker crash after request delivery.

1. Mark the operation verification_required; stop automatic mutation retries and retain the business-key lease.
2. Record attempt, request hash, delivery timestamps, target/source markers, and last preflight readback.
3. Search Odoo using the fixed command-specific source key and operation marker within the allowed company.
4. Read candidate records and compare full command readback invariants.
5. If exactly one equivalent result exists, attach readback and mark confirmed.
6. If no result exists and evidence proves no side effect, create a new attempt under the same operation.
7. If multiple/partial/conflicting results exist, mark repair_required and assign Odoo/accounting review.
8. Never delete or reverse a possible Odoo result solely to make a retry easier.

Exit: confirmed result, proven safe retry, or approved repair plan; no unknown operation exceeds 24 hours without an active incident.

Settlement or payout imbalance

Trigger: settlement equation mismatch, unknown line type, payout/bank amount mismatch, or Odoo clearing residual.

1. Freeze settlement close and payout posting for the tenant/provider/currency; order ingestion may continue.
2. Recompute source line counts, signed type totals, provider payout, bank amount, and Odoo clearing readback.
3. Check duplicate/missing settlement lines, late refunds, reserves, withholdings, disputes, FX, timing, and mapping drift.
4. Re-extract the bounded settlement period without deleting original evidence.
5. Classify every difference through an approved mapping/policy; do not create an unclassified balancing line.
6. Require accounting approval for mapping, date, account, FX, or rounding changes.
7. Post/reconcile only through a new immutable intent/operation and verify Odoo readback.
8. Refresh settlement/clearing marts and prove the control equation.

Exit: provider, expected, bank, and Odoo amounts reconcile or every residual is an approved explicit exception.

Credential rotation or suspected compromise

Trigger: scheduled rotation, authentication failure, leak scanner, suspicious source/Odoo access, or staff/vendor access change.

1. For suspected compromise, declare security incident and disable affected connection/workflows.
2. Identify secret reference/version and consumers without exposing secret value.
3. Create replacement credential with least privileges and validate target account/company identity.
4. Update the secrets manager reference and canary read-only health check.
5. Re-establish webhooks/Airbyte/Odoo access and run capability scans.
6. Revoke old credential after confirmed cutover; for compromise, revoke immediately where risk requires it.
7. Inspect audit/provider logs for misuse and reconcile activity during the exposure window.
8. Record rotation evidence, affected runs/operations, and next rotation date.

Exit: old credential revoked, new capability evidence supported, no unexplained activity, and queues resume safely.

Source or canonical schema drift

Trigger: unknown required field/enum, validation spike, connector catalog hash change, or adapter fixture failure.

1. Quarantine affected records and mark capability drifted; do not coerce unknown financial values.
2. Capture minimal redacted payload/schema evidence, connector version, catalog hash, and first occurrence.
3. Determine additive optional versus breaking semantic change.
4. Update source schema, adapter, canonical/event version, fixtures, and compatibility tests.
5. Recanonicalize affected records in dry-run mode and compare accounting/mart diffs.
6. Obtain data/accounting approval when amount, sign, tax, identity, or state semantics change.
7. Deploy canary, replay bounded records, and restore capability only after controls pass.

Exit: no quarantined backlog, compatible schema deployed, control totals pass, affected marts rebuilt.

Stale or failed warehouse mart

Trigger: refresh failure, quality gate failure, stale SLO, or dashboard/control mismatch.

1. Keep the last valid partition published and mark freshness/incomplete status visibly.
2. Identify tenant, mart, partition, source high watermarks, batch, and failed test.
3. Compare canonical/readback/reconciliation source controls to atomic facts before inspecting aggregates.
4. Repair model/data dependency; rebuild into a shadow partition.
5. Run uniqueness, relationship, tenant, currency, metric, and control-total tests.
6. Publish atomically only when every gate passes; invalidate affected caches.
7. Verify reporting API freshness/batch IDs and representative dashboard totals.

Exit: valid partition published, controls reconcile, freshness restored, root cause recorded.

Suspected cross-tenant data exposure

Trigger: any evidence that one tenant can access another tenant's API, database, queue, cache, object, search, mart, log, or export data.

1. Declare SEV-1, involve security/privacy, and preserve evidence.
2. Disable affected endpoint/job/export path or broader service where necessary; do not destroy logs/evidence.
3. Revoke compromised sessions/credentials and block active exports/download links.
4. Determine affected tenants, data classes, time window, users/processes, and storage surfaces.
5. Test RLS/authorization/cache keys/object policies using safe synthetic probes.
6. Repair the boundary, rotate credentials/keys where needed, and run the full negative isolation suite.
7. Follow legal/privacy notification decision procedures and approved communications.
8. Restore through canary tenants and enhanced monitoring.

Exit: containment confirmed across every surface, negative tests pass, notification decision recorded, post-incident review scheduled.

Restore and disaster recovery

1. Declare restore scope and freeze conflicting writes/operations as required.
2. Restore PostgreSQL/object evidence/secrets references under isolated recovery credentials.
3. Verify tenant RLS, composite keys, operation uniqueness, outbox/inbox consistency, and holds/retention state.
4. Reconcile queue/broker state to durable outbox and operation records; do not trust acknowledgement alone.
5. Resume source ingestion from recorded cursors with overlap and deduplication.
6. Verify unknown Odoo operations before retrying any mutation.
7. Rebuild marts and compare control hashes/counts/amounts.
8. Canary one tenant, then resume cohorts under monitoring.

Exit: RPO/RTO met or incident exception recorded, no cross-tenant/idempotency violation, controls and SLOs restored.

Runbook testing and change control

• SEV-1 isolation and credential scenarios are exercised at least twice yearly.
• Odoo unknown-outcome, ingestion backfill, settlement imbalance, mart rebuild, and restore are exercised before pilot go-live and after material architecture changes.
• Each exercise records duration, decisions, missing access/evidence, failed steps, and tracked remediation.
• A runbook change that alters accounting mutation, retention, authorization, or incident notification requires the corresponding contract owner approval.