Troubleshooting Burst Kit

Verbosity

Level	Intended use
error	Service startup or fatal operational failures
warn	Degraded durability, failed maintenance and explicitly enabled diagnostics
info	Startup, HTTP outcome/duration and maintenance summaries; default
debug	Storage RPC method/status/duration with request correlation
trace	Policy evaluation timing and replica acknowledgment phases; short troubleshooting sessions

The API returns X-Request-ID, and the console includes it with failures. The ID is propagated as gRPC metadata to storage logs. Search the same ID across control and node logs. Logs exclude request bodies, tokens, cookies, private keys and plaintext at every level. The shared handler also redacts sensitive attribute keys. Worker failures expose only bounded kinds such as memory_limit, deadline, evaluation and worker_configuration, plus an exit code; raw worker stderr and source are never logged. Do not add payload logging to work around a failed request.

Example flags:

-log-level debug -log-format json

The operator console's Show network status reports active nodes/grants, objects, staging and pending deletion. Run repair and expiry returns expired/repaired/deleted/pending/degraded counts. Use a fresh maintenance idempotency key for a new API run; reusing a key returns its original report.

Protected diagnostics

Diagnostics is disabled by default and is never mounted on the public API mux. Create an independent private token once:

(umask 077; set -C; openssl rand -hex 32 > .burst/debug-token)

Enable it explicitly with:

-debug-listen 127.0.0.1:9090 -debug-token-file .burst/debug-token

The listener must bind a loopback IP. Every request requires Authorization: Bearer TOKEN. /debug/status returns runtime/build information; /debug/pprof/ exposes Go profiles. Profiles reveal internal execution details, so enable them only for the troubleshooting window. A diagnostic listener failure fails startup rather than silently disabling it.

To avoid putting the bearer token in process arguments, create a private curl configuration:

python3 - <<'PY'
import os,pathlib
value=pathlib.Path('.burst/debug-token').read_text().strip()
fd=os.open('.burst/debug-curl.conf',os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600)
with os.fdopen(fd,'w') as output:
    output.write('header = "Authorization: Bearer '+value+'"\n')
PY
curl --config .burst/debug-curl.conf http://127.0.0.1:9090/debug/status

Do not reuse OIDC credentials, a bootstrap token or an object wrapping key as this diagnostics token. Diagnostic configuration is not included in ordinary HTTP responses.

Common failures

Code / symptom	Check
`authentication_required`	Session expired, was rotated or was logged out. Sign in again. API ID tokens are not browser sessions.
`csrf_rejected`	Use the current session CSRF value and exact configured Origin. Browser mutations send both automatically.
`permission_denied`	Exact principal/group IDs, space/resource scope, membership, grant expiry, revocation and deny precedence. Email/group claims from clients are not trusted.
`policy_worker_unavailable`	Configure an absolute worker executable path. Build using scripts/build.sh.
Worker self-test fails / worker exits 2	On non-Linux systems, opt into development soft-memory mode explicitly. On Linux, check inherited process limits and executable permissions.
`policy_denied`	Policy allow result, active version/expiry, result schema, fixed obligations and worker resource limits. Worker errors deny access; they do not authorize fallback access.
`policy_changed_during_upload`	Publication changed the active revision during transfer. Retry with a new operation key after reviewing the current contract; abandoned staging copies are cleaned up.
`insufficient_compliant_capacity`	Required distinct domains/regions, active unquarantined memberships, effective reserved capacity and pending deletions. Requirements are never relaxed automatically.
`replica_acknowledgment_failed`	Endpoint/listener, trusted roots, exact node/control identities, certificate validity and disk space. Inspect storage RPC status with the same request ID.
`no_verified_replica`	Corruption, unavailable nodes, residency policy or quarantine. Reconcile reports degraded durability; add compliant replacement capacity.
`object_expired`	Retention is authoritative. A later TTL increase cannot revive this object.
`idempotency_conflict`	The same key was reused with changed input. Keep a key for exact retries only.
`certificate_superseded` / `attestation_superseded`	Use the renewed certificate and matching private key. An old credential cannot roll back rotation.
Master-key fingerprint mismatch	Restore the original wrapping key. Do not regenerate a key over an existing deployment's key file.
PostgreSQL schema missing	Run explicit migration with an owner role, then start with a separate runtime role. Do not apply the old planning schema as the runtime migration.

Policy budgets and platform limits

Policies have at most 64 KiB source and 100,000 execution steps. The parent process enforces a one-second worker deadline and an 8 KiB output bound. The worker uses a 64 MiB Go memory target. On Linux it also lowers RLIMIT_AS to 8 GiB, RLIMIT_DATA to 256 MiB and RLIMIT_CPU to two seconds. The virtual-address ceiling allows Go’s sparse arena reservations; writable data remains separately capped at 256 MiB. These are process limits, not a claim of a 64 MiB hard RSS/cgroup limit. Production hosts should apply their own container/cgroup budgets as well.

Other platforms refuse hard-isolation mode; -development-policy-soft-memory is an explicit weaker development mode. Starlark has no filesystem/network/process APIs or ambient clock exposed, and module loading is disabled. It is a policy interpreter, not the application-worker sandbox.

Reproduce checks

sh scripts/check.sh
sh scripts/build.sh dev

For a dedicated PostgreSQL test database, set BURST_TEST_POSTGRES_DSN before running checks. Add BURST_TEST_FULL_POSTGRES=1 to run the complete control/data suite against that backend, not only the adapter smoke test. The adapter test creates a uniquely named schema and removes only that schema afterward. Never point the test variable at a shared production database. Without the variable, that test explicitly skips; it is not counted as a successful PostgreSQL validation.

Set BURST_TEST_NATIVE_SOCKETS=1 to run control and storage integration over native TCP/TLS where listeners are allowed. The default uses in-memory connections for restricted development hosts. Native Linux/PostgreSQL checks passed; real-provider/browser interoperability remains separate. CI provisions PostgreSQL but is not a substitute for running deployment acceptance checks on your target infrastructure.

Component acceptance checks

sh scripts/test-today.sh local runs the nine-module race/vet suite, builds six commands, verifies the manifest and exercises the Starburst explain/plan CLIs. sh scripts/test-today.sh all additionally requires configured Vault credentials, Docker/runsc and PostgreSQL tools and runs live gates. See component completion for exact requirements and checks that remain pending. /metrics on the optional protected diagnostics listener exports bounded request counts and cumulative duration without resource labels.

key wrapper startup check failed means the configured custody adapter could not wrap and unwrap a probe. Verify Vault TLS trust, agent token permissions, Transit key name/type and encrypt/decrypt policies. Custody identity is pinned in metadata: changing the endpoint/mount/key or switching between local and Vault requires an explicit migration. Never replace keys to clear a startup failure.

credential rotation rollback or invalid peer identity during a new handshake means managed files failed identity, chain, usage, expiry or NotBefore checks. Publish matching key/certificate files atomically, retain private key permissions, and refresh the node's attested membership. Root-bundle changes require restart.

Worker cleanup_pending includes the generated container name. A supervisor must retry Worker.Cleanup; operator inspection of the trusted Docker host is needed if the daemon is unavailable. A failed create response can still leave a container.