Verification checklist

Use this page after installation or after changing policy. The automated test suite covers the code paths; these checks prove the deployed GitHub App, Worker, Turnstile, model provider, and repository policy work together.

Local checks

Run these before deploying or when editing the docs/site:

npm run typecheck
npm test
npm run build

For release readiness, also run:

npm audit --audit-level=high
npx wrangler deploy --dry-run

First repository smoke

Use a demo repository with the GitHub App installed.

• Open a PR from a non-maintainer account. The check should be queued and the PR should receive a VOUCHA comment.
• If require_approval: first_time applies, the challenge should wait for /voucha approve.
• Comment /voucha approve from a write or admin user. The comment should update with the challenge link.
• Open the link as the PR author, accept the challenge terms, pass Turnstile, and answer the quiz.
• Confirm the start and question pages show the automated-agent warning, and that challenge HTML includes voucha-agent-policy, data-agent-policy, and data-ai-use="prohibited" without relying on those declarations as the server-side enforcement mechanism.
• Confirm the page shows 30 seconds per question, refresh does not reset the timer, and no minimum wait prevents an immediate human answer.
• Confirm the check turns green, the PR comment records the attestation, and the check-run output includes the risk report.
• On a clean pass with output.labels.passed enabled, confirm the PR receives VOUCHA:passed and no longer carries VOUCHA:failed or a stale VOUCHA:flagged label.

Policy scenarios

Walk each scenario that the repository intends to rely on:

• docs-only or skipped paths produce an exempt success check;
• draft PRs stay quiet when draft_prs: ignore;
• missing required accountability fields fail before a quiz is created;
• valid acknowledgement and AI disclosure allow the PR to continue;
• a linked trusted issue can exempt planned work;
• untrusted or weakly related issue links fall through to the gate;
• configured github_team or repository_permission exemptions work for a known account;
• prior_merged_prs exempts an author with enough merged PRs;
• configured code_honeypot patterns appear as maintainer-facing findings when introduced in added diff lines;
• a docs-only delta after a pass carries that pass forward under the default rechallenge policy;
• a meaningful delta after a pass creates a short follow-up quiz scoped only to commits after the passed head.

Failure drills

VOUCHA should fail open for service-side problems and fail closed only for repository policy requirements.

• Temporarily break the model provider or model name, then start a quiz. The check should become neutral rather than blocking the merge.
• Configure context.investigator: flue without a working Flue service. Quiz generation should become neutral.
• Submit a wrong quiz answer. A non-final failure should offer Try again in the app immediately by default, without requiring a return to GitHub, and the retry should get a fresh quiz. With output.labels.failed enabled, the PR should receive VOUCHA:failed and drop stale success labels.
• Start a quiz and revisit its start URL in a second tab. Both tabs should lead to the same active attempt, not generate another question set or consume a second attempt.
• Abandon an admitted quiz beyond its overall deadline. The sweep should close and consume that attempt. With a positive cooldown_minutes, later attempts should use increasing waits capped at eight times the base.
• Verify a production Turnstile response is rejected when its hostname, action, or signed cData does not match the current challenge. A rejected token is a hard verification failure; a Siteverify outage is neutral.
• Complete a correct quiz with two documented ambiguous interaction signals. The check should wait for confirmation rather than pass or fail. An author with an established passkey can confirm in the browser. Otherwise, a write-capable maintainer other than the PR author can comment /voucha confirm.
• After a clean pass, test optional passkey enrollment. Confirm cancellation or an unsupported browser leaves the contributor unblocked and the maintainer confirmation path remains available.
• Set confirmation.webauthn: false. Confirm the clean-pass page has no enrollment control, all passkey endpoints return 403, and a paused result shows only the independent maintainer /voucha confirm path.
• Exhaust all attempts. The check should stay failed for manual maintainer review. Comment /voucha retry from a write-capable maintainer; VOUCHA should preserve the previous audit and start a fresh challenge on the same commit.
• With enforcement.auto_close enabled, exhaust all attempts or trigger a hard assistance failure. The check should stay failed and the PR should close.
• Leave an awaiting or ready challenge untouched. The scheduled sweep should neutralize stale setup after the configured stale window.

What to record

For each rollout, record:

• repository and PR used for the smoke test;
• effective .github/voucha.yml;
• expected and observed check conclusion;
• whether a PR comment was created or updated;
• whether the author-facing quiz link worked;
• the configured label object and observed passed, failed, or flagged label transitions;
• any risk-report signals or neutral outcomes;
• whether the result was acceptable for branch protection.

Known limitations

VOUCHA is an accountability and review-evidence layer, not an unbeatable security boundary.

• An agent with browser control can potentially take the quiz on behalf of the author. The product records an attestation and risk report; it does not prove humanness.
• Pull request text and diff content are untrusted input to quiz generation. A hostile contributor can try prompt injection. Correct answers stay server-side, so this can make a quiz weaker but should not reveal an answer key.
• Webhook handling is asynchronous. If background work fails after GitHub accepts the webhook, recovery depends on GitHub redelivery, idempotent event handling, the scheduled sweep, and maintainer-triggered /voucha retry for a terminal challenge after the service recovers.
• PR comment lookup currently checks one page of issue comments. Very noisy PRs can receive a duplicate VOUCHA comment if the tracked comment is beyond that page.