Review PR contributions

This workflow requires an administrator account, GitHub attribution enabled, and a configured GitHub integration with access to the repository. All onboarded repositories that are eligible for collection participate. The PR must already appear in the evidence ledger with a recorded merge date. Read Understanding contributions for definitions and privacy limits.

Enable automatic assessments

1. Open Settings → Productivity → Assessments. Enable Automatically classify merged PRs.
2. Configure the internal AI provider and model under Settings if needed. The model needs known input and output pricing. In Assessment settings, inspect the displayed provider and model, set a positive daily budget in USD, and choose the historical window (default 90 days, maximum 365).
3. Enable Automatically classify merged PRs and save. This authorizes sending PR descriptions, linked-issue context, file paths, and bounded patches from all onboarded repositories to the displayed provider. New merged PRs and imported history are queued automatically. Existing installations keep automation off until you enable it.
4. Inspect Automatic assessments for queued, running, completed, and failed totals. Completed results appear as AI assessed or Rule assessed, according to their decision method, in the table and history without human confirmation. Use Human reviewed only when you want a narrower view.
5. If PRs are missing, open Settings → Productivity → Repositories and use Scan now. Automatic backfill fills missing assessments within the configured history window. To replace completed assessments, use Recompute… on the Productivity page.

Turning off Automatically classify merged PRs stops background classification. Previously enabled individual AI requests remain available, with source and cost approval for each request.

The daily budget is shared with individual AI requests. Budget or concurrency limits pause queued work; the worker resumes when capacity is available. A provider or pricing change pauses automation until you save approval for the updated settings. Review recent failures and the PR's History → AI requests and costs before choosing Retry failed assessments: retrying can incur additional cost. While a historical reprocessing run is active, retries can rejoin that run; failures from other runs wait until it finishes. Each retry request resumes at most one historical run, whose progress appears under Recompute…. Retry requests have their own configuration audit entries with the retry count, selected filters, and historical run reference when applicable. Retries skip assessments that changed after the original request. Start a new recomputation to include their current versions; manual or reviewed assessments still require the explicit replacement option.

After a classification rubric upgrade, inspect the new definitions and save assessment settings to renew approval. Existing assessments and failed jobs are retained. The upgrade does not automatically replay the historical window. Start with a small failure-category retry batch in Health → Productivity, inspect the resulting types and citations, and expand only when the results are accurate. Turning automation off pauses further work; retained assessment history remains available for review and correction.

Assessment history retains PR and linked-issue metadata, including titles and labels, under Retention for new revisions and groups (days). The default is 90 days, configurable from 1 to 395 days; existing revisions keep their original expiry. Raw PR descriptions and patch text are not retained in assessment history. The optional PR evidence cache has separate storage and retention controls.

Use conservative classification rules is optional and off by default. A matching source preview offers Create rule proposal with no model charge or source transfer to an AI provider. Rules require complete inspected source and consistent context; a title prefix alone is insufficient. Automatic processing still pauses at its configured provider, budget, and concurrency limits.

Configure PR evidence caching

1. Open Settings → Productivity → Assessments → PR evidence cache. Caching is disabled by default. Enable Cache PR evidence to retain bounded raw patches, descriptions, and linked-issue context in the workspace database for reuse.
2. Set the total size, per-entry size, retention, and metadata freshness limits. Defaults are 512 MiB total, 8 MiB per stored entry, 90 days retention, and 24 hours freshness. Select Save cache settings.
3. Inspect stored PRs, compressed size, fresh snapshot count, last collection, and hit/miss/eviction counters. Counters describe the server process answering this request and reset on restart; in a replicated deployment, different requests may show different process counters.
4. To replace cached source, choose the entire cache, one repository, or one PR number within a repository. Select a suggested repository or enter its exact owner/repository name, including repositories that are no longer onboarded. For a renamed repository, use its current name; clearing follows its registered identity and includes entries stored under earlier names. Changing the repository clears the PR number; confirm the displayed target before proceeding. Select Clear cache / refresh on next use and confirm. Source is fetched again when next requested. Clearing does not start a bulk scan or AI assessment, and does not delete existing assessments, reviews, or reports.
5. To stop retaining raw source, disable caching and save. Existing entries become unavailable for reuse immediately and are removed in small background batches.

Changing size or age limits also schedules bounded cleanup. Counts can temporarily include entries awaiting removal. Cache deletion makes database space reusable; it does not necessarily shrink the database file. Oversized or unavailable patches retain their source limitations and may be processed without caching. Existing assessment history cannot populate the raw cache because it does not retain patch text.

Changes to ingested PR evidence, along with authenticated PR lifecycle, description, label, and revision updates, invalidate cached metadata. Reviewer assignments, locks, and draft transitions preserve the cache when the evidence is unchanged. Freshness expiry refreshes context and checks comparison revisions; unchanged patches can be reused. Unobserved changes, including issue-only edits and base revision changes without a PR update event, are detected at the next freshness refresh. Warm reuse is a bounded-age snapshot, not a fresh GitHub check. Choose explicit refresh when a demo or investigation needs the latest GitHub state. A cache hit avoids source retrieval, but a requested new AI assessment still uses the configured provider and budget.

The assessment concurrency setting defaults to two and accepts one to eight workers. Keep two until a representative run shows a benefit from increasing it. GitHub retrieval retains a separate shared four-request limit and quota pacing; increasing model workers does not increase GitHub's quota. Very low known quota can pause collection until reset. Inspect Health → Productivity for failures and budget/capacity pauses before changing concurrency.

Recompute classifications and scores

1. Open Productivity → Outcomes and choose Recompute…. This is the common action for historical recomputation.
2. Select the data to rebuild. PR classifications and assessments reuses compatible fresh cached evidence when enabled, or retrieves PR and linked-issue context, file paths, and bounded patches, then reassesses work type (feature development, bug fix, and other categories), summary, evidence, and other assessment fields. Each successful assessment updates its own PR’s productivity points when scoring is enabled. You can clear Productivity scores to avoid rebuilding all other scores in the period.
3. Select Productivity scores alone to recalculate points from stored eligible line counts, existing classifications, and the weights applicable to each PR. This option does not call GitHub or an AI provider and does not change weight settings. A PR without usable stored file statistics still needs repository collection.
4. Optionally select AI Effectiveness to rebuild weekly PR facts and effectiveness scores with saved effectiveness settings. This operation rebuilds all retained history; it does not use the merge period selected for the other options. Accepted rebuilds survive server restarts and retry after failures. Disabling AI Effectiveness pauses pending work until the feature is enabled again.
5. Choose the last 30 or 90 days, custom merge dates, or All imported history. The assessment rerun can include history older than the automatic backfill window. It covers eligible configured repositories independently of the report's team, project, repository, and branch filters.
6. Manual, reviewed, and disputed assessments are preserved by default. Select Also replace manual, reviewed, and disputed assessments only when you intend to replace them with new automatic assessments. This override clears when you turn classifications off or reopen the dialog. The replacements are not marked as human reviewed. An edit made after your request takes precedence over queued or running reprocessing.
7. Leave Refresh evidence from GitHub first unchecked to allow compatible cache reuse, or select it to fetch fresh source for every selected PR. Select Preview recomputation. Inspect the eligible, preserved, currently running, and estimated cached PR counts. Preview makes no GitHub or model calls. Confirm the selected work and any assessment AI usage, then select Start recomputation.
8. Follow queued, running, completed, and failed assessment counts on the Productivity page. Productivity scores show Recalculating while queued; each successful new assessment updates its PR's points again. AI Effectiveness refreshes asynchronously. Inspect failures before retrying.

If assessment settings fail to load, retry loading them before selecting classifications. You can still recompute enabled productivity scores and AI Effectiveness. A failed overview load does not disable these independent actions; preview checks whether each feature is available. Audit records identify whether reviewed assessments are included and make clear that effectiveness rebuilds all retained history. Effectiveness rebuild requests, including those from existing API clients, survive service restarts and resume after a feature pause.

Existing results remain visible until replacements succeed. Previous assessment revisions retain their original retention policy. If the assessment saved in a queued request expires or changes, that job stops before replacing it. Start a new recomputation to include the current state. Failed assessments keep their prior results. Reprocessing does not import missing PRs, refresh scoring line counts, change score weights, recompute AI-session attribution, or change published reports.

Classification reruns use the approved provider and existing daily AI budget, including the concurrency limits shared with individual assessments. When enabled, conservative rules handle matching changes locally. Other changes send assessment evidence to the configured provider. Work pauses when the daily budget is exhausted and resumes after its UTC reset; large runs can take several days. Turning automatic assessments off pauses classification work. Only one bulk assessment rerun can be active at a time; PRs already being assessed are skipped when the run is queued.

Review a sample or assess manually

1. Select Review queue to see disputes, changed sources, manual drafts, incomplete or unknown classifications, and a quality sample. Set the sample percentage in Assessment settings; it defaults to 10%. Sampling uses PR numbers systematically, rather than a random statistical sample.
2. Open a merged PR or assessment, select the Source tab, and choose Preview source. Inspect the description, file inventory, patches, revisions, and limitations. Preview fetches source but does not call the model.
3. To assess manually, choose Create or correct assessment, fill in the assessment, cite supporting patches, and choose Save proposal. Manual assessment works with AI disabled. For an individual AI request, inspect and approve the quoted provider, model, source, and reserved cost, then choose Request AI proposal.
4. Optionally correct the summary, type, stage, significance, citations, and limitations. Add a review reason, acknowledge inspection, and choose Confirm reviewed. This changes the provenance to Human reviewed; AI assessments already appear in default reporting.

Incomplete source requires unknown significance. Source or concurrent-review changes require a fresh preview. Labels describe observed changes; they do not establish business value or developer performance.

Source preview inventories up to 1,000 changed files and selects at most 24 evidence blocks totaling 64 KiB of patch text. Large patches can contribute exact excerpts, with implementation, tests, documentation, and supporting files considered across all file pages. An excerpt can start inside an original diff hunk. Missing patches and omitted lines remain visible as incomplete source; inspect the full PR when the selected evidence does not establish its primary purpose.

Explore and compare development mix

1. Set the date range and project, repository, developer, and team filters at the top of Productivity. These filters scope the charts and PR tables together. Multiple values within a filter are alternatives; different filters narrow each other.
2. In Development mix, choose Compare teams, Compare projects, Compare repositories, or Compare developers, then select two or more groups to compare. The initial comparison shows up to six entities with the most PRs, one bar per entity. Select up to twelve entities.
3. Use Share to compare proportions or PR count to compare volumes. Read each group's classified/imported coverage alongside its mix. Unclassified includes eligible assessments with an unknown work type and explicit manual Unclassified choices. PRs without an eligible assessment or manual choice affect coverage but are excluded from category shares.
4. Click a group name to focus its weekly trend and Work mix PRs. Click a colored segment to also filter to its category. Work-type filters apply to the charts and Work mix PRs using the effective classification. Use Clear exploration to clear the group and category selection. The comparison stays empty until you select groups again. Changing the comparison type starts a new default comparison.
5. Use View data or Export CSV for the comparison data, and View trend data or Export trend for weekly data. Weeks follow PR merge dates. Select Human reviewed only to narrow reporting to reviewed assessments.
6. Chart segments open Work mix PRs, including manually classified PRs without an assessment. Search by PR number, repository, title, or assessment summary. Use the Work type column filter to change the selected type, the Classification filter to choose manual or assessment-based classifications, and the Assessment filter to choose human-reviewed, not human-reviewed, or no eligible assessment. These filters apply to the PR table and retain the chart's entity and reporting period. Open a row for its classification and available evidence. Choose Review assessments to return to the assessment workflow.
7. In Assessed PRs, search and filter assessment status, or select Review queue to inspect exceptions and the quality sample. These table controls do not narrow the charts. Enable optional project and team columns when you need that context. This review table retains the saved assessment values; a manual work-type choice does not rewrite the assessment.

Correct, dispute, and recognize

1. Open an assessment from Assessed PRs. In Overview, read the change summary, baseline, work type and supporting evidence. Select an evidence file to fetch its source patch, or choose Source → Preview source. Fetching source does not request an AI assessment.
2. Choose Correct assessment to inspect source and edit the assessment. If the preview differs from the assessed revision, inspect the warning and current source before submitting a correction. Open History for the latest 50 retained assessment revisions and latest 20 AI request outcomes and costs.
3. To challenge a claim, choose Flag a concern, enter a reason, and choose Record dispute. It is immediately excluded from reporting. Resolve it by inspecting the source and submitting a corrected review.
4. Select current reviewed PRs in the assessment table and choose Group selected. Describe the related work, why the PRs belong together, and the observed result or acceptance evidence. Optionally recognize the contribution in this admin view. This creates no rank, points, email, or Slack message. Expand Related work and recognition to inspect existing groups.

Troubleshoot

Result	Action
Empty assessments after backfill	Check automatic analysis settings, repository onboarding, provider approval, budget, and queue failures. Repository scanning imports PRs; automatic backfill queues missing assessments. Use Recompute… for a fresh assessment of existing history.
Assessment available after merge	Wait for the PR to merge and for ingestion to record its merge date. Basic PR and AI evidence remain available for open or closed-unmerged PRs.
Source unavailable	Inspect the displayed diagnostic and correlation ID, then open Health → Productivity → Manual assessment operation failures to find the affected PR and source stage. A source failure marked not dispatched made no model call. Honor any retry time. Check repository allowlist, GitHub permissions, and whether ingestion has caught up with the current head. Empty changes, unavailable text patches, or changes beyond the source limits may not be assessable. Large text patches can usually supply bounded excerpts.
Partial source	Inspect the displayed limitations. Keep significance unknown; do not infer omitted changes.
AI unavailable	Check the configured internal model, known positive pricing, and source size. Manual assessment remains available.
Budget or concurrency limit	Wait for existing calls or the next UTC budget day, or deliberately change the budget. Only one analysis per PR and two across PRs can run at once. A failed call can retain its reserved cost.
Provider cost exceeded reservation	Inspect AI request history, provider usage, and configured pricing. Actual reported cost is retained and can exceed the dispatch budget; further requests are blocked if that budget is exhausted. The estimate is not a provider billing guarantee.
AI response rejected	The provider answered, but its output failed the assessment schema or evidence checks. Inspect the validation reason in Automatic assessments or History → AI requests and costs. A single JSON code block is accepted; unrelated prose, unknown fields, and invented citations are rejected. Correct the cause before retrying failed assessments.
AI provider request failed	Check the internal AI provider and its availability. Assessment requests can use the two-minute assessment window; source retrieval and a shorter caller deadline reduce the time available. A failed request retains its reservation when actual cost cannot be confirmed.
Analysis accounting could not be confirmed	Refresh History → AI requests and costs before retrying. Inspect the request and its saved-result association; a lost response does not prove the model was unpaid. Storage recovery reuses the existing result without another model call. A retained reservation remains conservative when actual usage cannot be confirmed.
Assessment changed / stale	Refresh history and preview again. Do not reuse an old source or provider approval.
Recognition suspended	Inspect every supporting PR and resolve the disputed, changed, or expired assessment.

Human review is optional for descriptive reporting. Use the review queue and quality sample to check model quality. Automatic analysis does not create contribution rankings or comparative rewards.

For manual source and finalization failures, filter Health → Productivity → Manual assessment operation failures by stage, cause, repository, and time. Open an event to inspect its PR, correlation ID, model-dispatch state, and available GitHub request metadata. Repeated attempts can produce several events for one PR. This panel records failures only, so it cannot measure a success rate. Events are retained for seven days; the first observed failure does not prove complete instrumentation coverage, and earlier versions are not reconstructed. If an error says its event was not recorded, use its correlation ID in server logs.