Use AI effectiveness

Tasks

Compare AI effectiveness with productivity in the administrator view. Developers use their personal views to inspect their own results and compare with teammates who choose to share their scores.

Prerequisites

1. Enable the feature. Developer Effectiveness is off by default. An admin turns it on under Settings ▸ General ▸ Features ▸ Developer Effectiveness. Until then the AI Effectiveness tab displays an enablement message, and effectiveness scoring, coaching, and digest sections remain disabled.
2. GitHub delivery attribution is configured and ingesting merged PRs — see Configure GitHub attribution. The leaderboard is built entirely on attributed PRs; no repositories means no data.
3. Developers' GitHub logins are resolved to identities (Settings → Users → Identity mappings → Unmapped lists unresolved logins). Unresolved developers still appear, keyed by their GitHub login, and any developer without a resolved team (unresolved, or resolved but not yet assigned to a team) shows up under the team selector's Unassigned cohort rather than being excluded from team-scoped views.

Scores populate automatically: facts are derived as PRs merge, and weekly rollups recompute daily. A freshly connected installation shows data for past weeks after the initial 90-day backfill completes.

Read the board

1. Open Productivity ▸ Outcomes ▸ Overview to compare AI effectiveness, AI PRs, and Productivity score in one table. Choose AI Effectiveness for the effectiveness view.
2. Choose Teams, Projects, Repositories, or Developers. Select the last 7, 30, or 90 days, or custom dates. Search for the entity you want to inspect.
3. Read the AI effectiveness score and its band alongside the AI PR count and percentage. Missing evidence remains unavailable. Use Repository status to inspect collection problems; existing results remain visible while scans run.
4. Select an entity or its AI effectiveness score to open the detail drawer. Inspect score dimensions, the weekly trend, and the merged PRs supporting the selected period.
5. Select a PR to inspect its assessment and supporting evidence. Expand How these metrics are calculated for scope and calculation details. Close the drawer to return to the comparison table.

If a selected developer is no longer available, select Choose developer, then choose another account in the scope picker. Your period and PR filters stay selected. To switch from a developer to a team, repository, or project, change Scope type and select an entity. If the directory expires, select Retry scope search.

Existing weekly leaderboard bookmarks open AI Effectiveness with the same week, grouping, and team, project, or repository filters. Retained filters appear as removable tags and also apply to PR details. A bookmark without a grouping opens Developers for the last full week.

In the entity detail drawer and My merged PRs in the Builder Portal, search by title, repository, or PR number and select a column heading to sort the matching PRs across all pages. Search and work-type filters affect the PR table; summary scores and trends continue to describe the selected period.

Existing published-comparison bookmarks open Published report history, preserving the selected report and metric. Use Back to productivity to return to the current overview.

AI effectiveness uses fixed, administrator-configured score bands. The defaults are:

Band	Score range
Low	Below 50
Medium	50 to below 75
High	75 to below 90
Elite	90 through 100

A score of 57.5 is Medium under these defaults, even if it is the highest score in the comparison group. Bands depend on the score and configured cutoffs, not peer rank or group size. Scores still use complexity-adjusted peer comparisons and sample-size adjustment; a score of 55/100 does not mean 55% effective. Insufficient evidence receives no band. AI adoption is displayed as counts and percentages without Low/Medium/High/Elite labels.

Scores use PR facts for the selected period. Weekly percentile ranks are not averaged to produce a period score. Current team and project membership determine group comparisons. Current coaching recommendations remain independent of the selected historical period.

Generic sessions and declarations do not establish whether AI was used for coding, design, or testing. Adoption details label those types as unmeasured and link to Activity for work without a PR. “No assistance recorded” is not a conclusion that no AI was used.

What developers see

Builders use My Productivity for their score, work types, and PR evidence, and My AI Effectiveness for their AI effectiveness trend and dimensions. Both use the same selectable date ranges as the administrator view. See Builder Portal.

Developers retain access to their own PR evidence and coaching. Administrator visibility controls whether comparisons are named. Under Member choice, individual preferences control names:

• My AI Effectiveness places the score, colored band, and coaching in the top card. My Activity shows bands, scores, and whole-number percentiles even for anonymous colleagues; only their names are private. The My AI Effectiveness team table retains band-only anonymous rows under Member choice. Weekly history and team visibility contains completed-week history and the Show my name to teammates preference.
• The weekly usage digest email includes an AI Effectiveness section (score, week-over-week trend, and up to two active recommendations) once the developer has scored weeks.
• Developers who prefer a lower email cadence can set their digest frequency to Monthly (Settings ▸ Notifications) to receive a 30-day summary with the effectiveness trend instead.
• Developers with an active coaching recommendation get a short daily tip — an in-app notification (and an in-IDE message when a Claude Code session closes), rotating across their recommendations. Admins can turn tips off per channel under Settings ▸ Notifications ▸ Coaching Tips. Tips are emailed only to developers who explicitly enable Email coaching tips in Settings ▸ Notifications — off by default.

Visibility on upgrade

Existing saved Team and Global values take precedence over individual opt-outs after an upgrade, including Team values saved by older settings forms as their default. Select Member choice to restore opt-in names.

Member choice is the default when developer visibility has never been configured: team comparisons use individual name-sharing preferences. An explicit administrator Team or Global setting enables named comparisons within that scope and overrides individual opt-outs. Self disables comparisons. Administrators can change this policy under Settings ▸ Productivity ▸ AI Effectiveness ▸ Developer visibility. The member sharing switch shows Managed by administrator while an explicit policy is in effect; saved individual preferences apply again if the administrator returns to Member choice. Developers without a team see their own data; they do not fall back to an organization-wide developer list.

Tune the model

Configure the productivity score

1. Onboard repositories in Settings → Productivity → Repositories.
2. Enable Productivity score in Settings → General → Features. The initial scan covers the last 90 days; new merges and periodic catch-up update results automatically.
3. Open Settings → Productivity → Productivity score to set work-type weights. Added and deleted eligible lines are multiplied by the PR's work-type weight.
4. On saving weights, choose future merges only, recalculate the last 30 or 90 days, or select a custom past period. Recalculation does not fetch source or call the AI provider.
5. Open Productivity → Outcomes. Compare AI effectiveness, AI PRs, and Productivity score together in Overview. Select an entity's metric for details.
6. For collection problems, open Settings → Productivity → Repositories. Use Sync selected for multiple repositories or Scan now and Retry failed in a repository's detail drawer.

AI effectiveness model

The AI effectiveness settings link opens Settings ▸ Productivity ▸ AI Effectiveness. Configuration lives in Settings; scores update automatically without a report publishing step. The settings include:

• Productivity score: work-type weights and the period to recalculate when those weights change.
• AI Effectiveness: fixed effectiveness band thresholds, dimension weights, score calibration (shrinkage, organization AI target, and minimum cohort size), and developer visibility.
• Assessments: automatic classification, its budget, and retention settings.
• Practices & signals: enabled practices and thresholds, design-doc patterns, and review-comment classification.

Save effectiveness bands updates live overall and dimension bands, developer histories, personal views, and newly generated digests without recalculating numeric scores. Thresholds must satisfy 0 < Medium minimum < High minimum < Elite minimum ≤ 100. Each minimum is inclusive. Existing published reports and already-sent digests remain unchanged.

Each section has its own save action. AI effectiveness and practices saves preserve other configuration sections, including changes made by another administrator. To rebuild stored weekly history after an AI effectiveness model change, save the settings, open Recompute… on the Productivity page, and select AI Effectiveness. Historical published reports retain their existing rules and results.

The same configuration is available over the API: GET/PUT/PATCH /api/v1/effectiveness/config (admin) holds it as a single JSON document. PUT replaces the document. PATCH atomically replaces only the supplied top-level fields and returns the merged configuration; omitted fields are preserved. Nested maps such as weights and practices are replaced as complete fields:

{
  "weights": { "ai_leverage": 0.2, "review_efficiency": 0.2, "model_fit": 0.15,
               "cycle_efficiency": 0.15, "quality": 0.15, "resource_discipline": 0.05 },
  "shrinkK": 5,
  "orgAITarget": 0.8,
  "minCohortDevs": 4,
  "effectivenessBands": { "mediumMin": 50, "highMin": 75, "eliteMin": 90 },
  "complexity": { "weights": { "loc": 0.3, "files": 0.2, "dirs": 0.15,
                               "langs": 0.1, "tests": 0.1, "risk": 0.15 } },
  "designDocPatterns": ["docs/design/**", "specs/*.md"],
  "classifyReviewComments": false,
  "llmNarratives": false,
  "llmTokenCeiling": 100000
}

GET also returns the shipped defaults for reference. Omitted fields keep their defaults.

designDocPatterns is off by default (empty). When set, a PR whose file list matches any pattern (case-insensitive; dir/** for a directory prefix, otherwise a path glob) is recorded as carrying a design document — a tracked fact for future Design & Planning scoring, not a scored dimension yet.

classifyReviewComments is off by default — when enabled (and an internal AI provider is configured under Settings ▸ Providers), review comments are topic-labeled (tests, security, style, …) by a nightly job capped at 20 PRs per run. Only the label is stored; comment text is never persisted. Each classification call is audited and cost-attributed like every internal AI call.

llmNarratives is off by default — when enabled, each developer's weekly scores get a short AI-phrased summary (2–3 sentences, generated only from the structured scores, never from code or prompts) shown on the developer detail panel, the self-view, and in digest emails. Every view renders fully without it. llmTokenCeiling bounds one nightly generation run's total AI tokens (default 100000); developers past the ceiling pick up their narrative on the next run.

Recompute history

After changing the configuration (or resolving a batch of identities), use the unified action to rebuild facts and rollups:

1. Save the updated settings, then open Productivity → Outcomes → Recompute….
2. Select AI Effectiveness. Clear classifications and productivity scores unless you also want to recompute those metrics.
3. Choose Preview recomputation, review the selected operations, confirm, and choose Start recomputation.

AI Effectiveness rebuilds all retained history using saved settings, regardless of the merge period selected for the other operations. The rebuild runs asynchronously: PR complexity, facts, weekly rollups, and effectiveness scores are recalculated. Accepted work is saved durably, survives server restarts, and retries after failures. Disabling AI Effectiveness pauses pending work until the feature is enabled again. Existing published reports retain their results.

For the scope, preservation rules, and AI costs of the other options, see recompute historical PR data.