Explore engineering outcomes

Tasks

Use Productivity → Outcomes to compare observed AI activity with recorded engineering output, then inspect the supporting evidence.

Sign in with an administrator account. Repository onboarding and collection must be enabled to see PR-based outcomes; gateway activity also requires retained session and request evidence. See manage productivity repositories if collection needs attention.

1. Set the shared reporting context.

   Choose Organization, Team, Developer, Repository, or Project, then a period and timezone. The default is the last 7 complete days. Last 7/30/90 complete days exclude today. Last week/month use calendar boundaries. Current periods are marked in progress. Custom dates include both endpoints. Your timezone is saved to your account and shared with your Builder portal preference. The timezone picker matches Baselines: search by city, country, IANA timezone, or UTC offset, or use Browser timezone, Pacific, Mountain, Central, Eastern, or UTC. UTC offsets use the reporting period’s end date and account for daylight saving time. Refresh data reloads a timezone saved in another session.

   The global Scope selector stays available on Overview, Compare, and Evidence, including Organization Overview. Changing the scope type starts a new selection. The current scope remains applied and visible until you choose an entity. Choosing Organization applies the organization scope immediately.

   Developer selectors list users alphabetically by name or email. Search by either name or email, scroll to load more users, or select Load more matching entities inside the dropdown. Users with no activity in the selected period remain selectable.

   Optional PR scope filters constrain delivery measurements. They do not filter gateway requests or sessions to only activity linked to those PRs.

2. Inspect Overview.

   When you select one developer, Overview shows the same Activity report as their Builder portal: productivity band and percentile, points, merged PRs, gateway requests, peak five-minute session activity, comparisons, repository and project contributions, PR evidence, and work types. Expand a chart to inspect it in a larger view, or use Inspect a day to choose a date within the selected period. Projects appear on the left and repositories on the right in equally sized cards, stacked on narrow screens. Select a contribution to filter its supporting PRs. These charts omit projects and repositories with zero merged PRs for the selected period and PR filters.

   Use the same period, timezone, and PR filters to compare values with the Builder portal. The reporting timezone belongs to your administrator account; viewing a developer never changes their saved timezone. Comparisons retain that developer's team and organization visibility rules. Classification actions use your administrator permissions.

   For Team, read the four compact cards for productivity points, Gateway requests, peak concurrent sessions and merged PRs. Each thumbnail covers the selected period. Points per active contributor sits beside the active count and full roster count. Use Highest first or Lowest first and the independent page controls in either user chart to explore all members.

   For report request fields, signed pages, and snapshot invalidation, see the Team productivity API reference.

   In a Team report, select Explore activity and outcomes to open Relationships on the accepted report. Choose Across members or Over time, activity and delivery metrics, a retained period, and evidence quality. Over-time views also offer an outcome lag in weeks. The chart shows all valid pairs; the exact-pair table pages independently. Select Inspect pair to read both date ranges and coverage. Quiet known zeros remain distinct from missing evidence, and excluded pairs remain visible.

   Read the full-population count, exclusions, and Spearman association before interpreting the chart. Requests and five-minute session peaks are activity proxies; association does not establish causation or individual performance. Selector changes retain the same Team report. Select Refresh to capture new evidence; this closes the prior relationship view. Access or source changes can invalidate the accepted report and require Refresh.

   Compare the four stacked time series over the same dates: productivity points, merged PRs, Gateway requests and five-minute peak sessions. Expand a chart or open its interval data to inspect values and coverage. The overview uses Gateway activity; PR filters apply only to delivery measures. There is no separate activity source, work-type or trend-period selector.

   Search, sort or page through the member table; members without observed activity remain visible. Open Output or Spending detail for highest and lowest eligible members, or open score, review, session and cost evidence. Use Team peer evidence within Output detail to inspect the comparison population and exclusions. Missing or incomplete evidence can leave standing unavailable while the team's own history remains visible.

   If AI activity is rebuilding, the Team report keeps its roster and available PR output visible. Requests, session peaks, costs and AI participation remain unavailable. After recovery, select Refresh to capture AI evidence with a new report. Existing pages and details do not fill in recovered activity automatically.

   In Investment, choose Source participation to compare observed successful LLM activity across the whole roster and among PR-active contributors. Each row shows its participant count and denominator. Missing prices do not erase observed participation. Both rows show a percentage when source and identity coverage qualify the comparison; otherwise they retain observed counts and withhold the percentage. Gateway participation excludes failed, blocked and tool-only requests. Observer participation uses authenticated completion events.

   Choose Observer investment to open separately captured Observer costs, price and identity coverage, models, providers, ratios and source participation. The drawer also offers Observer spending rankings. Its source, dates and capture time remain visible. Closing before the first evidence loads cancels that capture; reopening starts a new one. After evidence loads, closing and reopening retains it. Closing during a refresh keeps the previous evidence. Refresh Observer evidence intentionally captures new evidence. Expiry keeps the captured totals visible with a refresh notice and pauses further exploration until you select Refresh Observer evidence. Source changes on requested reads show a refresh prompt. Retrying a temporary detail failure retains its capture. Unavailable or expired Observer evidence does not remove the independently accepted Gateway overview. Observer current-rate estimates and Gateway recorded estimates are separate views; do not add them together or treat either as an invoice.

   Select a name or bar in either user chart to open that member's Developer report in a new tab with the same dates and compatible PR filters. The Team tab retains its selected charts and pages. Opening a member from the table also preserves compatible dates and PR filters. The navigation notice identifies dropped Team choices. Use Back to Team report to return. Compare and Evidence use an Outcomes context with the selected scope and compatible reporting choices, as the transition notice explains.

   Organization Overview shows four summary cards: productivity points, Gateway requests, five-minute peak sessions, and merged PRs. Four Top 5 panels rank developers, projects, teams, and repositories. One shared metric selector controls all four panels; choose productivity points, merged PRs, or PR-derived AI effectiveness. Select an entity to open its scoped report. Organization totals are calculated independently of these rankings; overlapping memberships do not make ranking totals additive.

   Below the organization summary, four aligned time series show productivity points, merged PRs, Gateway requests, and five-minute peak sessions. Expand a chart or inspect its interval data for coverage. Project Overview retains its productivity cards, contributor and repository rankings, time series, and repository table. Repository Overview retains the supported delivery and activity measures with their attribution qualifications.

   Period comparisons appear only when comparable evidence exists. Check coverage before interpreting a gap or a missing score.

3. Compare the full population or a selected pair.

   The global scope determines the comparison population. Organization can compare developers, teams, repositories, or projects. Team compares its own developers; Project compares its configured repositories; Repository compares developers using their contributions in that repository; Developer compares repositories or projects using that developer’s PR output. Highest and Lowest charts show 5, 10, or 20 entities; manual selection supports up to 20. Search, sorting, and pagination in the full table remain independent of the chart cohort.

   For a detailed pair, open Head-to-head, select two distinct entities of the same kind, and choose Compare all metrics. Overlay draws both series per metric; Side by side uses the same dates and value scale for each pair. Swap A/B to reverse the displayed difference without changing either underlying score.

   Repository, Project, and Developer comparisons show only PR-derived productivity points, merged PRs, and AI effectiveness. Organization comparisons of repositories or projects use the same restriction. Person-level requests, sessions, and parallel activity cannot be uniquely assigned to those contribution scopes and are omitted from their charts, tables, and CSV exports.

   Use Entity total for group output and Median observed user for typical observed activity. Read the contributor count: unobserved people are not recorded zeros.

4. Follow the evidence.

   Open Evidence for work mix, merged PRs, AI activity, and Related work. PR filters narrow the selected-period list by work type, score status, repository, project, and search. Open a PR for its score calculation, metadata, and provenance; use Previous PR / Next PR to inspect neighbors without discarding your filters.

   Unclassified remains distinct from an unavailable score. Work mix includes unclassified work and discloses incomplete scoring evidence.

5. Export or refresh the result.

   Outcomes keeps the displayed result stable. It does not automatically reload when you return to the browser tab, reconnect, or reach the result expiry time. Select Refresh data to load new observations. Changing the reporting selection or opening another view loads the requested data. If a result becomes invalid, Outcomes asks you to refresh instead of automatically replacing it. Completed Team, Project, and Repository reports stay visible across window focus, reconnection, and snapshot expiry. Expired reports keep their displayed values while exploration controls pause until refresh. Select Refresh to capture current evidence before continuing to explore an expired report. Requested reads still verify your access and the reporting context.

   Compare, Relationships, Work mix, Pull requests, and AI activity offer CSV exports of the full filtered result. Page size does not limit a download. Enable Chart entities only when you intentionally want that smaller population.

   Exports run asynchronously. Stop waiting abandons that export. If a result expires or its source changes, refresh before exporting again. Capacity errors require a narrower request; they do not return a partial success.

Open a Repository report

1. Open Productivity → Outcomes, then select Repository productivity and AI spending.
2. Use the Scope, Timeframe, and Timezone row. Choose all organization repositories, a configured Project, repositories touched by a Team, or manual repository selection. Team scope selects repositories with merged PRs by current primary-team members in the selected timeframe, then includes output from all contributors to those repositories. For a manual set, search by provider host, owner, name, or canonical ID, select across pages, then apply the selection. Up to 1,000 repositories are supported. Repository links open their actual GitHub destinations.
3. Read the four overview cards: merged PRs, productivity points, PR-active contributor identities with points per active contributor identity, and attributed estimated spend with its source label. Check the evidence status before interpreting partial or unavailable values. Review selected-period thumbnails where available and the top contributors by PRs and points. These values cover the complete selected scope. Ties and unavailable scores remain visible; these previews do not assign bands or percentiles. Select View contributors on either card to open its matching metric, then switch between merged PRs and points in the detail view.
4. Expand Filters and display options to choose a spending source, PR work types, trend window, and table columns. Use the contributor filter to select observed identities across the complete repository report. Search and page the choices; provider accounts remain selectable without a linked developer. This filter changes the report scope.
5. Choose one trend and its window. Use the contributor table to inspect observed output and attributed spend. Search, sorting, page size, and the PR-active or AI-active table population affect the displayed rows; they do not redefine the cards.
6. Open focused details for stored score evidence, received reviews, linked sessions, investment, or qualified rankings. When the report covers one verified repository, its GitHub link is also available inside the detail drawer. PR evidence links open the corresponding pull request on GitHub. In productivity-point details, choose Output concentration to inspect top-one and top-three point shares and author identity gaps. In received-review details, choose Review concentration to inspect human-review counts and reviewer identity gaps for the merged-PR cohort. Read missing-data and exclusion reasons before interpreting a value. See Repository report semantics.
7. Select Refresh to capture current evidence. Details and pages reuse the accepted report. If access, repository eligibility, or retained evidence changes, refresh or choose another eligible repository.

On small screens, navigation collapses to icons. The repository link and Refresh wrap onto separate lines when needed. Resizing keeps the selected report and its totals.

Correct PR work types

1. Open Productivity → Outcomes → Evidence → Pull requests.
2. Choose a different value in the PR's Work type menu. The change saves immediately. Each row shows Saving… and then Saved. Continue editing other rows without waiting for a report query; your page, filters, sorting, and visible rows stay in place.
3. Open Classification details to inspect the original work type, latest editor, and change time. Unclassified reasons distinguish missing, expired, stale, or ineligible assessments from an explicit Unclassified choice; retained assessment references remain visible. A manual choice takes precedence over future automated assessments. Choosing Unclassified also preserves a manual choice.

If a save conflicts with another change, inspect the refreshed value and choose again. If the response is lost, Retry save retries the same request. A work-type change does not mark the full assessment as human reviewed. Published reports remain unchanged.

The table shows Checking permissions… while loading the current work type and edit permission. An open PR drawer shows Loading current work type… until that read completes. These loading states also appear when the current classification refreshes, when you return to the page. After a save, only the edited row reloads its current classification. A saved-classifications notice marks points, matching counts, and report metrics as needing a refresh. Choose Refresh results when you finish editing. This loads new results for the displayed reporting period and returns the evidence list to its first page with the same filters and sorting. An edited PR can then leave the table if it no longer matches the filter. Recalculation is scheduled on the server for every accepted change; it does not depend on this button. Use Refresh activity in Builder or Refresh data in Outcomes to resolve a fresh reporting period.

A missing current value is shown as unavailable, rather than as Unclassified or an older report value.

If a save succeeds but the current value cannot be refreshed, the table shows the Saved selection read-only. Select Refresh to load the current work type and permissions. A retry may acknowledge an earlier accepted change, so the saved selection is not a claim that no later edit occurred.

The saved-classifications notice remains visible if the report refresh fails, including when the response arrives but contains errored metrics or coverage. Within the same account and scope, changing the period, global PR filters, or reporting timezone also retains the notice until the new report and saved PRs have been verified. Select Refresh results to retry; this does not submit the classifications again. A change accepted while a refresh is running still needs another refresh. If the reporting context expires or becomes invalid, use Refresh data in Outcomes or reload the Builder list to continue.

Manage developer classification permissions

1. Open Settings → Productivity → Manual PR classification.
2. Set Allow developers to classify unclassified PRs and Allow developers to reclassify classified PRs independently. Both are enabled by default and save immediately.

These controls apply to developers' own attributed PRs in the Builder Portal. Administrators can always classify PRs in their authorized productivity scope. Viewer accounts cannot edit work types.

If a refresh fails after a successful save, the last confirmed permissions stay visible. Select Refresh permissions to check for changes and enable editing again.

Inspect classification history

1. In the Outcomes PR evidence view, open Classification history. To inspect one PR, open its classification details and select View classification history.
2. Filter by change period, PR or repository text, actor email or ID, previous and new work types, or initial classification versus reclassification. The change period uses your browser timezone and is separate from the PR merge period.
3. Open a row for recorded editor and change details. With the keyboard, focus the PR name and press Enter. Use the separate external PR link to open the source pull request.

History shows retained Activity events, newest first. Refresh starts at the first page and keeps your filters. Older events follow your configured Activity retention policy; the PR retains its original and latest-change metadata.

note

Parallel activity is unavailable when retained session evidence cannot verify its denominator. Sparse PR evidence can also leave individual effectiveness intervals unavailable even when the full-period score exists. These states are not zeros and do not prove that no work occurred.

For scoring definitions, current team/project semantics, legacy views, and collection guidance, see Engineering productivity.

Shared reporting periods also include Last 12 complete weeks, Last year, Last 12 complete months, This quarter, and This year. Saved Outcomes links preserve these selections. Complete periods use civil calendar boundaries in the reporting timezone; in-progress periods end at the captured report cutoff. Availability still depends on retained source evidence.

Repository report fields and snapshot behavior are listed in the API reference. All-repository developer links change scope explicitly. Relationships is outside the shared Overview, Compare, and Evidence navigation. Existing Team report exploration and authorized historical relationship links remain available where their metrics preserve the selected scope.

Inspect pooled repository PR flow

In Repository Productivity, open received-review details and select PR flow. The table pools individual timings from the selected merged-PR cohort for opened-to-merged, opened-to-first-human-review and observed-ready-to-merged. It shows median, p75, valid PRs out of eligible merged PRs, and missing or invalid samples for each endpoint pair. The median uses the middle value or the mean of the middle pair; p75 uses nearest rank. Search and sorting do not change the underlying sample.

Missing timestamps and invalid intervals stay unobserved. A recorded zero-duration interval remains zero. Copilot and bot reviews never supply the first-human-review endpoint. Ready time is webhook-observed, rather than a claim about historical ready time. These durations describe merged PRs; they do not measure open backlog, deployment lead time or quality. The detail retains the accepted report's scope, filters, timezone and cutoff until Refresh.