Exclude intentional credit-guardrail tests from prod-main health rollups by Copilot · Pull Request #47263 · github/gh-aw

Copilot · 2026-07-22T10:04:09Z

Daily Credit Limit Test and Daily Max AI Credits Test are guardrail stress tests designed to fail, depressing the fleet-wide success rate by ~1.5pts daily (92.7% raw → 94.2% actual).

Frontmatter marker

Added intentional-failure: true to both workflow files and registered the field in main_workflow_schema.json. This is the canonical opt-out signal.

Go — detection & propagation

pkg/workflow/resolve.go — IsIntentionalFailure(workflowPath): reads local markdown frontmatter for the marker. Fails open (returns false) when the file isn't available locally.
pkg/cli/logs_report.go — RunData.IntentionalFailure bool populated per run; LogsSummary.IntentionalFailureRuns int aggregated. Agents consuming the logs MCP output can now filter with runs[].intentional_failure / summary.intentional_failure_runs.
pkg/cli/health_metrics.go — WorkflowHealth.IntentionalFailure bool; CalculateHealthSummary skips these workflows when computing HealthyWorkflows and BelowThreshold rollup counts (they still appear in the per-workflow table).
pkg/cli/health_command.go — displayHealthSummary populates IntentionalFailure per workflow from the first run's WorkflowPath.

Workflow prompt instructions

audit-workflows.md and deep-report.md both gain an explicit section instructing agents to exclude intentional_failure runs from fleet success-rate rollups and to report the adjusted rate alongside the raw one (e.g. "92.7% raw (94.2% excl. intentional failures)").

Tests

TestIsIntentionalFailure — covers .lock.yml path, .md path, bare ID, normal workflow (false), missing file (fail-open), empty input.
TestCalculateHealthSummaryExcludesIntentionalFailure — verifies the two guardrail workflows don't affect HealthyWorkflows/BelowThreshold while still appearing in TotalWorkflows and the Workflows slice.

Fixes [deep-report] Exclude intentional credit-guardrail tests from prod-main health rollups #47099

Generated by 👨‍🍳 PR Sous Chef · gpt54 14.4 AIC · ⌖ 7.63 AIC · ⊞ 7K · ◷
_{Comment /souschef to run again}

- Add intentional-failure: true to daily-credit-limit-test.md and daily-max-ai-credits-test.md - Add intentional-failure field to main_workflow_schema.json - Add IsIntentionalFailure() helper in pkg/workflow/resolve.go - Add IntentionalFailure field to RunData and IntentionalFailureRuns to LogsSummary - Exclude intentional-failure workflows from CalculateHealthSummary rollup counts - Update audit-workflows.md and deep-report.md to exclude these workflows from success-rate rollups - Recompile both affected workflows - Add unit tests for IsIntentionalFailure and CalculateHealthSummaryExcludesIntentionalFailure Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-22T10:59:53Z

      "description": "Mark the workflow as private, preventing it from being added to other repositories via 'gh aw add'. A workflow with private: true is not meant to be shared outside its repository.",
      "examples": [true, false]
    },
+    "intentional-failure": {


@copilot move to features.intentional-failure in the front matter

Done in the latest commit. Moved intentional-failure: true under the features: block in both workflow files, updated IsIntentionalFailure in resolve.go to read features["intentional-failure"], updated the test fixture, and moved the schema definition from a top-level property into the documented features.properties object.

Copilot

Pull request overview

Adds an intentional-failure marker so expected guardrail failures can be excluded from fleet-health rollups.

Changes:

Registers and detects the new frontmatter marker.
Propagates exclusion metadata through logs and health summaries.
Updates audit prompts, guardrail workflows, generated locks, and tests.

Show a summary per file

File	Description
`pkg/workflow/resolve.go`	Detects intentional-failure frontmatter.
`pkg/workflow/resolve_test.go`	Tests marker detection.
`pkg/parser/schemas/main_workflow_schema.json`	Registers the marker.
`pkg/cli/logs_report.go`	Adds run and summary metadata.
`pkg/cli/health_metrics.go`	Excludes marked workflows from counts.
`pkg/cli/health_metrics_test.go`	Tests health exclusion.
`pkg/cli/health_command.go`	Propagates marker into health data.
`.github/workflows/deep-report.md`	Adds adjusted-rate instructions.
`.github/workflows/deep-report.lock.yml`	Updates generated metadata.
`.github/workflows/daily-max-ai-credits-test.md`	Marks expected failure.
`.github/workflows/daily-max-ai-credits-test.lock.yml`	Updates generated metadata.
`.github/workflows/daily-credit-limit-test.md`	Marks expected failure.
`.github/workflows/daily-credit-limit-test.lock.yml`	Updates generated metadata.
`.github/workflows/audit-workflows.md`	Adds rollup exclusion instructions.
`.github/workflows/audit-workflows.lock.yml`	Updates generated metadata.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 15/15 changed files
Comments generated: 3
Review effort level: Medium

Copilot · 2026-07-22T12:13:16Z

+		mdPath = filepath.Join(constants.GetWorkflowDir(), normalizedName+".md")
+	}
+
+	content, err := os.ReadFile(mdPath)


Fixed in c9e9350. health_command.go now skips IsIntentionalFailure entirely when config.RepoOverride != "" (remote repo). logs_report.go (buildLogsData) calls GetCurrentRepoSlug() once before the loop and skips classification when the run's Repository field is set and differs from the local slug — failing open in both cases.

Copilot · 2026-07-22T12:13:16Z

+		runData.IntentionalFailure = workflow.IsIntentionalFailure(runData.WorkflowPath)
+		if runData.IntentionalFailure {
+			intentionalFailureRuns++


Added TestBuildLogsDataIntentionalFailure in logs_report_test.go (c9e9350). It sets up a temp workflow dir with a features.intentional-failure: true fixture, runs buildLogsData with one marked and one unmarked run, and asserts runs[].intentional_failure and summary.intentional_failure_runs.

Copilot · 2026-07-22T12:13:16Z

+		if wh.IntentionalFailure {
+			// Exclude from rollup counts — these workflows are expected to fail.
+			continue
+		}


Fixed in c9e9350. The success branch now reads "All evaluated workflows above %.0f%% success threshold" to make clear that intentional-failure workflows are excluded from the count.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-07-22T11:39:37Z

@copilot merge main and recompile

gh-aw-bot · 2026-07-22T11:41:23Z

@copilot please run the pr-finisher skill, review the latest PR state, and address any remaining maintainer or bot feedback.

This branch is not cleanly mergeable yet (mergeStateStatus=DIRTY), so please first rebase or otherwise update the branch cleanly, then rerun checks.

Generated by 👨‍🍳 PR Sous Chef · gpt54 14.4 AIC · ⌖ 7.63 AIC · ⊞ 7K · ◷
_{Comment /souschef to run again}

…xclude-intentional-tests # Conflicts: # pkg/cli/health_metrics.go # pkg/cli/logs_report.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-22T11:54:08Z

@copilot merge main and recompile

Done. Merged main into this branch — resolved conflicts in health_metrics.go (combined DriverExitCount/AgentLogicFailureCount from main with IntentionalFailure from our branch) and logs_report.go (added FailureKind alongside IntentionalFailure) — then recompiled all 260 workflows successfully.

…fix misleading summary text Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-22T12:13:24Z

@copilot please run the pr-finisher skill, review the latest PR state, and address any remaining maintainer or bot feedback.

This branch is not cleanly mergeable yet (mergeStateStatus=DIRTY), so ...

Done. Addressed remaining review feedback in c9e9350:

Remote-repo safety (resolve.go thread): health_command.go skips detection when --repo is set; logs_report.go compares run repository against local slug before classifying.
Untested path (logs_report.go thread): Added TestBuildLogsDataIntentionalFailure test.
Misleading summary text (health_metrics.go thread): Changed to "All evaluated workflows above threshold".

All 260 workflows compile cleanly and unit tests pass locally.

Initial plan

9bd0a4c

Copilot AI assigned Copilot and pelikhan Jul 22, 2026

Copilot started work on behalf of pelikhan July 22, 2026 10:04 View session

Copilot AI linked an issue Jul 22, 2026 that may be closed by this pull request

[deep-report] Exclude intentional credit-guardrail tests from prod-main health rollups #47099

Closed

Copilot AI changed the title ~~[WIP] Tag intentional credit-guardrail tests to exclude from health rollups~~ Exclude intentional credit-guardrail tests from prod-main health rollups Jul 22, 2026

Copilot finished work on behalf of pelikhan July 22, 2026 10:33

Copilot AI requested a review from pelikhan July 22, 2026 10:33

pelikhan marked this pull request as ready for review July 22, 2026 10:59

Copilot AI review requested due to automatic review settings July 22, 2026 10:59

Copilot started reviewing on behalf of pelikhan July 22, 2026 10:59 View session

pelikhan reviewed Jul 22, 2026

View reviewed changes

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Copilot started work on behalf of pelikhan July 22, 2026 11:05 View session

Move intentional-failure to features.intentional-failure in frontmatter

f6ac389

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 22, 2026 11:19

Copilot AI requested a review from pelikhan July 22, 2026 11:19

Copilot started work on behalf of pelikhan July 22, 2026 11:40 View session

github-actions Bot mentioned this pull request Jul 22, 2026

[pr-sous-chef] PR Sous Chef run report #47158

Open

Merge remote-tracking branch 'origin/main' into copilot/deep-report-e…

d218fe6

…xclude-intentional-tests # Conflicts: # pkg/cli/health_metrics.go # pkg/cli/logs_report.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 22, 2026 11:54

Copilot started work on behalf of gh-aw-bot July 22, 2026 11:55 View session

Address review feedback: remote-repo safety, add buildLogsData test, …

c9e9350

…fix misleading summary text Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 22, 2026 12:14

Copilot AI requested a review from gh-aw-bot July 22, 2026 12:14

Merge branch 'main' into copilot/deep-report-exclude-intentional-tests

e6c16ac

pelikhan merged commit 9df505e into main Jul 22, 2026
24 checks passed

pelikhan deleted the copilot/deep-report-exclude-intentional-tests branch July 22, 2026 12:37

github-actions Bot mentioned this pull request Jul 22, 2026

Smoke Test: MAI-Code-1-Flash - 29923975843 #47316

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Exclude intentional credit-guardrail tests from prod-main health rollups#47263

Exclude intentional credit-guardrail tests from prod-main health rollups#47263
pelikhan merged 6 commits into
mainfrom
copilot/deep-report-exclude-intentional-tests

Copilot AI commented Jul 22, 2026 •

edited by github-actions Bot

Loading

Uh oh!

pelikhan Jul 22, 2026

Uh oh!

Copilot AI Jul 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 22, 2026

Uh oh!

Copilot AI Jul 22, 2026

Uh oh!

Copilot AI Jul 22, 2026

Uh oh!

pelikhan commented Jul 22, 2026

Uh oh!

gh-aw-bot commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Jul 22, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Frontmatter marker

Go — detection & propagation

Workflow prompt instructions

Tests

Uh oh!

pelikhan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Copilot AI Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Jul 22, 2026

Uh oh!

gh-aw-bot commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

Copilot AI commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 22, 2026 •

edited by github-actions Bot

Loading