Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
954b395
ci(e2e): run checks for settings integration PRs
Jul 20, 2026
bdd0a48
Add reusable Playwright E2E foundation (#5792)
BillLeoutsakosvl346 Jul 21, 2026
b0b0470
Add deterministic E2E settings personas (#5813)
BillLeoutsakosvl346 Jul 21, 2026
fc8a7c2
Harden Playwright foundation and persona fidelity (#5820)
BillLeoutsakosvl346 Jul 21, 2026
040963e
Add settings navigation acceptance contracts (#5837)
BillLeoutsakosvl346 Jul 22, 2026
029f501
Add settings authorization acceptance contracts (#5843)
BillLeoutsakosvl346 Jul 22, 2026
02e3f46
test(e2e): cover credential settings workflows (#5846)
BillLeoutsakosvl346 Jul 22, 2026
1afa5d3
test(e2e): add enterprise settings workflows (#5849)
BillLeoutsakosvl346 Jul 22, 2026
2d283c3
merge: sync settings Playwright with staging
Jul 22, 2026
1a631d9
test(e2e): cover enterprise integrations (#5868)
BillLeoutsakosvl346 Jul 23, 2026
beba0c2
test(e2e): finalize settings suite stabilization (#5885)
BillLeoutsakosvl346 Jul 23, 2026
72b749b
Merge staging into e2e/settings-playwright
Jul 23, 2026
7d38c4f
test(e2e): cover session policy settings
Jul 23, 2026
e60658d
chore(ci): update API validation route baseline
Jul 23, 2026
e37ad74
docs(e2e): add settings maintenance guidance (#5899)
BillLeoutsakosvl346 Jul 23, 2026
fea201b
fix(auth): serialize SSO configuration mutations
Jul 23, 2026
ee20cbc
chore(e2e): finalize staging handoff
Jul 23, 2026
41c1f24
fix(e2e): stabilize sandbox dependency resolution
Jul 24, 2026
417aed1
Merge staging into e2e/settings-playwright
Jul 24, 2026
32208b2
fix(ci): allow settings E2E setup overhead
Jul 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions .agents/skills/add-settings-e2e-test/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
---
name: add-settings-e2e-test
description: Add or update durable Playwright coverage for Sim settings using the correct literal contract or owning spec, existing personas, guarded orchestration, semantic assertions, cleanup, and secret-safe diagnostics. Use when asked to add a settings browser test, cover a settings regression, or update settings E2E acceptance behavior.
argument-hint: <settings-behavior-or-regression>
---

# Add Settings E2E Test

Add the smallest durable browser proof for the requested observable behavior.
Do not begin by copying a nearby test: first identify which acceptance dataset
or spec owns the behavior.

## Read first

1. `apps/sim/e2e/MAINTENANCE.md` — change type to contract/spec ownership.
2. `apps/sim/e2e/README.md` — orchestrator, project boundaries, focused commands,
diagnostics, acceptance and security boundaries, and CI policy.

Both files are canonical. This skill is the authoring procedure, not a second
copy of their inventories or commands. If this procedure ever conflicts with a
canonical document, the canonical document wins.

## Procedure

### 1. Define the observable contract

Write down:

- the user persona and resource plane (account, organization, or workspace);
- the action or direct URL;
- the exact observable result: path, semantic readiness, visible/hidden item,
access outcome, enabled/disabled control, warning, or persisted state;
- whether product behavior is intentionally changing or an implementation
change must preserve the existing contract.

Intended behavior changes update product code and literal acceptance
expectations together. Behavior-preserving fixes add or strengthen proof without
rewriting unrelated expectations.

### 2. Choose the owner before writing code

- **Canonical section, route, copy, readiness, or persona visibility:** update
`apps/sim/e2e/settings/navigation/contracts.ts`; its specs generate cases from
the literal dataset, including unauthenticated redirect cases.
- **Browser Back, app Back, direct-entry fallback, or return destination:** edit
`apps/sim/e2e/settings/navigation/history.spec.ts` directly.
- **Direct access, plan/role/entitlement gate, or mutation availability:** update
`apps/sim/e2e/settings/authorization/contracts.ts`; its access and mutation
specs generate cases from that dataset.
- If an existing navigation case already owns the sidebar proof, add or reuse
an `existingNavigationProofs` entry and set
`sidebar.existingProofId`. Do not execute the same sidebar assertion again
in the authorization case.
- **Unsaved-change behavior:** edit
`apps/sim/e2e/settings/authorization/unsaved-changes.spec.ts` directly.
- **Secrets or API key lifecycle:** edit the owning credential spec directly.
Change `apps/sim/e2e/settings/credentials/contracts.ts` only when reused
authorization proof IDs or boundaries change.
- **People, access control, SSO, data retention, or MCP lifecycle:** edit the
owning workflow spec. Change `apps/sim/e2e/settings/workflows/contracts.ts`
when stable lifecycle or cross-contract proof references change.
- **Persona or seeded relationship:** update
`apps/sim/e2e/settings/personas.ts` and the relevant
scenario/factory/integrity proof.

The categories are non-exclusive. Prefer extending an existing literal row or
owning workflow over adding a duplicate standalone test.

### 3. Use the owning fixture wrapper

- Browser-touching settings specs must not import runtime `test` or `expect`
from `@playwright/test`. Type-only imports such as `Page` or `Response` are
allowed.
- Navigation, authorization, persona-contract, and persona-isolation browser
specs use `apps/sim/e2e/fixtures/persona-test.ts`.
- Credential specs use the local `credential-test.ts`; workflow specs use the
local `workflow-test.ts`; authenticated or unauthenticated smoke and
browser-touching harness-level specs use
`apps/sim/e2e/fixtures/browser-test.ts`.
- Pure dataset-only contract-integrity specs and non-browser foundation policy
specs may use the Playwright test runner directly. If they begin creating a
browser, context, or page, move them to the owning wrapper first.
- Persona-based tests must create contexts through `contextForPersona` or an
owning helper that calls it. The inherited `page` and `context` fixtures on
`persona-test` are not persona-authenticated or network-guarded.
- `browser-test` guards its default context. Every manual
`browser.newContext()` must install `installBrowserNetworkGuard()` immediately,
then close the context and assert the guard even when the test fails; follow
the aggregate-cleanup pattern in authenticated smoke coverage. Do not use
`browser.newPage()`: it creates an implicit unguarded context; create and guard
an explicit context, then call `context.newPage()`.
- Reuse the owning directory's helpers. Credential tests use their local
`newPersonaPage` helper so the page is registered for failure sanitization.
Wrapper selection alone does not install every context, cleanup, attachment,
redaction, or artifact safeguard; bypassing the guarded fixture/helper path is
a test-safety bug.

### 4. Reuse the existing world

- Reuse an existing persona and seeded resource when it expresses the required
role, plan, entitlement, and ownership boundary.
- Add a persona or scenario edge only when no existing driver can prove the
contract without changing its meaning.
- Use run-namespaced factories and unique resources. Do not introduce shared
mutable fixture state.
- Keep sensitive fixture values browser-resident through the existing helpers.
The Playwright process must not receive database credentials, admin keys,
persona passwords, or plaintext values that the owning spec keeps in-page.

### 5. Assert semantics, not implementation details

- Use accessible roles, labels, names, and visible text.
- Assert exact paths and user-facing contract copy when those are the behavior
under test.
- Wait on semantic readiness, relevant same-origin responses, or explicit
authorization state—not arbitrary sleeps or CSS classes.
- For access controls, prove direct URL behavior as well as sidebar visibility.
- Do not import production navigation or authorization implementations to
generate expected values.

### 6. Make mutations reversible

- Register cleanup before the first mutation.
- Create uniquely named resources and restore the exact captured baseline.
- Use the production UI and same-origin APIs for behavior under test; use a
trusted database probe only where the existing suite defines one.
- Keep cleanup LIFO and safe after partial failure. Never weaken fixture
invariants merely to make a test pass.

### 7. Respect external and diagnostic boundaries

- Use the existing strict loopback Stripe and MCP fakes, the mail mock-success
path with provider credentials absent, and the other reviewed test
boundaries. An unexpected external request must fail rather than silently
egress.
- Credentials, tokens, certificates, verification values, storage state, and
sensitive response bodies may be used only through existing reviewed paths;
never log, attach, or retain them in diagnostics.
- The SSO workflow's public-certificate browser input is one reviewed example;
preserve its input clearing and diagnostic suppression.
- Preserve the owning spec's trace, screenshot, video, and attachment policy.
New suppression requires a security reason and corresponding safety coverage.

### 8. Keep unit and integrity proof aligned

- Update affected production unit tests for navigation, route gates,
authorization, billing, or business rules.
- Update the owning `contract-integrity.spec.ts` when adding IDs, references, or
a new contract axis.
- Preserve stable proof IDs; do not rerun a browser case merely because another
contract can reference its existing proof.

### 9. Verify through the orchestrator

- Run the affected unit and contract-integrity tests.
- Run the focused canonical project and path documented in
`apps/sim/e2e/README.md`.
- Invoke only `bun run test:e2e`; never invoke raw `playwright test`.
- Use `--reuse-build` and a single explicit project with `--no-deps` for local
iteration when the README permits it.
- Follow the retry policy in the canonical README and Playwright config. It is
currently zero; do not enable retries or change worker policy in a feature
test. Do not use `test.only`, unexplained skips, arbitrary browser sleeps, or
test-local environment bypasses.
- After focused proof passes, rely on the required complete CI suite. Run a
repeated stability gate only when the change's scope or acceptance plan calls
for one.

## Completion checklist

- [ ] Every observable facet has one clear owner, and all applicable owners were
updated.
- [ ] Allowed and denied outcomes exist where the feature is gated.
- [ ] Locators and readiness assertions are semantic and accessible.
- [ ] Mutations register cleanup first and restore the exact baseline.
- [ ] No secret or external-egress boundary was weakened.
- [ ] Related unit and contract-integrity tests are aligned and pass.
- [ ] Focused execution used the guarded orchestrator and complied with the
canonical retry policy.
139 changes: 139 additions & 0 deletions .agents/skills/reviewed-development/IMPLEMENTATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Implementation Mode

Implement the requested change completely, validate it in proportion to risk,
and subject the current result to repeated independent review before delivery.

## 1. Confirm the implementation contract

Read the accepted plan, current user request, relevant repository instructions,
and current branch state. Establish:

- Objective, scope, non-goals, and definition of done.
- Files and systems expected to change.
- Tests, rollout, documentation, and manual acceptance required.
- Whether commit, push, branch, or PR operations were explicitly authorized.
- Existing user changes that must be preserved.

If the plan has become stale, verify the current repository and adapt it
carefully. Escalate only decisions that materially change product behavior,
risk, or destructive outcomes.

## 2. Re-ground in current code

Inspect the implementation paths and nearest established patterns before
editing. Do not rely solely on summaries or the plan's file list. Check for
changes made since planning, hidden coupling, generated artifacts, migration
rules, and repository-specific validation commands.

Create a concise execution checklist for substantial work. Implement in
coherent phases, keeping only one phase actively changing the repository at a
time.

## 3. Implement the smallest complete solution

- Follow existing architecture, naming, contracts, and ownership boundaries.
- Address root causes instead of suppressing failures or adding broad
workarounds.
- Keep security, authorization, data integrity, secret handling, and cleanup
fail-closed.
- Preserve backward compatibility and safe rollout ordering where versions can
overlap.
- Avoid unrelated cleanup, speculative abstraction, duplicate helpers, and
infrastructure without a demonstrated need.
- Update tests and documentation alongside the behavior they protect.

Do not weaken invariants or tests merely to obtain a passing result.

## 4. Verify continuously

After each meaningful phase, run the cheapest relevant proof. Before review,
run the complete validation justified by the change, such as:

- Focused unit and integration tests.
- Type checking, linting, formatting, generated-file, and boundary checks.
- Migration, deployment, or compatibility verification.
- Browser or end-to-end tests through the project's supported orchestrator.
- Manual checks that cannot be automated.

Read failures as evidence. Fix introduced problems, distinguish unrelated
baseline failures explicitly, and do not report a check as passed unless it
actually ran successfully.

## 5. Self-review the current diff

Before commissioning reviewers:

- Compare the implementation with every plan requirement.
- Inspect the full diff, including generated and staged files.
- Check edge cases, negative paths, concurrency, partial failure, and cleanup.
- Confirm tests prove observable behavior rather than implementation details.
- Remove accidental duplication, debugging output, dead code, and unnecessary
complexity.
- Verify no credentials, local artifacts, or unrelated user changes entered the
diff.

## 6. Commission independent reviews

Generate two complete, neutral prompts from the current task, plan, repository,
implementation, diff, and verification evidence according to the
independent-review protocol. Launch two fresh reviewers concurrently when
possible.

Do not ask reviewers to validate the parent's preferred approach. Ask for their
genuine assessment of correctness, security, pattern fit, test quality,
maintainability, redundancy, overengineering, operational safety, and complete
coverage of the user's objective.

Reviewers must inspect the repository directly and remain read-only.

## 7. Triage, fix, and re-verify

Independently verify every finding:

- Fix valid issues with the smallest clean solution that matches repository
patterns.
- Push back on false positives or harmful suggestions with concrete evidence.
- Resolve contradictions by examining code and requirements, not by choosing
the more confident reviewer.
- Re-run all checks affected by substantive changes.
- Resume each reviewer separately with the current implementation and the
response to its findings. Continue until it accepts the resolution or the
protocol's unresolved-disagreement stop condition applies.

Do not tell the other reviewer or reviewers in later cycles what an earlier
agent found.

## 8. Re-review the current implementation

Generate fresh prompts describing only the current objective, code, diff, and
evidence. Begin another independent review cycle with an entirely new pair.
Never reuse reviewers from an earlier cycle.

If this review causes substantive edits, reviewers disagree materially, or
meaningful uncertainty remains, run one additional fresh pair after
re-verification, subject to the protocol's three-cycle cap. The final accepted
review must apply to the implementation being delivered.

## 9. Delivery gate

Do not declare completion until:

- The implementation satisfies the current plan and definition of done.
- Relevant automated checks pass, with manual-only checks identified.
- No Critical, High, or Medium concern remains unresolved without an explicit,
evidence-backed decision.
- The final substantive revision has been independently reviewed.
- The diff is scoped, clean, and free of secrets or temporary artifacts.

Commit, push, or create a PR only if the user's request authorized those
mutations. Follow repository-specific git and PR procedures rather than
inventing a generic release flow.

Report concisely:

- What changed and why.
- Verification performed and outcomes.
- Material issues found and corrected.
- Evidence-backed pushbacks.
- Remaining manual actions or known limitations.
- Commit, branch, or PR details when delivery was authorized.
Loading
Loading