diff --git a/.changeset/drizzle-encrypted-indexes.md b/.changeset/drizzle-encrypted-indexes.md index d9f39025..9af771af 100644 --- a/.changeset/drizzle-encrypted-indexes.md +++ b/.changeset/drizzle-encrypted-indexes.md @@ -2,7 +2,7 @@ '@cipherstash/stack-drizzle': minor --- -New `encryptedIndexes` helper on the `/v3` entry: spread +New `encryptedIndexes` helper on the package root: spread `...encryptedIndexes(t)` in `pgTable`'s third-argument callback and it derives the recommended functional indexes for every encrypted column in the table — named `__`, tracked by `drizzle-kit generate` like diff --git a/.changeset/dynamodb-v2-read-table-forwarding.md b/.changeset/dynamodb-v2-read-table-forwarding.md new file mode 100644 index 00000000..82918224 --- /dev/null +++ b/.changeset/dynamodb-v2-read-table-forwarding.md @@ -0,0 +1,27 @@ +--- +'@cipherstash/stack': patch +--- + +Fix `encryptedDynamoDB`'s EQL v2 read path, which failed against a v3-configured +client — the case the v2 read support exists for. + +`decryptModel` and `bulkDecryptModels` forwarded the table to the encryption +client unconditionally. The second argument is only meaningful to the typed EQL +v3 client, which resolves it against its own reconstructor map; a legacy v2 +table is not in that map, so the call failed with: + +``` +[eql/v3]: decryptModel received a table this client was not initialized with +``` + +That contradicted the adapter's documented contract — writes are EQL v3 only, +but decrypt keeps accepting a v2 table so previously stored items stay readable. +In practice the failure hit exactly the customers the compatibility promise was +written for: upgraded to a v3 schema, with v2 items still in the table. + +The table is now forwarded only when it is an EQL v3 table. A v2 table takes the +table-less form, and the client derives the table from the payloads themselves. + +Both paths are covered by a credential-free test asserting what the adapter +forwards, so a regression no longer depends on a live-credential integration run +to surface. diff --git a/.changeset/eql-v3-sole-docs.md b/.changeset/eql-v3-sole-docs.md index 1e1a7202..540d34a1 100644 --- a/.changeset/eql-v3-sole-docs.md +++ b/.changeset/eql-v3-sole-docs.md @@ -5,12 +5,22 @@ Docs: EQL v3 is now the sole documented approach. The `stash-encryption`, `stash-drizzle`, and `stash-supabase` skills and the `@cipherstash/stack` -README teach only the v3 typed surface (`EncryptionV3`, `types.*` concrete -domains, `@cipherstash/stack-drizzle/v3`, `encryptedSupabaseV3`); EQL v2 -shrinks to one short Legacy section per document. Two explicit exceptions are -called out: DynamoDB still requires the v2 schema surface (#657), and the -encrypt rollout tooling (`stash encrypt backfill`/`cutover`, -`@cipherstash/migrate`) currently targets v2 columns (#648) — its guidance is -kept under a version callout. Also corrects the legacy `@cipherstash/drizzle` -README's pointer to the removed `@cipherstash/stack/drizzle` subpath (now the -separate `@cipherstash/stack-drizzle` package). +README teach only the v3 typed surface (`Encryption`, the `types.*` concrete +domains, the `@cipherstash/stack-drizzle` package root, `encryptedSupabase`); +EQL v2 shrinks to one short Legacy section per document. Two places keep more +than a Legacy section, because EQL v2 is still reachable there: + +- **DynamoDB reads.** `encryptedDynamoDB` writes EQL v3 only, but `decryptModel` + / `bulkDecryptModels` still accept an EQL v2 table so previously stored v2 + items stay readable, so the `stash-dynamodb` skill keeps the v2 schema shape + documented for that read path (#657). +- **The encrypt rollout lifecycle.** `stash encrypt *` and `@cipherstash/migrate` + detect a column's generation from its Postgres domain type, with EQL v3 as the + recognised and default generation; a legacy `eql_v2_encrypted` column does not + classify and falls through to the v2 lifecycle, which ends in + `stash encrypt cutover` rather than the v3 `stash encrypt drop`. That + difference is kept under a version callout (#648). + +Also corrects the legacy `@cipherstash/drizzle` README's pointer to the removed +`@cipherstash/stack/drizzle` subpath (now the separate `@cipherstash/stack-drizzle` +package). diff --git a/.changeset/init-placeholder-eql-v3.md b/.changeset/init-placeholder-eql-v3.md index b82cf31c..6be0fe42 100644 --- a/.changeset/init-placeholder-eql-v3.md +++ b/.changeset/init-placeholder-eql-v3.md @@ -17,3 +17,7 @@ the concrete-domain `types.*` factories (`types.TextSearch`, `types.IntegerOrd`, `types.Text`, `types.Json`, …), and the `@cipherstash/stack-drizzle/v3` entry (`extractEncryptionSchemaV3`) for Drizzle. The `encryptionClient` export shape and the empty-schema "no schemas yet" error path are unchanged. + +**Superseded later in this release** by the `@cipherstash/stack-drizzle` EQL v2 +removal: the scaffolds now emit `Encryption` from `@cipherstash/stack/v3` and +`extractEncryptionSchema` from the collapsed `@cipherstash/stack-drizzle` root. diff --git a/.changeset/one-sided-eql-detection-docs.md b/.changeset/one-sided-eql-detection-docs.md new file mode 100644 index 00000000..0da0531a --- /dev/null +++ b/.changeset/one-sided-eql-detection-docs.md @@ -0,0 +1,31 @@ +--- +'stash': patch +'@cipherstash/migrate': patch +'@cipherstash/stack': patch +--- + +Correct shipped documentation that claimed the tooling detects a column's EQL +**v2** generation. It does not, and has not since `classifyEqlDomain` dropped v2: +detection is one-sided — a `public.eql_v3_*` Postgres domain classifies as **v3**, +and anything else (a plaintext column, or a legacy `eql_v2_encrypted` one) +classifies as *unknown* and falls through to the **v2** lifecycle. The v2 path is +reached by fallback, not by detection, and a v2 column records no `eqlVersion` in +`.cipherstash/migrations.json`, so `stash encrypt status` reports no version for +it. + +- `skills/stash-supabase/SKILL.md` said the CLI "still auto-detects a v2 column" + (twice, once inside the "Stay on v2 for now" bullet — exactly the case it got + wrong) and that `stash encrypt drop` picks its target from a version the CLI + "auto-detects". All three now describe the one-sided rule, matching the correct + wording already in the same file's EQL version note. This skill is copied into + customer repos by `stash init`, so the wrong version of it was being installed + as guidance. +- `packages/migrate/README.md` documented `detectColumnEqlVersion(client, table, + column)` as returning `2`, `3`, or `null`. It cannot return `2` — the return + type is now stated as `3` or `null`, with what a `null` means for the caller. + The lifecycle intro no longer presents the v2 ladder as a detection result. +- `packages/stack/README.md`'s Supabase example imported and called + `encryptedSupabaseV3`, the `@deprecated` alias, contradicting the same file's + package table and v3-only note. It now uses `encryptedSupabase`. + +Documentation only — no behaviour change. diff --git a/.changeset/prisma-next-v3-client-config.md b/.changeset/prisma-next-v3-client-config.md new file mode 100644 index 00000000..8d17b9a5 --- /dev/null +++ b/.changeset/prisma-next-v3-client-config.md @@ -0,0 +1,32 @@ +--- +'@cipherstash/prisma-next': major +--- + +**Breaking:** `CipherstashFromStackV3Options.encryptionConfig` — the config +passed through to the encryption client by `cipherstashFromStack` — is narrowed +from `ClientConfig` to `V3ClientConfig` (`ClientConfig` without the legacy +`eqlVersion: 2` escape hatch). Forcing EQL v2 no longer type-checks: + +```ts +const cipherstash = await cipherstashFromStack({ + contractJson, + encryptionConfig: { eqlVersion: 2 }, + // ^^^^^^^^^^ error TS2322: Type '2' is not assignable to type '3'. +}) +``` + +The option never did what it looked like it did. This package is EQL v3 only, +and `eqlVersion: 2` selects `Encryption`'s nominal (untyped) overload at +runtime — not the `TypedEncryptionClient` that `cipherstashFromStack` returns as +`encryptionClient`. The field disagreed with the client you got back. + +**Migration:** drop the `eqlVersion` field. Every other `ClientConfig` option +(`workspaceCrn`, `clientId`, `clientKey`, `accessKey`, `authStrategy`, logging, +…) is unchanged and still accepted. + +To read legacy EQL v2 rows, decrypt through `@cipherstash/stack` rather than +asking this adapter for a v2 client: the decrypt path is generation-agnostic and +reads both v2 and v3 payloads. Use the returned `encryptionClient` — `decrypt(…)` +for a single value, or the no-table `decryptModel(row)` / `bulkDecryptModels(rows)` +form for whole models, which is the supported path for models written before the +v3 upgrade. diff --git a/.changeset/rewriter-never-drops-ciphertext.md b/.changeset/rewriter-never-drops-ciphertext.md new file mode 100644 index 00000000..b3577f8b --- /dev/null +++ b/.changeset/rewriter-never-drops-ciphertext.md @@ -0,0 +1,31 @@ +--- +'@cipherstash/wizard': patch +'stash': patch +--- + +Fix a data-loss bug in the Drizzle migration rewriter: a **commented-out** +`ALTER … SET DATA TYPE` was rewritten into executable SQL. The matcher was +comment-blind and the replacement is multi-line, so the author's `-- ` survived +on the first line only — the `DROP COLUMN` on the next line emitted live and +dropped a populated column. + +A statement is now left exactly as written whenever it is inert — inside a `--` +line comment, inside a `/* … */` block, or inside a single-quoted string +literal, where an `ALTER` is data rather than SQL. (Rewriting one splices +`--> statement-breakpoint` markers *inside* the literal, so splitting the file +the way drizzle's migrator does yields a bare, live `DROP COLUMN` as a chunk of +its own.) Quoting is tokenised properly in the process: a `--` inside a string +no longer opens a comment, an apostrophe inside a quoted identifier such as +`"o'brien_data"` no longer opens a phantom string literal, a doubled `''` or +`""` reads as an escape rather than a delimiter, and an unterminated quote of +either kind makes the rest of the file inert rather than live. + +The sweep also refuses to rewrite a column the migration corpus already gives an +encrypted type, so changing a column's encrypted domain no longer drops a column +full of ciphertext. Skipped statements report why they were left alone. + +An unreadable migration directory (`EACCES`) is reported rather than silently +treated as empty, and the wizard's `Run the migration now?` prompt defaults to No +whenever the sweep rewrote anything, flagged anything, or could not check a +directory at all — naming the directories that went unchecked, and making no +claim about data destruction for a directory nothing is known about. diff --git a/.changeset/skills-v3-lifecycle-honesty.md b/.changeset/skills-v3-lifecycle-honesty.md new file mode 100644 index 00000000..5808d215 --- /dev/null +++ b/.changeset/skills-v3-lifecycle-honesty.md @@ -0,0 +1,27 @@ +--- +'stash': patch +--- + +Correct the EQL v2/v3 rollout lifecycle in the bundled `stash-encryption`, +`stash-supabase` and `stash-drizzle` agent skills. Each described the **v2** +lifecycle as the unqualified default even though v3 is the default generation, +so an agent following the prose would run steps that do not apply — and, in one +case, expect the wrong column to be dropped. + +- `stash encrypt drop` was documented as removing `_plaintext`. That is the + **v2** target. On a v3 column there is no `_plaintext`: the command drops + the **original ``**, guarded by a `DO` block that takes `ACCESS EXCLUSIVE` + and re-counts unencrypted rows at apply time, raising instead of dropping if + any remain. Each step in the cutover table is now marked v2-only or v3, and the + drop preconditions (`cut-over` for v2, `backfilled` for v3) are stated. +- "The pending row will be promoted to active by `stash encrypt cutover`" was + false for v3, where cutover short-circuits before touching any configuration. + `stash db activate` is the only promotion path there. +- The CipherStash Proxy call-outs told every reader to run `stash db push`. + `db push`/`db activate` manage `eql_v2_configuration`, which EQL v3 does not + ship — on a v3-only database `db push` reports "Nothing to do." and exits 0, + and `db activate` errors. The call-outs are now scoped to the EQL v2 + Proxy + path. + +Skills ship inside the `stash` tarball and are copied into user projects at +`stash init`, so this guidance was being installed into customer repos. diff --git a/.changeset/stack-audit-on-decrypt.md b/.changeset/stack-audit-on-decrypt.md index dd21fd17..de4b34cd 100644 --- a/.changeset/stack-audit-on-decrypt.md +++ b/.changeset/stack-audit-on-decrypt.md @@ -32,10 +32,25 @@ already passing EQL v3 tables to plain `Encryption`, you now receive the typed client rather than the nominal one — its `decryptModel` / `bulkDecryptModels` return type changes, and the two-argument form reconstructs `Date` columns from `cast_as` instead of leaving them as ISO strings. Code that read those columns as -strings needs updating. As part of this collapse -`EncryptionV3` no longer independently pins the wire format — like `Encryption`, -it now honours an explicit `config.eqlVersion` (the retained migration escape -hatch). The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 +strings needs updating. + +The v3 overload takes a non-empty tuple of tables and a `V3ClientConfig` — +`ClientConfig` without the deprecated `eqlVersion` escape hatch. So +`Encryption({ schemas: [] })` no longer type-checks (it used to compile and then +throw), and `config: { eqlVersion: 2 }` selects the nominal overload, which is +the client you actually get back. Callers passing a plain `AnyV3Table[]` rather +than an array literal must narrow it to `readonly [AnyV3Table, ...AnyV3Table[]]`. +`Awaited>` names the nominal client whatever you +pass, because `ReturnType` reads the last overload; use the exported +`EncryptionClientFor` to name the client for a schema tuple. + +`decryptModel` / `bulkDecryptModels` on the typed client also accept a call with +no table, matching the runtime, which has always allowed it — that is the path +for reading models written before the upgrade, above all legacy EQL v2 ones, +whose table cannot be a member of a v3 schema tuple. Prefer the two-argument +form whenever the table is registered. + +The `eqlVersion` config field and the `@cipherstash/stack/schema` EQL v2 builders remain available (now marked `@deprecated`) for reading and migrating legacy v2 data; the client authors EQL v3 only. Their full removal is deferred to a later PR. diff --git a/.changeset/stack-dynamodb-v2-write-removal.md b/.changeset/stack-dynamodb-v2-write-removal.md index 0f5058e3..445bd372 100644 --- a/.changeset/stack-dynamodb-v2-write-removal.md +++ b/.changeset/stack-dynamodb-v2-write-removal.md @@ -3,8 +3,9 @@ --- **Breaking (DynamoDB adapter):** `encryptedDynamoDB(...).encryptModel` and -`bulkEncryptModels` no longer accept an EQL v2 table — write is EQL v3 only. The -v2 write type overloads have been removed, narrowing encrypt to `AnyV3Table`. +`bulkEncryptModels` no longer accept an EQL v2 table. The v2 write type overloads +have been removed, narrowing encrypt to `AnyV3Table`. The narrowing is +type-level — treat the type as the contract, not a runtime guard. **Decrypt still reads existing v2 items.** `decryptModel` / `bulkDecryptModels` continue to accept an EQL v2 table (`encryptedColumn` / `encryptedField` from diff --git a/.changeset/supabase-interface-row-types.md b/.changeset/supabase-interface-row-types.md new file mode 100644 index 00000000..688927d2 --- /dev/null +++ b/.changeset/supabase-interface-row-types.md @@ -0,0 +1,30 @@ +--- +'@cipherstash/stack-supabase': minor +--- + +Row-type generics now accept an `interface`, not just a `type` alias. + +`from()`, `returns()` and `single().returns()` constrained their row +parameter to `Record`. An `interface` has no implicit index +signature, so the most ordinary way to declare a row type failed to compile: + +```typescript +interface User { id: string; email: string } + +// before: TS2344 — Index signature for type 'string' is missing in type 'User' +// after: fine +const { data } = await supabase.from('users').select('id, email') +``` + +A `type User = { … }` alias worked, which is why the existing type tests never +caught it. The constraint is now `object`, which still rejects `string`/`number` +row types. upstream `postgrest-js` constrains `returns` to nothing at all, so +this brings the adapter in line with the API it mirrors rather than being +stricter than it. + +Also corrects the `EncryptedSingleQueryBuilder` documentation, which claimed +that "everything that only re-types or re-configures the pending request is +carried over" after `single()`/`maybeSingle()`. `overrideTypes` and `setHeader` +are not carried over — they have no adapter equivalent, and since +`single()`/`maybeSingle()` return the same builder instance rather than a +passthrough, calling them would fail at runtime, not just fail to typecheck. diff --git a/.changeset/supabase-single-row-typing.md b/.changeset/supabase-single-row-typing.md index eff4d8e0..7df5932f 100644 --- a/.changeset/supabase-single-row-typing.md +++ b/.changeset/supabase-single-row-typing.md @@ -21,12 +21,18 @@ awaits to `EncryptedSupabaseResponse` (`data: T | null`). That covers the zero-row case for `maybeSingle()` and the error case for both, so no separate null modelling was needed. -Filters and transforms are not chainable after `single()`/`maybeSingle()`, +Filters and transforms are no longer chainable after `single()`/`maybeSingle()`, matching supabase-js — applying one afterwards would change the query the -single-row promise was made about. `returns()` preserves the awaited shape, -so `.single().returns()` still awaits one row. +single-row promise was made about. `.single().eq(...)`, `.single().limit(...)` +and friends were previously accepted and are now compile errors. What only +re-types or re-configures the pending request is carried over: `returns()` +(preserving the awaited shape, so `.single().returns()` awaits one row), +`abortSignal()`, `throwOnError()`, `withLockContext()` and `audit()`. +`EncryptedSingleQueryBuilder` is exported so a stored builder can be +annotated. **Migration:** delete the cast. Code that worked around the old typing with `data as unknown as Row` (or read `data![0]`) should now use `data` directly; the cast still compiles but is no longer needed, and `data![0]` becomes a type -error. +error. Move any filter or transform chained after `single()`/`maybeSingle()` to +before it. diff --git a/.github/workflows/tests-bench.yml b/.github/workflows/tests-bench.yml index 5f1d8c5a..5b8da973 100644 --- a/.github/workflows/tests-bench.yml +++ b/.github/workflows/tests-bench.yml @@ -13,6 +13,10 @@ on: paths: - 'packages/bench/**' - 'packages/stack/src/eql/v3/**' + # The bench drives the typed v3 CLIENT, which lives in `encryption/`, not + # `eql/v3/` (that is the authoring DSL). Without this the job stayed green + # by never running for the changes most likely to break it. + - 'packages/stack/src/encryption/**' - 'packages/stack-drizzle/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' @@ -26,6 +30,10 @@ on: paths: - 'packages/bench/**' - 'packages/stack/src/eql/v3/**' + # The bench drives the typed v3 CLIENT, which lives in `encryption/`, not + # `eql/v3/` (that is the authoring DSL). Without this the job stayed green + # by never running for the changes most likely to break it. + - 'packages/stack/src/encryption/**' - 'packages/stack-drizzle/**' - 'packages/test-kit/**' - 'packages/cli/src/installer/**' diff --git a/docs/reference/supabase-sdk.md b/docs/reference/supabase-sdk.md index 7294a8eb..220ea773 100644 --- a/docs/reference/supabase-sdk.md +++ b/docs/reference/supabase-sdk.md @@ -10,6 +10,9 @@ One entry point, EQL v3 only: |---|---|---| | `encryptedSupabase` | `@cipherstash/stack/eql/v3` (EQL v3) | native `public.eql_v3_*` domains | +Rows already written as EQL v2 still decrypt through `@cipherstash/stack`; what +is gone is the ability to author new v2 columns here. + `encryptedSupabaseV3` remains as a `@deprecated`, type-identical alias. The old EQL v2 authoring wrapper — `encryptedSupabase({ encryptionClient, supabaseClient })` — has been removed; the name now binds to the v3 factory @@ -134,11 +137,11 @@ The domains use SQL-standard type names (`integer`, `smallint`, `real`, ### Install EQL ```bash -# v2 (default) +# v3 (the default) stash eql install --supabase -# v3 -stash eql install --eql-version 3 --supabase +# v2 (legacy installs only) +stash eql install --eql-version 2 --supabase ``` For **v2**, `--supabase` selects the opclass-stripped bundle (operator diff --git a/packages/cli/src/__tests__/rewrite-migrations.test.ts b/packages/cli/src/__tests__/rewrite-migrations.test.ts index 173cba0f..4f449965 100644 --- a/packages/cli/src/__tests__/rewrite-migrations.test.ts +++ b/packages/cli/src/__tests__/rewrite-migrations.test.ts @@ -507,6 +507,369 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped).toEqual([]) }) + // A multi-line replacement inherits the author's `-- ` on line 1 ONLY, so + // rewriting a commented-out ALTER turns lines 2+ — including DROP COLUMN — + // into live SQL. Commented SQL never runs; leave it exactly as written. + describe('commented-out statements', () => { + it.each([ + ['a line comment', '-- '], + ['an indented line comment', ' -- '], + ['a drizzle statement-breakpoint style prefix', '--> '], + ])('leaves an ALTER behind %s untouched', async (_label, prefix) => { + const original = `${prefix}ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` + const filePath = path.join(tmpDir, '0030_commented.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves an ALTER inside a block comment untouched', async () => { + const original = [ + '/* superseded by 0031', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_block.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('leaves an ALTER inside a NESTED block comment untouched', async () => { + const original = [ + '/* outer /* inner */', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_nested.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('does not report a commented-out near-miss', async () => { + const filePath = path.join(tmpDir, '0030_commented-using.sql') + fs.writeFileSync( + filePath, + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toEqual([]) + }) + + // The comment scan must not be fooled by `--` inside a string literal, or + // it would skip a live statement and leave broken SQL to fail at migrate. + it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + const filePath = path.join(tmpDir, '0030_literal.sql') + fs.writeFileSync( + filePath, + [ + `INSERT INTO "notes" ("body") VALUES ('a -- b');`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain( + 'ALTER TABLE "users" DROP COLUMN "email";', + ) + }) + + // An apostrophe inside a DOUBLE-QUOTED identifier is not a string + // delimiter. Reading it as one opens a phantom literal whose "closing" + // quote is the apostrophe in the SAME identifier further down the file — + // PAST the commented-out ALTER — so the scan concludes the ALTER is live + // and rewrites it into a real DROP COLUMN. The CREATE that declared the + // column always sits above the ALTER, so a real corpus produces exactly + // this shape. + it('leaves a commented-out ALTER untouched when an earlier identifier holds an apostrophe', async () => { + const original = [ + 'CREATE TABLE "users" (', + '\t"id" serial PRIMARY KEY NOT NULL,', + '\t"o\'brien_data" text', + ');', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "o\'brien_data" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0031_apostrophe.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // A statement inside a single-quoted literal is DATA, not SQL. Rewriting it + // splices `--> statement-breakpoint` markers INSIDE the literal, so + // splitting the file the way drizzle's migrator does yields a bare, live + // `ALTER TABLE ... DROP COLUMN ...;` as a chunk of its own. + it('leaves an ALTER inside a string literal untouched', async () => { + const original = [ + `INSERT INTO "audit_log" ("note") VALUES ('the reverted migration read:`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + `(do not run it again)');`, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0032_string-literal.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + expect(updated).not.toContain('--> statement-breakpoint') + }) + + // An UNTERMINATED quoted identifier must fail the same way an unterminated + // string literal does — by swallowing the rest of the file as inert. If it + // instead runs the scan cursor to the end, the loop exits and every + // commented-out ALTER below it is reported live and rewritten: the same + // destructive outcome as the apostrophe case above, one branch over. + it('leaves a commented-out ALTER untouched after an unterminated quoted identifier', async () => { + const original = [ + 'CREATE TABLE "users" ("id" serial PRIMARY KEY NOT NULL, "email" text);', + '--> statement-breakpoint', + 'SELECT "unclosed FROM users;', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_unterminated-identifier.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // Regression pin, not a bug fix — this already behaves. A commented-out + // ALTER in a CRLF file must come back byte-identical. + it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { + const original = [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\r\n') + const filePath = path.join(tmpDir, '0033_crlf.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + }) + + // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and + // unlike the plaintext case there is nothing left anywhere to backfill from. + describe('columns that are already encrypted', () => { + it('refuses to rewrite a domain change on a column created encrypted', async () => { + const create = path.join(tmpDir, '0000_create.sql') + fs.writeFileSync( + create, + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('refuses to rewrite a domain change on a column ADDed encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The + // column it renamed onto is encrypted, so a later domain change on it is + // just as destructive. + it('follows a RENAME from a previous sweep', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + it('reports the destructive statement once, not twice', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + [ + '-- Custom SQL migration file, put your code below! --', + '', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // The scoping matters: encrypting `contacts.email` must not be blocked by + // an unrelated `users.email` that happens to share a column name. + it('scopes the check to the table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_contacts.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "contacts" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The ordinary case this rewrite exists for: plaintext today, encrypted + // after the ALTER. Nothing to preserve, so rewrite it. + it('still rewrites a plaintext column created in an earlier migration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" (\n\t"id" integer PRIMARY KEY,\n\t"email" text\n);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A commented-out ADD never ran, so it says nothing about the live schema. + it('ignores an encrypted ADD COLUMN that is commented out', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + }) + + // `options.skip` excludes a file from being EDITED, not from describing the + // schema the other files are altering. + it('honours an encrypted column defined in the skipped file', async () => { + const skipPath = path.join(tmpDir, '0000_install.sql') + fs.writeFileSync( + skipPath, + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: skipPath, + }, + ) + + expect(rewritten).toEqual([]) + expect(skipped[0]?.reason).toBe('already-encrypted') + }) + }) + it('handles multiple ALTER statements in one file', async () => { const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v2_encrypted;', @@ -524,4 +887,25 @@ describe('rewriteEncryptedAlterColumns', () => { // Non-matching statement preserved expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') }) + + // Regression pin, not a bug fix — the matchers carry `/gi`, so a + // hand-lowercased migration is rewritten just like drizzle-kit's output. + it('rewrites a lowercase alter table ... set data type', async () => { + const filePath = path.join(tmpDir, '0034_lowercase.sql') + fs.writeFileSync( + filePath, + 'alter table "users" alter column "email" set data type eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') + expect(updated).not.toMatch(/set data type/i) + }) }) diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index 8b230552..60cfb0aa 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -34,7 +34,10 @@ import { detectSupabaseProject, type SupabaseProjectInfo, } from './detect.js' -import { rewriteEncryptedAlterColumns } from './rewrite-migrations.js' +import { + describeSkipReason, + rewriteEncryptedAlterColumns, +} from './rewrite-migrations.js' import { SUPABASE_EQL_MIGRATION_FILENAME, writeSupabaseEqlMigration, @@ -657,10 +660,11 @@ export async function generateDrizzleMigration( if (skipped.length > 0) { sweepIncomplete = true p.log.warn( - `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep could not rewrite automatically. Review and fix them before running your migrations:`, + `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep left alone. Review and fix them before running your migrations:`, ) - for (const { file, statement } of skipped) { + for (const { file, statement, reason } of skipped) { p.log.step(` - ${file}: ${statement}`) + p.log.step(` ${describeSkipReason(reason)}`) } } } catch (error) { diff --git a/packages/cli/src/commands/db/rewrite-migrations.ts b/packages/cli/src/commands/db/rewrite-migrations.ts index 966cd54d..550cb282 100644 --- a/packages/cli/src/commands/db/rewrite-migrations.ts +++ b/packages/cli/src/commands/db/rewrite-migrations.ts @@ -116,12 +116,242 @@ function trimStatementPreamble(statement: string): string { return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() } +/** + * True when the character at `index` is INERT — it sits inside a SQL comment (a + * `--` line comment, or a nestable block comment) or inside a single-quoted + * string literal. Either way it is not a statement the database will execute, + * so the rewrite must leave it exactly as written. + * + * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is + * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a + * commented-out `-- ALTER TABLE … SET DATA TYPE …;` therefore leaves the + * author's `-- ` prefix on line 1 only — lines 2+, including `DROP COLUMN`, + * become live executable SQL and destroy the column. Commented SQL is inert by + * definition, so the only correct move is to leave it exactly as written. + * + * **Why string literals count as inert too:** an ALTER quoted inside an + * `INSERT … VALUES ('…')` is DATA. Rewriting it splices `--> statement-breakpoint` + * markers INSIDE the literal, so splitting the file the way drizzle's migrator + * does yields a bare, live `ALTER TABLE … DROP COLUMN …;` as a chunk of its own. + * Escaping the injected text's own apostrophe (`@cipherstash/stack's`) would fix + * only the syntax error, not that live DROP COLUMN — so the statement is left + * as written, exactly like a commented one. + * + * Double-quoted identifiers are tokenised BEFORE `'` is considered: an + * apostrophe inside `"o'brien_data"` must not open a phantom literal, or the + * scan runs to the NEXT apostrophe — typically the same identifier in a + * commented-out ALTER further down — decides that ALTER is live, and rewrites + * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in + * an identifier) is an escape and does not close the token. + * + * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a + * comment/literal here, which can only make us skip a rewrite (the statement + * then fails loudly at migrate time), never perform a destructive one. + */ +function isInsideCommentOrString(sql: string, index: number): boolean { + let i = 0 + while (i < index) { + if (sql.startsWith('--', i)) { + const eol = sql.indexOf('\n', i) + if (eol === -1 || eol >= index) return true + i = eol + 1 + } else if (sql.startsWith('/*', i)) { + // Postgres block comments nest, so track depth rather than stopping at + // the first `*/` — a nested close would otherwise end the comment early + // and let the text after it read as live SQL. + let depth = 1 + let j = i + 2 + while (j < sql.length && depth > 0) { + if (sql.startsWith('/*', j)) { + depth += 1 + j += 2 + } else if (sql.startsWith('*/', j)) { + depth -= 1 + j += 2 + } else { + j += 1 + } + } + if (j > index) return true + i = j + } else if (sql[i] === '"') { + // A quoted identifier is live SQL, but its body is not: consuming it here + // — before the `'` branch below — is what stops an apostrophe inside one + // from opening a string literal that never really existed. + const end = endOfQuoted(sql, i, '"') + // An unterminated identifier swallows the rest of the file, so treat it + // as inert exactly like the unterminated literal below. Running the + // cursor to the end instead would exit the loop and report `false` — + // "live" — which is how the apostrophe bug destroyed a column in the + // first place, one branch over. + if (end > index) return true + i = end + } else if (sql[i] === "'") { + const end = endOfQuoted(sql, i, "'") + // Unterminated, or the literal runs past `index`: `index` is inside a + // string literal, which is every bit as inert as a comment. + if (end > index) return true + i = end + } else { + i += 1 + } + } + return false +} + +/** + * The index just past the `quote`-delimited token that opens at `open`, or + * `sql.length` when it is never closed. A doubled delimiter inside the token is + * an escaped one (`''`, `""`) and does not end it. + */ +function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { + let i = open + 1 + while (i < sql.length) { + if (sql[i] !== quote) { + i += 1 + } else if (sql[i + 1] === quote) { + i += 2 + } else { + return i + 1 + } + } + return sql.length +} + +/** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ +const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` + +/** + * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a + * delimiter so a bare domain cannot match a prefix of a longer identifier. + */ +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` + +/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +const ADD_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** `ALTER TABLE … RENAME COLUMN "a" TO "b"` — $1/$2 table, $3 from, $4 to. */ +const RENAME_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+RENAME COLUMN\s+"([^"]+)"\s+TO\s+"([^"]+)"`, + 'gi', +) + +/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ +const CREATE_TABLE_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, + 'gi', +) + +/** `"col" ` inside a CREATE TABLE body — $1 column. */ +const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** Splits a `TABLE_REF` capture pair into its schema and table halves. */ +function tableOf( + first: string, + second: string | undefined, +): { schema?: string; table: string } { + // When schema-qualified (`"app"."users"`) the first capture is the schema and + // the second is the table; otherwise the first is the table. + return second === undefined + ? { table: first } + : { schema: first, table: second } +} + +/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +function columnKey(table: string, column: string, schema?: string): string { + return JSON.stringify([schema ?? '', table, column]) +} + +/** + * Index every column the migration corpus gives an ENCRYPTED type, so the + * rewrite can tell the change it exists for (plaintext → encrypted) from one it + * must never touch (encrypted → encrypted). + * + * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. + * A column whose encrypted domain merely changes (`types.TextEq` → + * `types.TextSearch`) matches just as well as a plaintext column, and the + * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext + * left anywhere to backfill from, so unlike the plaintext case the data is not + * recoverable from the application at all. Changing an encrypted column's domain + * changes its index terms, so the data has to be re-encrypted through the client + * regardless; the sweep flags the statement and leaves it for the user. + * + * The index is corpus-wide rather than ordered by migration: over-detecting + * "encrypted" costs a flagged statement the user must handle by hand, while + * under-detecting costs irrecoverable ciphertext. Only comment-free statements + * count, for the same reason {@link isInsideCommentOrString} exists. + */ +function indexEncryptedColumns(contents: readonly string[]): Set { + const encrypted = new Set() + + for (const sql of contents) { + for (const created of sql.matchAll(CREATE_TABLE_RE)) { + if (isInsideCommentOrString(sql, created.index)) continue + const { schema, table } = tableOf(created[1], created[2]) + for (const column of created[3].matchAll( + CREATE_TABLE_ENCRYPTED_COLUMN_RE, + )) { + encrypted.add(columnKey(table, column[1], schema)) + } + } + + for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { + if (isInsideCommentOrString(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + encrypted.add(columnKey(table, added[3], schema)) + } + + // A rename carries the column's type with it — and `__cipherstash_tmp` + // renamed onto the real name is exactly what a previous sweep of this very + // directory emitted. Run after ADD so that tmp column is already indexed. + for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { + if (isInsideCommentOrString(sql, renamed.index)) continue + const { schema, table } = tableOf(renamed[1], renamed[2]) + if (encrypted.has(columnKey(table, renamed[3], schema))) { + encrypted.add(columnKey(table, renamed[4], schema)) + } + } + } + + return encrypted +} + +/** Why a recognised ALTER-to-encrypted statement was left alone. */ +export type SkipReason = + /** Outside the strict matcher — hand-authored `USING`, or an unknown form. */ + | 'unrecognised-form' + /** The column already holds an encrypted domain; rewriting drops ciphertext. */ + | 'already-encrypted' + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ file: string /** The offending statement, verbatim (trimmed), for the user to review. */ statement: string + /** Why it was left alone — the caller turns this into user-facing guidance. */ + reason: SkipReason +} + +/** + * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print + * next to the statement. Lives here so every caller says the same thing — the + * two reasons need very different action from the user, and a single generic + * "could not rewrite automatically" hides that. + */ +export function describeSkipReason(reason: SkipReason): string { + switch (reason) { + case 'already-encrypted': + return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" + case 'unrecognised-form': + return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + } } /** Outcome of a sweep: the files rewritten, and near-misses left for review. */ @@ -154,27 +384,58 @@ export interface RewriteResult { * which keeps both columns alive across deploys. Each rewritten file carries a * header comment saying exactly this. * - * Returns {@link RewriteResult}: the files rewritten, plus `skipped` near-misses - * — statements that look like an ALTER-to-encrypted but fall outside the strict - * matcher (a hand-authored `SET DATA TYPE … USING …;`, or a future drizzle-kit - * form). Near-misses are left untouched on disk and surfaced non-fatally so the - * caller can tell the user to review them, rather than silently shipping broken - * SQL. + * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements + * left for a human — ones outside the strict matcher (a hand-authored + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting + * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. + * Both are left untouched on disk and surfaced non-fatally so the caller can + * tell the user to review them, rather than silently shipping broken SQL or + * destroying data. Statements sitting inside a SQL comment — or inside a + * single-quoted string literal, where they are data rather than SQL — are inert + * and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, options: { skip?: string } = {}, ): Promise { - const entries = await readdir(outDir).catch(() => []) + const entries = await readdir(outDir).catch( + (error: NodeJS.ErrnoException) => { + // A missing directory is simply nothing to sweep. Anything else — EACCES + // above all — is a sweep that did NOT happen, and the caller reports it + // rather than letting the user believe their migrations were checked. + if (error.code === 'ENOENT') return [] as string[] + throw error + }, + ) const rewritten: string[] = [] const skipped: SkippedAlter[] = [] + const seen = new Set() + + /** Record a skip once — the strict pass and the broad scan can both find it. */ + const skip = (file: string, statement: string, reason: SkipReason): void => { + // Keyed on collapsed whitespace: the two passes trim the same statement + // by slightly different rules, and the strict pass runs first so its + // more specific reason is the one kept. + const key = `${file} :: ${statement.replace(/\s+/g, ' ')}` + if (seen.has(key)) return + seen.add(key) + skipped.push({ file, statement, reason }) + } - for (const entry of entries) { - if (!entry.endsWith('.sql')) continue + const sqlFiles = entries.filter((entry) => entry.endsWith('.sql')).sort() + const contents = new Map() + for (const entry of sqlFiles) { const filePath = join(outDir, entry) - if (options.skip && filePath === options.skip) continue + contents.set(filePath, await readFile(filePath, 'utf-8')) + } + + // Built from the WHOLE corpus, including `options.skip`: a column's current + // type comes from the migrations that ran before this one, not just the files + // we are allowed to edit. + const encryptedColumns = indexEncryptedColumns([...contents.values()]) - const original = await readFile(filePath, 'utf-8') + for (const [filePath, original] of contents) { + if (options.skip && filePath === options.skip) continue // Reset the regex's lastIndex — it's stateful on /g ALTER_COLUMN_TO_ENCRYPTED_RE.lastIndex = 0 @@ -187,11 +448,21 @@ export async function rewriteEncryptedAlterColumns( second: string | undefined, column: string, mangledType: string, + offset: number, ) => { - // When schema-qualified (`"app"."users"`) the first capture is the - // schema and the second is the table; otherwise the first is the table. - const schema = second === undefined ? undefined : first - const table = second === undefined ? first : second + // Commented-out SQL never runs, and a multi-line replacement would only + // inherit the `-- ` on its first line — leaving the rest live. + if (isInsideCommentOrString(original, offset)) return match + + const { schema, table } = tableOf(first, second) + + // Already encrypted: the ADD+DROP+RENAME would drop the ciphertext and + // there is no plaintext left to backfill from. Flag, never guess. + if (encryptedColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'already-encrypted') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. @@ -210,10 +481,16 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ - file: filePath, - statement: trimStatementPreamble(nearMiss[0]), - }) + const statement = trimStatementPreamble(nearMiss[0]) + // Anchor the comment test on the `SET DATA TYPE` itself: the match starts + // at the previous `;`, so its own offset sits before any preamble. + const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) + if ( + isInsideCommentOrString(updated, nearMiss.index + Math.max(keyword, 0)) + ) { + continue + } + skip(filePath, statement, 'unrecognised-form') } } diff --git a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts index f3cb0afc..b4e595a1 100644 --- a/packages/cli/src/commands/encrypt/lib/resolve-eql.ts +++ b/packages/cli/src/commands/encrypt/lib/resolve-eql.ts @@ -75,8 +75,7 @@ export async function resolveColumnLifecycle( * "no pending config", …). Since `classifyEqlDomain` recognises `eql_v3_*` * only, that case now also covers the post-cutover v2 state — `` was * renamed onto the ciphertext, and its `eql_v2_encrypted` domain is no longer - * classified, so the column never appears as a candidate. (It used to arrive - * here as a `version: 2` candidate and needed its own exemption.) + * classified, so the column never appears as a candidate. * * A non-empty candidate list therefore means EQL v3 columns exist but none is * identifiable — the caller must fail closed with this message rather than diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index 8923f747..c2adf39c 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -10,7 +10,10 @@ import { printNextSteps, SAFE_MIGRATION_NAME, } from '@/commands/db/install.js' -import { rewriteEncryptedAlterColumns } from '@/commands/db/rewrite-migrations.js' +import { + describeSkipReason, + rewriteEncryptedAlterColumns, +} from '@/commands/db/rewrite-migrations.js' import { detectPackageManager, execArgv, @@ -245,10 +248,11 @@ async function generateDrizzleEqlMigration( if (skipped.length > 0) { sweepIncomplete = true p.log.warn( - `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep could not rewrite automatically. Review and fix them before running your migrations:`, + `Found ${skipped.length} ALTER-to-encrypted statement(s) the sweep left alone. Review and fix them before running your migrations:`, ) - for (const { file, statement } of skipped) { + for (const { file, statement, reason } of skipped) { p.log.step(` - ${file}: ${statement}`) + p.log.step(` ${describeSkipReason(reason)}`) } } } catch (error) { diff --git a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md index 411114a0..7e640066 100644 --- a/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md +++ b/packages/cli/src/commands/init/doctrine/AGENTS-doctrine.md @@ -7,7 +7,7 @@ This document is the **durable rule book** for any agent working on this codebas ## What you are working with - **Encryption client** — a per-project module that defines which tables and columns are encrypted, what data type each column holds, and which search operations are enabled. The path is in `.cipherstash/context.json` under `encryptionClientPath`. -- **EQL extension** — a Postgres extension installed via `stash eql install` that provides server-side functions for searchable encryption (`eql_v2.*`). Required before any migration that creates encrypted columns. +- **EQL extension** — installed via `stash eql install`, providing the column domains and server-side functions for searchable encryption. `stash eql install` installs **EQL v3** by default: the `public.eql_v3_*` column domains and the `eql_v3.*` functions behind them. (A project set up before v3 may instead carry the legacy `eql_v2` schema and `eql_v2_encrypted` column type; both generations decrypt, but new columns are v3.) Required before any migration that creates encrypted columns. - **Project context** — `.cipherstash/context.json` records what `stash init` discovered: integration, package manager, env key names (never values), schemas, install command, CLI version. Treat it as authoritative. - **Action plan** — `.cipherstash/setup-prompt.md` is the project-specific to-do list for the current setup run. Read it first. @@ -18,7 +18,7 @@ This document is the **durable rule book** for any agent working on this codebas 3. **Never read or echo secrets.** Env key *names* (`CS_WORKSPACE_CRN`, `CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, `DATABASE_URL`) are fine to reference in code and docs. Their *values* are not. New env keys go in `.env.example` with placeholders; instruct the user to add the real value locally. Never read, `cat`, `grep`, or echo `~/.cipherstash/secretkey.json` (the development key), `~/.cipherstash/auth.json` (OAuth token and JWTs), anything under `~/.cipherstash/workspaces/`, or a value-bearing env file (`.env`, `.env.local`, `.env.production`, …). `.env.example` is the exception — it holds placeholders, not values, and you are expected to edit it. The CLI reads the credentials itself; no command needs you to open them. If a command fails on authentication, re-run `stash auth login` rather than inspecting the profile. 4. **Never invent CipherStash APIs.** If you don't know how a function is called, read the relevant skill (see below) — don't guess. The TypeScript types in `@cipherstash/stack` are the source of truth for what's callable. 5. **Never run database introspection yourself.** Don't run `psql`, `\d`, `pg_dump`, `supabase db dump`, or `drizzle-kit introspect`. The CLI already did this; the result is in `context.json`. If you need fresh introspection, ask the user to re-run `stash init`. -6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The `eql_v2` schema and `eql_v2_*` functions (CLI-managed; missing function ⇒ `stash eql upgrade`, not a hand-edit). +6. **Never modify these files.** `stash.config.ts` (generated by init — edits go in `.env`). `.cipherstash/` (CLI-owned). `~/.cipherstash/` (CLI-owned credentials — see invariant 3). The EQL schemas and functions installed by the CLI — `eql_v3`/`public.eql_v3_*` today, `eql_v2`/`eql_v2_*` on a legacy install (missing function ⇒ `stash eql upgrade`, not a hand-edit). 7. **`@cipherstash/stack` must be excluded from any bundler.** The package wraps a native FFI module (`@cipherstash/protect-ffi`) that cannot be bundled. The moment you `npm install @cipherstash/stack` in a project with a bundler, configure the exclusion *before* writing any code that imports it. Concretely: Next.js needs `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` in `next.config.{js,ts,mjs}`; webpack needs `externals` entries; esbuild needs `external`; Vite SSR needs `ssr.external`. Skipping this surfaces as `Cannot find module '@cipherstash/protect-ffi-*'` at runtime, often after the user has shipped to production. If you're declaring an encrypted column for the first time in a project, treat configuring this exclusion as part of the same change. ## Migrations — three phases, always reversible diff --git a/packages/cli/src/commands/init/lib/setup-prompt.ts b/packages/cli/src/commands/init/lib/setup-prompt.ts index 6380cacf..dd5ba0de 100644 --- a/packages/cli/src/commands/init/lib/setup-prompt.ts +++ b/packages/cli/src/commands/init/lib/setup-prompt.ts @@ -277,7 +277,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { 'Use when the column **does not yet exist** in the database (no plaintext predecessor to preserve). This is normal Drizzle / Supabase work plus the encryption client patterns from the integration skill.', '', "1. **If this is the first encrypted column in the project, configure the bundler exclusion first.** `@cipherstash/stack` cannot be bundled (it wraps a native FFI module). Next.js: add `serverExternalPackages: ['@cipherstash/stack', '@cipherstash/protect-ffi']` to `next.config.*`. Webpack: `externals`. esbuild: `external`. Vite SSR: `ssr.external`. Without this, the encryption client crashes at runtime with `Cannot find module '@cipherstash/protect-ffi-*'`. See the `stash-encryption` skill's Installation section for the full snippets.", - "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories for Drizzle, `encryptedColumn` for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", + "2. Edit the user's real schema file (`src/db/schema.ts` or wherever they keep it) to declare the new encrypted column. Use the patterns in the integration skill — the `types.*` domain factories from `@cipherstash/stack-drizzle` for Drizzle, and the `types.*` factories from `@cipherstash/stack/eql/v3` (via `encryptedTable`, passed as `schemas`) for Supabase. Encrypted columns must be **nullable `jsonb`** at creation time. Never `.notNull()`.", `3. Generate the schema migration${migration ? ` — \`${migration.generate}\` (${migration.tool})` : " using the project's existing migration tooling"}.`, `4. Show the user the generated SQL before applying${migration ? ` — \`${migration.apply}\`` : ''}.`, '5. Wire the column through the application code: insert paths encrypt before write, select paths decrypt after read, query paths use the right operator (`protectOps.eq`, etc. — see the integration skill).', @@ -303,7 +303,7 @@ export function renderImplementPrompt(ctx: SetupPromptContext): string { '#### Backfill and switch — after dual-writes are live', '', `3. **Backfill.** Run \`${cli} encrypt backfill --table --column \`. The CLI prompts the user (or accepts \`--confirm-dual-writes-deployed\` non-interactively) to confirm dual-writes are live, then chunks through the existing rows. Resumable; checkpoints to \`cs_migrations\` after every chunk. SIGINT-safe.`, - `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2:** update the schema file to declare the encrypted column under its final name (drop the twin suffix, switch \`\` to \`encryptedType\`), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`).`, + `4. **Switch reads to the encrypted column.** The step depends on the EQL version (\`${cli} encrypt backfill\` prints it; \`${cli} encrypt status\` shows it). **EQL v3 (the default):** there is no rename — update the schema and queries to read/write the encrypted column by its own name, and wire decryption through the encryption client. **EQL v2 (legacy data only):** update the schema file to declare the encrypted column under its final name (drop the twin suffix), then \`${cli} encrypt cutover --table --column \` runs the rename in one transaction (\`\` → \`_plaintext\`, twin → \`\`). The adapters no longer author v2 — \`@cipherstash/stack-drizzle\` removed \`encryptedType\` — so declare the column with a \`types.*\` v3 domain and reach v2 rows through \`@cipherstash/stack\`'s decrypt path.`, '5. **Wire the read path through the encryption client.** The read column now holds ciphertext. Read code paths must decrypt before returning the value to callers — `decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, or the equivalent `decrypt`/`bulkDecryptModels` calls. Without this step, your read paths return raw encrypted payloads to end users. The integration skill has the exact API.', '6. **Remove the dual-write code.** The plaintext column (still `` on v3; renamed `_plaintext` on v2) is no longer authoritative. Delete the dual-write logic from the persistence layer.', `7. **Drop.** Run \`${cli} encrypt drop --table --column \`. Generates a migration that removes the now-unused plaintext column (on v3 it first verifies no rows are still plaintext-only). Apply with the project's normal migration tooling.`, diff --git a/packages/cli/src/commands/init/steps/install-eql.ts b/packages/cli/src/commands/init/steps/install-eql.ts index f6c91246..31c9e5b9 100644 --- a/packages/cli/src/commands/init/steps/install-eql.ts +++ b/packages/cli/src/commands/init/steps/install-eql.ts @@ -104,13 +104,12 @@ export const installEqlStep: InitStep = { // --drizzle`) rather than routing through `eql install`. // // `eql install --drizzle` is v2-only — under the v3 default it rejects the - // flag outright, so init used to pin `eqlVersion: '2'` to get a migration - // at all. That pin made `stash init --drizzle` the one flow that provisions - // a v2 database while every other integration (and a bare `stash eql - // install`) gets v3, and it contradicted the stash-drizzle skill we install - // into the very same project — that skill documents the v3 surface - // (`types.*` domains, `Encryption`) and would have the user's agent - // author v3 code against a v2 database. + // flag outright, so routing Drizzle through it would provision a v2 + // database while every other integration (and a bare `stash eql install`) + // gets v3. That also contradicts the stash-drizzle skill installed into the + // very same project, which documents the v3 surface (`types.*` domains, + // `Encryption`) and would have the user's agent author v3 code against a v2 + // database. // // `stash eql migration --drizzle` (added in #691) closes that gap: v3 SQL, // still migration-first, and it bundles the `cs_migrations` tracking schema diff --git a/packages/migrate/README.md b/packages/migrate/README.md index 30d03a96..bd41337f 100644 --- a/packages/migrate/README.md +++ b/packages/migrate/README.md @@ -6,7 +6,7 @@ Backs the `stash encrypt` CLI command group, but also exported for direct use ## Lifecycle -Each column walks through these phases — the ladder depends on the column's EQL version (auto-detected from its Postgres domain type via `detectColumnEqlVersion`): +Each column walks through these phases — the ladder depends on the column's EQL version, and detection is one-sided: `detectColumnEqlVersion` recognises an `eql_v3_*` Postgres domain as **v3**, and everything else as *unknown* (`null`). The v2 ladder is the fallback for an unknown column, not a detection result: ```text EQL v2: schema-added → dual-writing → backfilling → backfilled → cut-over → dropped @@ -59,7 +59,7 @@ Thin wrappers around `eql_v2.rename_encrypted_columns()` (the **v2** cut-over pr ### `detectColumnEqlVersion` / `resolveEncryptedColumn` / `listEncryptedColumns` / `classifyEqlDomain` -The EQL types are self-describing, and these are the domain-type primitives everything version-specific above branches on. `detectColumnEqlVersion(client, table, column)` inspects one column's Postgres domain type and returns `2`, `3`, or `null` (not an EQL column); resolution is case-exact (quoted-identifier semantics, matching the rest of the pipeline) and honours `search_path`. `resolveEncryptedColumn(client, table, plaintextColumn, hint?)` finds a plaintext column's encrypted counterpart from the domain types — an explicit hint (e.g. the manifest's recorded `encryptedColumn`) wins, then the `_encrypted` convention, then the table's sole EQL column; the name is never assumed. `listEncryptedColumns` returns every EQL-domain column on a table, classified. +The EQL types are self-describing, and these are the domain-type primitives everything version-specific above branches on. `detectColumnEqlVersion(client, table, column)` inspects one column's Postgres domain type and returns `3` (an `eql_v3_*` domain) or `null` — it never returns `2`. `null` means *unknown*: a plaintext column, or a legacy `eql_v2_encrypted` one, is indistinguishable here, and callers fall through to the v2 lifecycle (a v2 column's version is carried by the manifest's recorded `eqlVersion`, if it has one). Resolution is case-exact (quoted-identifier semantics, matching the rest of the pipeline) and honours `search_path`. `resolveEncryptedColumn(client, table, plaintextColumn, hint?)` finds a plaintext column's encrypted counterpart from the domain types — an explicit hint (e.g. the manifest's recorded `encryptedColumn`) wins, then the `_encrypted` convention, then the table's sole EQL column; the name is never assumed. `listEncryptedColumns` returns every EQL-domain column on a table, classified. ### `countEncrypted` / `countUnencrypted` diff --git a/packages/prisma-next/src/stack/from-stack-v3.ts b/packages/prisma-next/src/stack/from-stack-v3.ts index 28ab7045..2f569b55 100644 --- a/packages/prisma-next/src/stack/from-stack-v3.ts +++ b/packages/prisma-next/src/stack/from-stack-v3.ts @@ -25,7 +25,7 @@ */ import type { AnyV3Table } from '@cipherstash/stack/eql/v3' -import type { ClientConfig } from '@cipherstash/stack/types' +import type { V3ClientConfig } from '@cipherstash/stack/types' import { EncryptionV3, type TypedEncryptionClient } from '@cipherstash/stack/v3' import type { SqlMiddleware, @@ -55,8 +55,14 @@ export interface CipherstashFromStackV3Options { */ readonly schemasV3?: ReadonlyArray - /** Pass-through to `EncryptionV3({ config })` (keyset overrides, logging, …). */ - readonly encryptionConfig?: ClientConfig + /** + * Pass-through to `EncryptionV3({ config })` (keyset overrides, logging, …). + * + * `V3ClientConfig`, not `ClientConfig`: this package is EQL v3 only, and the + * legacy `eqlVersion: 2` escape hatch returns the nominal (untyped) client at + * runtime, which is not what this entry point hands back. + */ + readonly encryptionConfig?: V3ClientConfig } export interface CipherstashFromStackV3Result { @@ -83,8 +89,11 @@ export async function cipherstashFromStack( ) } - const derived = deriveStackSchemasV3(opts.contractJson) - if (derived.length === 0) { + // Destructured rather than length-checked so the non-emptiness survives into + // the type layer: `Encryption` requires a non-empty schema tuple (an empty one + // used to type-check and then throw at runtime). + const [firstDerived, ...restDerived] = deriveStackSchemasV3(opts.contractJson) + if (firstDerived === undefined) { throw new Error( 'cipherstashFromStack: no v3 cipherstash columns found in contract.json. ' + 'Declare at least one v3 `cipherstash.*()` column (e.g. `cipherstash.TextSearch()`) in prisma/schema.prisma ' + @@ -92,7 +101,10 @@ export async function cipherstashFromStack( ) } - const schemas = resolveV3Schemas(derived, opts.schemasV3) + const schemas = resolveV3Schemas( + [firstDerived, ...restDerived], + opts.schemasV3, + ) const encryptionClient = await EncryptionV3({ schemas, @@ -132,15 +144,18 @@ function collectNonV3CipherstashCodecIds( return [...ids].sort() } +/** A schema list carrying its non-emptiness in the type, as `Encryption` wants. */ +type NonEmptyV3Schemas = readonly [AnyV3Table, ...AnyV3Table[]] + /** * Validate contract-declared tables against their overrides (exact * domain identity) and append override-only tables — same merge * semantics as the v2 `resolveSchemas`. */ function resolveV3Schemas( - derived: ReadonlyArray, + derived: NonEmptyV3Schemas, override: ReadonlyArray | undefined, -): ReadonlyArray { +): NonEmptyV3Schemas { if (override === undefined || override.length === 0) return derived const derivedByName = new Map(derived.map((t) => [t.tableName, t])) @@ -153,7 +168,8 @@ function resolveV3Schemas( } return [ - ...derived, + derived[0], + ...derived.slice(1), ...override.filter((t) => !derivedByName.has(t.tableName)), ] } diff --git a/packages/stack-drizzle/README.md b/packages/stack-drizzle/README.md index db8a1b45..60462d1b 100644 --- a/packages/stack-drizzle/README.md +++ b/packages/stack-drizzle/README.md @@ -46,7 +46,7 @@ const rows = await db .select() .from(users) .where(await ops.and( - ops.contains(users.email, 'alice'), // free-text containment over ciphertext + ops.matches(users.email, 'alice'), // free-text token match over ciphertext ops.between(users.age, 18, 65), )) .orderBy(ops.asc(users.age)) diff --git a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts index 9d0717ea..f1b81923 100644 --- a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts +++ b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts @@ -11,6 +11,7 @@ import { describe, expectTypeOf, it } from 'vitest' import { type EncryptedQueryBuilder, type EncryptedQueryBuilderV3, + type EncryptedSingleQueryBuilder, type EncryptedSupabaseResponse, encryptedSupabase, encryptedSupabaseV3, @@ -460,3 +461,111 @@ describe('canonical (unsuffixed) exports', () => { >() }) }) + +// --------------------------------------------------------------------------- +// single() / maybeSingle() +// +// The single-row builder is a DIFFERENT type from the array builder, so every +// method it does and does not carry is public API. Filters and transforms are +// deliberately absent — one applied after `single()` would change the query the +// single-row promise was made about. What stays available is exactly `then`, +// `abortSignal`, `throwOnError`, `returns`, `withLockContext` and `audit`; this +// is NOT parity with postgrest-js, whose `overrideTypes` and `setHeader` have no +// adapter equivalent. `returns()` in particular is documented in the +// changeset for this change and was unreachable from the public surface until +// now (#772 review, SB-1). +// --------------------------------------------------------------------------- + +/** A typed builder for the single-row assertions below. */ +type MixedRow = UserRow & { + tags: string[] + meta: Record + note: string +} + +describe('single-row builder surface', () => { + it('is exported so a stored builder can be annotated', () => { + expectTypeOf>().toMatchTypeOf< + PromiseLike> + >() + }) + + it('awaits to ONE row, not an array', async () => { + const { data } = await mixedBuilder.single() + expectTypeOf(data).toEqualTypeOf() + }) + + it('carries returns() with the single-row shape preserved', async () => { + const { data } = await mixedBuilder.single().returns() + expectTypeOf(data).toEqualTypeOf() + }) + + it('carries the encryption configurators, which are read at execute time', () => { + const builder = mixedBuilder.single() + expectTypeOf( + builder.withLockContext({ identityClaim: ['sub'] }), + ).toEqualTypeOf>() + expectTypeOf(builder.audit({ metadata: { a: 1 } })).toEqualTypeOf< + EncryptedSingleQueryBuilder + >() + }) + + it('does NOT carry filters or transforms', () => { + const builder = mixedBuilder.single() + // @ts-expect-error - a filter after single() would change the query + builder.eq('email', 'a@b.com') + // @ts-expect-error - a transform after single() would change the query + builder.limit(1) + // @ts-expect-error - single() is applied last + builder.single() + }) +}) + +// --------------------------------------------------------------------------- +// Row-type generics accept an `interface` +// +// DO NOT "simplify" `InterfaceRow` below into a `type` alias — the whole point +// of this block is the distinction. A `type` alias for an object literal gets an +// IMPLICIT INDEX SIGNATURE, so it satisfies `Record`; an +// `interface` does NOT, so an interface row type fails a +// `Row extends Record` constraint with TS2344 ("Index signature +// for type 'string' is missing"). Every other row-typed test in this file goes +// through `type UserRow = InferPlaintext` (line ~33) — an alias — +// which is exactly why none of them ever caught this. +// +// Interfaces are the ordinary way a Supabase user declares a row type (and what +// `supabase gen types` emits alongside its aliases), and upstream postgrest-js +// leaves the equivalent `returns()` type parameter entirely unconstrained, so +// the adapter must not be stricter than the API it mirrors. +// --------------------------------------------------------------------------- + +interface InterfaceRow { + id: string + email: string +} + +describe('row-type generics accept an interface (not just a type alias)', () => { + it('accepts an interface on the untyped instance from()', async () => { + const supabase = await encryptedSupabase(supabaseClient) + const { data } = await supabase.from('users').select('*') + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on the typed instance fallback from()', async () => { + const supabase = await encryptedSupabase(supabaseClient, { + schemas: { users }, + }) + const { data } = await supabase.from('orders').select('*') + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on returns()', async () => { + const { data } = await mixedBuilder.returns() + expectTypeOf(data).toEqualTypeOf() + }) + + it('accepts an interface on single().returns()', async () => { + const { data } = await mixedBuilder.single().returns() + expectTypeOf(data).toEqualTypeOf() + }) +}) diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index f8e929ce..23168ffe 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -1,7 +1,3 @@ -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' import type { QueryTypeName } from '@cipherstash/stack/types' import type { DbFilterString, @@ -11,16 +7,6 @@ import type { PendingOrCondition, } from './types' -/** - * Get the names of all encrypted columns defined in a table schema. - */ -export function getEncryptedColumnNames( - schema: EncryptedTable, -): string[] { - const built = schema.build() - return Object.keys(built.columns) -} - /** * Check whether a column name refers to an encrypted column in the schema. */ @@ -31,53 +17,6 @@ export function isEncryptedColumn( return encryptedColumnNames.includes(columnName) } -/** - * Parse a Supabase select string and add `::jsonb` casts to encrypted columns. - * - * Input: `'id, email, name'` - * Output: `'id, email::jsonb, name::jsonb'` (if email and name are encrypted) - * - * Handles whitespace, already-cast columns, and embedded functions. - */ -export function addJsonbCasts( - columns: string, - encryptedColumnNames: string[], -): DbSelect { - // The mapping below emits DB-space tokens; the brand is asserted once, here. - return columns - .split(',') - .map((col) => { - const trimmed = col.trim() - - // Skip empty segments - if (!trimmed) return col - - // If it already has a cast (e.g. `email::jsonb`), skip - if (trimmed.includes('::')) return col - - // If it contains parens (function call) or dots (foreign table), skip - if (trimmed.includes('(') || trimmed.includes('.')) return col - - // Check if the column name (possibly with alias) is encrypted - // Handle `column_name` or `column_name as alias` - const parts = trimmed.split(/\s+/) - const colName = parts[0] - - if (isEncryptedColumn(colName, encryptedColumnNames)) { - // Preserve original whitespace before the column - const leadingWhitespace = col.match(/^(\s*)/)?.[1] ?? '' - if (parts.length > 1) { - // Has alias: `email as e` -> `email::jsonb as e` - return `${leadingWhitespace}${colName}::jsonb ${parts.slice(1).join(' ')}` - } - return `${leadingWhitespace}${colName}::jsonb` - } - - return col - }) - .join(',') as DbSelect -} - /** * Resolve a select token to its DB column name, or `undefined`. * diff --git a/packages/stack-supabase/src/index.ts b/packages/stack-supabase/src/index.ts index 4dfec303..4710e6f5 100644 --- a/packages/stack-supabase/src/index.ts +++ b/packages/stack-supabase/src/index.ts @@ -253,6 +253,7 @@ export type { // Deprecated `*V3` aliases (Decision 5 — supabase keeps type-identical aliases). EncryptedQueryBuilderV3, EncryptedQueryBuilderV3Untyped, + EncryptedSingleQueryBuilder, EncryptedSupabaseError, EncryptedSupabaseInstance, EncryptedSupabaseOptions, diff --git a/packages/stack-supabase/src/query-builder.ts b/packages/stack-supabase/src/query-builder.ts index b2c9652e..d7356a1a 100644 --- a/packages/stack-supabase/src/query-builder.ts +++ b/packages/stack-supabase/src/query-builder.ts @@ -87,7 +87,7 @@ const warnedLikeDelegation = new Set() * {@link execute} below. */ export class EncryptedQueryBuilderImpl< - T extends Record = Record, + T extends object = Record, /** The shape this builder awaits to. `T[]` normally; narrowed to `T` by * {@link single}/{@link maybeSingle}, which return ONE row. Carried as a * parameter so the promise cannot keep advertising `T[]` after the runtime @@ -503,7 +503,7 @@ export class EncryptedQueryBuilderImpl< /** Re-type the ROW. The awaited SHAPE is preserved: called after * `single()`/`maybeSingle()` this still awaits one row, not `U[]`. */ - returns>(): EncryptedQueryBuilderImpl< + returns(): EncryptedQueryBuilderImpl< U, TData extends readonly unknown[] ? U[] : U > { diff --git a/packages/stack-supabase/src/query-encrypt.ts b/packages/stack-supabase/src/query-encrypt.ts index 56ac8d65..ae67e4bf 100644 --- a/packages/stack-supabase/src/query-encrypt.ts +++ b/packages/stack-supabase/src/query-encrypt.ts @@ -250,7 +250,6 @@ export async function encryptFilterValues( column, table: ctx.table, queryType, - returnType: 'composite-literal', }) termMap.push(mapping) } diff --git a/packages/stack-supabase/src/query-results.ts b/packages/stack-supabase/src/query-results.ts index 5838e012..128c06fc 100644 --- a/packages/stack-supabase/src/query-results.ts +++ b/packages/stack-supabase/src/query-results.ts @@ -80,10 +80,7 @@ function postprocessDecryptedRow( * than decrypted. To read v2 data, decrypt fetched rows with the core * `@cipherstash/stack` client, whose decrypt path is generation-agnostic. */ -export async function decryptResults< - T extends Record, - TData = T[], ->( +export async function decryptResults( result: RawSupabaseResult, ctx: DecryptContext, ): Promise> { diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index dd461127..6755b8b5 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -1,5 +1,4 @@ import type { AuditConfig } from '@cipherstash/stack/adapter-kit' -import type { EncryptionClient } from '@cipherstash/stack/encryption' import type { AnyV3Table, EqlTypeForColumn, @@ -7,11 +6,7 @@ import type { QueryTypesForColumn, } from '@cipherstash/stack/eql/v3' import type { EncryptionError } from '@cipherstash/stack/errors' -import type { LockContext, LockContextInput } from '@cipherstash/stack/identity' -import type { - EncryptedTable, - EncryptedTableColumn, -} from '@cipherstash/stack/schema' +import type { LockContextInput } from '@cipherstash/stack/identity' import type { ClientConfig } from '@cipherstash/stack/types' import type { V3Schemas } from './schema-builder' @@ -392,9 +387,8 @@ export interface EncryptedQueryBuilder< * union (which subsumes the encrypted column's `string`); the runtime resolves * the column and picks the encoding (and rejects the wrong-column-kind pairing). */ -export interface EncryptedQueryBuilderUntyped< - Row extends Record, -> extends EncryptedQueryBuilderCore< +export interface EncryptedQueryBuilderUntyped + extends EncryptedQueryBuilderCore< Row, StringKeyOf, EncryptedQueryBuilderUntyped @@ -429,7 +423,7 @@ export interface EncryptedQueryBuilderUntyped< /** Untyped instance (no `schemas`): rows default to `Record` * and `from` accepts any table name. */ export interface EncryptedSupabaseInstance { - from = Record>( + from>( tableName: string, ): EncryptedQueryBuilderUntyped } @@ -452,7 +446,7 @@ export interface TypedEncryptedSupabaseInstance { from( table: K, ): EncryptedQueryBuilder> - from = Record>( + from>( table: string, ): EncryptedQueryBuilderUntyped } @@ -465,15 +459,37 @@ export interface TypedEncryptedSupabaseInstance { * The builder returned by `single()`/`maybeSingle()`: awaits to a SINGLE row * (`data: T | null`) instead of an array. * - * Only the two post-hoc modifiers supabase-js also allows after `.single()` are - * carried over. Filters and transforms are deliberately absent — applying one - * after `single()` would change the query the single-row promise was made - * about. + * FILTERS and TRANSFORMS are deliberately absent — applying one after `single()` + * would change the query the single-row promise was made about. + * + * What IS carried is exactly: `then` (via `PromiseLike`), `abortSignal`, + * `throwOnError`, `returns`, `withLockContext` and `audit`. That is a + * hand-written list, not a passthrough — `single()`/`maybeSingle()` return the + * same builder instance, so a method absent here is absent at runtime too. + * + * It is therefore NOT parity with postgrest-js, which carries a different set: + * its `single()` returns a `PostgrestBuilder`, carrying + * `returns`/`overrideTypes`/`throwOnError`/`setHeader` (and NOT `abortSignal`, + * which lives on `PostgrestTransformBuilder`). Relative to that, this adapter + * keeps `abortSignal` as a deliberate superset — an abort is not a query change + * — and adds the two encryption-specific configurators, which the runtime reads + * at execute time and so remain valid after `single()`; but postgrest-js's + * `overrideTypes` and `setHeader` have NO adapter equivalent, on this surface or + * any other. */ export interface EncryptedSingleQueryBuilder extends PromiseLike> { abortSignal(signal: AbortSignal): EncryptedSingleQueryBuilder throwOnError(): EncryptedSingleQueryBuilder + /** Re-type the ROW. The single-row awaited shape is preserved — `U`, not `U[]`. + * `object`, not `Record`: an `interface` row type has no + * implicit index signature and would be rejected by the latter (upstream + * postgrest-js constrains its `returns` type parameter not at all). */ + returns(): EncryptedSingleQueryBuilder + /** Bind identity-aware encryption. Read at execute time, so order-independent. */ + withLockContext(lockContext: LockContextInput): EncryptedSingleQueryBuilder + /** Attach audit metadata. Read at execute time, so order-independent. */ + audit(config: AuditConfig): EncryptedSingleQueryBuilder } export type EncryptedSupabaseResponse = { @@ -630,7 +646,7 @@ declare const DbBrand: unique symbol */ export type DbName = string & { readonly [DbBrand]: 'column' } -/** A PostgREST select list, DB-space and `::jsonb`-cast. Minted by `addJsonbCasts`/`addJsonbCastsV3`. */ +/** A PostgREST select list, DB-space and `::jsonb`-cast. Minted by `addJsonbCastsV3`. */ export type DbSelect = string & { readonly [DbBrand]: 'select' } /** A PostgREST `or()` filter string in DB-space. Minted by `rebuildOrString`. */ @@ -819,7 +835,7 @@ type StringKeyOf = Extract * of which still serve plaintext columns. */ export interface EncryptedQueryBuilderCore< - T extends Record, + T extends object, FK extends StringKeyOf, Self, /** Keys `order()` accepts. The typed surface narrows it to the orderable @@ -943,8 +959,10 @@ export interface EncryptedQueryBuilderCore< abortSignal(signal: AbortSignal): Self throwOnError(): Self /** Escape hatch: re-types the rows and drops back to the untyped v3 builder - * surface. */ - returns>(): EncryptedQueryBuilderUntyped + * surface. `object`, not `Record`: an `interface` row type + * has no implicit index signature and would be rejected by the latter, while + * `object` still excludes `string`/`number` row types. */ + returns(): EncryptedQueryBuilderUntyped /** Bind identity-aware encryption. Accepts either a plain * `{ identityClaim }` (the common form) or a `LockContext` instance. */ withLockContext(lockContext: LockContextInput): Self diff --git a/packages/stack/README.md b/packages/stack/README.md index 942f65fc..ae45f1ea 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -443,12 +443,12 @@ Notes: ### Supabase Integration -`encryptedSupabaseV3` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: +`encryptedSupabase` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: ```typescript -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +import { encryptedSupabase } from "@cipherstash/stack-supabase" -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) +const es = await encryptedSupabase(supabaseUrl, supabaseKey) await es.from("users").insert({ email: "a@b.com", age: 30 }) await es.from("users").select("id, email").eq("email", "a@b.com") @@ -751,7 +751,7 @@ type UserEncrypted = InferEncrypted |-------|-----| | `@cipherstash/stack/v3` | `Encryption` typed client factory (`EncryptionV3` is a `@deprecated` alias), `typedClient`, plus re-exports of the EQL v3 authoring DSL | | `@cipherstash/stack/eql/v3` | EQL v3 authoring DSL: `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, ...) | -| `@cipherstash/stack` | `Encryption` client factory, auth strategies | +| `@cipherstash/stack` | `Encryption` — the single client factory (overloaded: an array of concrete EQL v3 tables yields the typed v3 client) — plus auth strategies | | `@cipherstash/stack/schema` | Legacy v2 schema builders (see [Legacy: EQL v2](#legacy-eql-v2)) | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) | @@ -795,16 +795,20 @@ Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/do ### Migrating from @cipherstash/protect -`@cipherstash/protect` users land on the legacy v2 surface first — the mapping -below is 1:1, and method signatures on the encryption client (`encrypt`, -`decrypt`, `encryptModel`, etc.) and the `Result` pattern (`data` / `failure`) -are unchanged. From there, adopt EQL v3 for new tables: +Method signatures on the encryption client (`encrypt`, `decrypt`, +`encryptModel`, ...) and the `Result` pattern (`data` / `failure`) are unchanged. +**Declare tables with the EQL v3 DSL** — the v2 builders below are `@deprecated` +and exist to read and migrate data already written as v2, not to author new +columns. A column's capabilities come from its `types.*` domain rather than +chained tuners: `csColumn("email").equality().freeTextSearch()` becomes +`types.TextSearch("email")`. -| `@cipherstash/protect` | `@cipherstash/stack` (legacy v2) | Import Path | +| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | |------------|-----------|-------| | `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | -| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | -| `csColumn(name)` | `encryptedColumn(name)` | `@cipherstash/stack/schema` | +| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/eql/v3` | +| `csColumn(name)` | `types.(name)` (e.g. `types.TextSearch`) | `@cipherstash/stack/eql/v3` | +| `csTable`/`csColumn` for READING legacy v2 data | `encryptedTable` / `encryptedColumn` (`@deprecated`) | `@cipherstash/stack/schema` | | `import { LockContext } from "@cipherstash/protect/identify"` | `import { LockContext } from "@cipherstash/stack/identity"` | `@cipherstash/stack/identity` | | N/A | CLI | `npx stash` | diff --git a/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts new file mode 100644 index 00000000..d8f1bc87 --- /dev/null +++ b/packages/stack/__tests__/dynamodb/v2-table-forwarding.test.ts @@ -0,0 +1,108 @@ +/** + * Which table (if any) the DynamoDB adapter forwards to the encryption client. + * + * `encryptedDynamoDB` promises that `decryptModel` / `bulkDecryptModels` keep + * accepting an EQL **v2** table so previously stored v2 items stay readable + * (see the contract note in `src/dynamodb/index.ts`). That promise breaks if the + * adapter forwards the v2 table to a v3-configured client: the typed client + * looks the table up in its own reconstructor map, does not find it, and fails + * with "decryptModel received a table this client was not initialized with". + * + * The nominal client derives the table from the payloads and needs no second + * argument, and the typed client now exposes a table-less overload for exactly + * this case — so the correct forward is conditional on the table's generation, + * not unconditional. + * + * Credential-free by construction: the adapter never touches the AWS SDK, and + * these drive it with a recording stub client, so the assertion is on the call + * the adapter makes rather than on a live decrypt. That matters because the only + * other coverage of this path is an integration suite requiring live ZeroKMS. + */ +import { describe, expect, it } from 'vitest' +import { encryptedDynamoDB } from '@/dynamodb' +import { encryptedTable as encryptedTableV3, types } from '@/eql/v3' +import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' + +const usersV2 = encryptedTableV2('users_v2', { + email: encryptedColumn('email').equality(), +}) + +const usersV3 = encryptedTableV3('users_v3', { + email: types.TextEq('email'), +}) + +/** + * A client that records how each decrypt method was called. `getEncryptConfig` + * reports `knownTables` so the construction-time version guard sees a client + * that knows the v3 table under test. + */ +function recordingClient(knownTables: string[]) { + const calls: { method: string; argCount: number; table: unknown }[] = [] + + const record = + (method: string) => + (...args: unknown[]) => { + calls.push({ method, argCount: args.length, table: args[1] }) + return Promise.resolve({ data: {} }) + } + + const client = { + getEncryptConfig: () => ({ + v: 1, + tables: Object.fromEntries(knownTables.map((t) => [t, {}])), + }), + encryptModel: record('encryptModel'), + bulkEncryptModels: record('bulkEncryptModels'), + decryptModel: record('decryptModel'), + bulkDecryptModels: record('bulkDecryptModels'), + } + + return { calls, client } +} + +describe('decryptModel table forwarding', () => { + it('does not forward an EQL v2 table to the client', async () => { + const { calls, client } = recordingClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.decryptModel({ pk: 'a' }, usersV2) + + expect(calls).toHaveLength(1) + expect(calls[0]?.method).toBe('decryptModel') + // The v2 table must not reach the client — a typed client would reject it. + expect(calls[0]?.table).toBeUndefined() + }) + + it('still forwards an EQL v3 table, which the typed client requires', async () => { + const { calls, client } = recordingClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.decryptModel({ pk: 'a' }, usersV3) + + expect(calls).toHaveLength(1) + expect(calls[0]?.table).toBe(usersV3) + }) +}) + +describe('bulkDecryptModels table forwarding', () => { + it('does not forward an EQL v2 table to the client', async () => { + const { calls, client } = recordingClient([]) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.bulkDecryptModels([{ pk: 'a' }], usersV2) + + expect(calls).toHaveLength(1) + expect(calls[0]?.method).toBe('bulkDecryptModels') + expect(calls[0]?.table).toBeUndefined() + }) + + it('still forwards an EQL v3 table, which the typed client requires', async () => { + const { calls, client } = recordingClient(['users_v3']) + const dynamo = encryptedDynamoDB({ encryptionClient: client as never }) + + await dynamo.bulkDecryptModels([{ pk: 'a' }], usersV3) + + expect(calls).toHaveLength(1) + expect(calls[0]?.table).toBe(usersV3) + }) +}) diff --git a/packages/stack/__tests__/encryption-overloads.test-d.ts b/packages/stack/__tests__/encryption-overloads.test-d.ts new file mode 100644 index 00000000..e3afd592 --- /dev/null +++ b/packages/stack/__tests__/encryption-overloads.test-d.ts @@ -0,0 +1,126 @@ +/** + * Type-level contract for the `Encryption` overload pair. + * + * `Encryption` is overloaded — an all-v3 schema tuple yields the typed client, + * everything else yields the nominal one — and the two are NOT mutually + * assignable. Overload selection is therefore load-bearing public API, and none + * of it is exercised by a runtime test. Each case below was a real defect found + * in review (#772): a call that type-checked as the typed client but returned + * the nominal one at runtime, a schema set the types accepted and the runtime + * threw on, and a `ReturnType` idiom that silently resolves to the wrong client. + */ +import { describe, expectTypeOf, it } from 'vitest' +import { Encryption, type EncryptionClient } from '@/encryption' +import { + type EncryptionClientFor, + encryptedTable, + type TypedEncryptionClient, +} from '@/encryption/v3' +import { types } from '@/eql/v3' +import { encryptedColumn, encryptedTable as encryptedTableV2 } from '@/schema' + +const users = encryptedTable('users', { + email: types.TextEq('email'), + createdAt: types.TimestampOrd('created_at'), +}) + +const usersV2 = encryptedTableV2('users_v2', { + email: encryptedColumn('email').equality(), +}) + +describe('overload selection', () => { + it('an all-v3 schema tuple yields the typed client', async () => { + const client = await Encryption({ schemas: [users] }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) + + it('a v2 schema set yields the nominal client', async () => { + const client = await Encryption({ schemas: [usersV2] }) + expectTypeOf(client).toEqualTypeOf() + }) + + // S-6: `readonly []` satisfies `readonly AnyV3Table[]`, so an empty schema set + // used to compile and then throw at runtime. Both overloads now require at + // least one table. + it('rejects an empty schema set', () => { + // @ts-expect-error - at least one encryptedTable is required + Encryption({ schemas: [] }) + }) + + // S-4: forcing v2 wire over v3 schemas returns the NOMINAL client at runtime + // (the typed client cannot author v3 columns in v2 mode). The types used to + // claim the typed client, so `decryptModel(row, table, lockContext)` compiled + // and then silently dropped `table` and `lockContext`. + it('forcing eqlVersion 2 over v3 schemas yields the nominal client', async () => { + const client = await Encryption({ + schemas: [users], + config: { eqlVersion: 2 }, + }) + expectTypeOf(client).toEqualTypeOf() + // The typed-only three-arg decrypt is therefore not available — which is + // exactly what the runtime does. + // @ts-expect-error - the nominal client takes the model alone + client.decryptModel({ email: 'x' }, users) + }) + + it('an explicit eqlVersion 3 keeps the typed client', async () => { + const client = await Encryption({ + schemas: [users], + config: { eqlVersion: 3 }, + }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) +}) + +describe('naming the client type', () => { + // S-2: `ReturnType` reads the LAST overload, so this idiom resolves to the + // nominal client no matter what schemas you pass. Pinned rather than fixed — + // overload order cannot satisfy both forms — so the surprise is documented and + // cannot change silently. + it('ReturnType resolves to the NOMINAL client', () => { + expectTypeOf< + Awaited> + >().toEqualTypeOf() + }) + + it('EncryptionClientFor names the typed client for a v3 tuple', async () => { + const client: EncryptionClientFor = + await Encryption({ schemas: [users] }) + expectTypeOf(client).toEqualTypeOf< + TypedEncryptionClient + >() + }) + + it('EncryptionClientFor falls back to the nominal client', () => { + expectTypeOf< + EncryptionClientFor + >().toEqualTypeOf() + }) +}) + +describe('reading legacy EQL v2 models through a typed client', () => { + // The compatibility promise: a v3-configured client must read rows written + // before the upgrade. Their table is not — and cannot be — a member of the + // client's v3 schema tuple, so the table-less call has to type-check. + it('accepts a table-less decryptModel', async () => { + const client = await Encryption({ schemas: [users] }) + expectTypeOf(client.decryptModel).toBeCallableWith({ + pk: 'a', + email: { k: 'ct', v: 2, c: 'ciphertext', i: { t: 'legacy', c: 'email' } }, + }) + expectTypeOf(client.bulkDecryptModels).toBeCallableWith([ + { pk: 'a', email: { k: 'ct', v: 2, c: 'x', i: { t: 'l', c: 'email' } } }, + ]) + }) + + it('still rejects a table this client was not built with', async () => { + const client = await Encryption({ schemas: [users] }) + const unregistered = encryptedTable('other', { note: types.TextEq('note') }) + // @ts-expect-error - `other` is not a member of the client's schema tuple + client.decryptModel({ note: 'x' }, unregistered) + }) +}) diff --git a/packages/stack/__tests__/resolve-eql-version.test.ts b/packages/stack/__tests__/resolve-eql-version.test.ts new file mode 100644 index 00000000..e2e93f0c --- /dev/null +++ b/packages/stack/__tests__/resolve-eql-version.test.ts @@ -0,0 +1,104 @@ +/** + * The wire-format detection matrix behind `Encryption({ schemas })`. + * + * `resolveEqlVersion` decides, from the schema set alone, which EQL wire format + * the FFI client will emit. It carries an `@internal exported for unit-test + * coverage of the detection matrix` marker — this file is that coverage. It had + * none until now, which mattered: the only place the v3 wire choice was + * exercised was `integration/shared/v2-decrypt-compat.integration.test.ts`, and + * that suite needs live ZeroKMS credentials, so a regression here was invisible + * to `pnpm test`. + * + * Why a silent regression here is dangerous rather than merely wrong: if v3 + * detection broke, `resolveEqlVersion` would return `undefined` (the FFI's v2 + * default) instead of throwing, so a v3-schema client would quietly start + * writing v2 wire into `eql_v3_*` columns. Every v2-read compatibility test + * would keep passing, because reading v2 is exactly what a v2-mode client does + * natively. Pin the mapping directly. + * + * Credential-free by construction: `resolveEqlVersion` is pure, and inspects + * only `build()` output and the `buildColumnKeyMap` marker. + */ +import { describe, expect, it } from 'vitest' +import { resolveEqlVersion } from '@/encryption' +import { encryptedTable as encryptedTableV3, types as typesV3 } from '@/eql/v3' +// The deprecated v2 authoring builders remain for reading/migrating legacy data. +import { encryptedColumn, encryptedTable } from '@/schema' + +const usersV3 = encryptedTableV3('users_v3', { + email: typesV3.TextSearch('email'), +}) + +const ordersV3 = encryptedTableV3('orders_v3', { + total: typesV3.IntegerOrd('total'), +}) + +const usersV2 = encryptedTable('users_v2', { + email: encryptedColumn('email').equality(), +}) + +const documentsV2SteVec = encryptedTable('documents_v2', { + metadata: encryptedColumn('metadata').searchableJson(), +}) + +describe('resolveEqlVersion — wire format detection', () => { + it('resolves an all-v3 schema set to 3', () => { + expect(resolveEqlVersion([usersV3])).toBe(3) + }) + + it('resolves several v3 tables to 3', () => { + expect(resolveEqlVersion([usersV3, ordersV3])).toBe(3) + }) + + it('leaves a v2 scalar schema set on the FFI default by returning undefined', () => { + // NOT `2`: the FFI's own default is v2, and `undefined` is what the client + // passes through to mean "don't override it". + expect(resolveEqlVersion([usersV2])).toBeUndefined() + }) + + it('throws on a mixed v2 + v3 schema set — one client emits one wire format', () => { + expect(() => resolveEqlVersion([usersV3, usersV2])).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) + + it('throws on a mixed set regardless of schema order', () => { + expect(() => resolveEqlVersion([usersV2, usersV3])).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) +}) + +describe('resolveEqlVersion — legacy v2 searchable JSON', () => { + it('throws for a v2 ste_vec column, which protect-ffi 0.30 cannot emit', () => { + expect(() => resolveEqlVersion([documentsV2SteVec])).toThrow( + /searchableJson\(\) on the legacy EQL v2 schema is not supported/, + ) + }) + + it('still throws when an explicit eqlVersion is supplied', () => { + // The explicit escape hatch bypasses DETECTION, not validation — otherwise + // it would emit v3 data into an eql_v2 column. + expect(() => resolveEqlVersion([documentsV2SteVec], 2)).toThrow( + /searchableJson\(\) on the legacy EQL v2 schema is not supported/, + ) + }) +}) + +describe('resolveEqlVersion — explicit config.eqlVersion', () => { + it('honours an explicit 2 over v3 schemas, for minting v2 wire during a migration', () => { + expect(resolveEqlVersion([usersV3], 2)).toBe(2) + }) + + it('honours an explicit 3 over a v2 schema set', () => { + expect(resolveEqlVersion([usersV2], 3)).toBe(3) + }) + + it('does not let an explicit version rescue a mixed schema set', () => { + // Mixing is unfixable by declaration: the two generations target different + // column types, so no single wire format serves both. + expect(() => resolveEqlVersion([usersV3, usersV2], 3)).toThrow( + /cannot mix EQL v2 and EQL v3 tables in one client/, + ) + }) +}) diff --git a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts index 46567cfa..7926837a 100644 --- a/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts +++ b/packages/stack/integration/shared/v2-decrypt-compat.integration.test.ts @@ -1,11 +1,11 @@ /** - * Acceptance #1a + #1b — EQL v2 READ compatibility after the v3 collapse (PR 3). + * Acceptance #1a + #1b + #1c — EQL v2 READ compatibility after the v3 collapse. * - * PR 3 makes EQL v3 the only generation the client authors/writes, but MUST keep - * reading previously stored EQL v2 payloads — on the core client (guardrail 1) - * AND through the DynamoDB adapter (guardrail 2). This suite mints v2 data with a - * v2-mode `Encryption` client (the retained `config: { eqlVersion: 2 }` escape - * hatch) and proves it still round-trips: + * The client authors EQL v3 only, but MUST keep reading previously stored EQL v2 + * payloads — on the core client (guardrail 1) AND through the DynamoDB adapter + * (guardrail 2). This suite mints v2 data with a v2-mode `Encryption` client (the + * retained `config: { eqlVersion: 2 }` escape hatch) and proves it still + * round-trips: * * - #1a: a v2 ciphertext + a v2 model decrypt through the collapsed `Encryption` * client's `decrypt` / `decryptModel`. @@ -13,15 +13,28 @@ * `toEncryptedDynamoItem(payload, attrs, false)` — decrypts through * `encryptedDynamoDB(...).decryptModel(item, v2Table)`, exercising the v2 * envelope reconstruction (`toItemWithEqlPayloads`, `v === 2` / `k: 'ct'`). + * - #1c: the SAME v2 payloads decrypt through a **v3-configured** client that + * has never heard of the v2 table. * - * Both fail if the v2 read path is removed. Live: requires real ZeroKMS - * credentials (the integration harness provisions them and throws otherwise). + * #1c is the invariant customers actually depend on and the reason this file + * cannot be a v2-only suite: after the removal their client is v3, their stored + * data is v2, and #1a/#1b would keep passing even if a v3 client had lost the + * ability to read v2 — because both mint AND read through the v2-mode client. + * Whatever eventually deletes the `eqlVersion: 2` minting path must keep #1c + * alive (against a checked-in ciphertext fixture, or a raw `EncryptConfig`), + * not delete it along with the escape hatch. + * + * Live: requires real ZeroKMS credentials (the integration harness provisions + * them and throws otherwise). The credential-free half of the v2 read path — the + * DynamoDB envelope split/reconstruction over static payloads — is covered by + * `__tests__/dynamodb/helpers.test.ts`, which needs no network at all. */ import { unwrapResult } from '@cipherstash/test-kit' import { beforeAll, describe, expect, it } from 'vitest' import { encryptedDynamoDB } from '@/dynamodb' import { toEncryptedDynamoItem } from '@/dynamodb/helpers' import type { EncryptionClient } from '@/encryption' +import { encryptedTable as encryptedTableV3, types as typesV3 } from '@/eql/v3' import { Encryption } from '@/index' // The deprecated v2 authoring builders remain for reading/migrating legacy data. import { encryptedColumn, encryptedTable } from '@/schema' @@ -32,17 +45,34 @@ const usersV2 = encryptedTable('v2_read_compat_users', { email: encryptedColumn('email').equality(), }) +// A DIFFERENT table, in v3 builders, so the v3 client below carries no knowledge +// of the v2 table whatsoever — reading v2 data must not depend on the v2 schema +// still being registered. +const unrelatedV3 = encryptedTableV3('v2_read_compat_unrelated_v3', { + note: typesV3.TextEq('note'), +}) + const SECRET = 'ada@example.com' // A v2-mode client: the explicit `eqlVersion: 2` escape hatch forces v2 wire so // this suite mints genuinely-v2 payloads, independent of schema auto-detection. let v2Client: EncryptionClient +// The client a customer is left with after the removal: v3 schemas, default +// (v3) wire version. Every read in #1c goes through this one. +// +// Typed through a thunk rather than annotated `EncryptionClient`: a concrete v3 +// schema set selects the TYPED overload, and `TypedEncryptionClient` is not +// assignable to the nominal `EncryptionClient`. +const makeV3Client = () => Encryption({ schemas: [unrelatedV3] }) +let v3Client: Awaited> + beforeAll(async () => { v2Client = await Encryption({ schemas: [usersV2], config: { eqlVersion: 2 }, }) + v3Client = await Encryption({ schemas: [unrelatedV3] }) }) describe('#1a — core client reads a stored EQL v2 payload', () => { @@ -92,3 +122,92 @@ describe('#1b — DynamoDB adapter reads a stored EQL v2 item', () => { expect(decrypted).toMatchObject({ pk: 'a', email: SECRET }) }, 30000) }) + +// The real-world shape of the compatibility promise: the customer upgraded, so +// their client is v3-configured and knows nothing about the v2 table — but the +// rows already in their database are v2. +describe('#1c — a v3-configured client reads stored EQL v2 payloads', () => { + // The precondition every other case in this block rests on. Without it the + // whole describe can pass vacuously: if v3 detection regressed, + // `resolveEqlVersion` would return `undefined` rather than throw, leaving + // `v3Client` on the FFI's v2 default — and a v2-mode client reads v2 payloads + // natively, so every assertion below would still be green while the thing + // under test (a *v3* client reading v2) was never exercised. `v3Client` is + // typed through a thunk, so the overload collapse wouldn't surface as a type + // error either. The credential-free half of this guard — the detection matrix + // itself — is `__tests__/resolve-eql-version.test.ts`. + it('is genuinely in EQL v3 mode, or the cases below prove nothing', async () => { + const v3Payload = unwrapResult( + await v3Client.encrypt('a v3-authored note', { + table: unrelatedV3, + column: unrelatedV3.note, + }), + ) + + expect(v3Payload).toMatchObject({ + v: 3, + i: { t: 'v2_read_compat_unrelated_v3', c: 'note' }, + }) + // The structural discriminator, and the stronger half of this guard: a v2 + // scalar carries `k: 'ct'`, a v3 scalar carries no `k` at all (see the + // narrowing note in `src/types.ts`). This still catches a regression that + // somehow preserved `v`. + expect(v3Payload).not.toHaveProperty('k') + }, 30000) + + it('decrypts a v2 ciphertext minted before the upgrade', async () => { + const encrypted = unwrapResult( + await v2Client.encrypt(SECRET, { table: usersV2, column: usersV2.email }), + ) + // Guard against a false pass: this must be a genuine v2 payload, or the + // test proves nothing about v2 compatibility. + expect(encrypted).toMatchObject({ v: 2 }) + + const decrypted = unwrapResult(await v3Client.decrypt(encrypted)) + expect(decrypted).toBe(SECRET) + }, 30000) + + it('decrypts a v2 model minted before the upgrade', async () => { + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + expect(encryptedModel.email).toMatchObject({ v: 2 }) + + const decrypted = unwrapResult(await v3Client.decryptModel(encryptedModel)) + expect(decrypted).toEqual({ pk: 'a', email: SECRET }) + }, 30000) + + it('bulk-decrypts v2 ciphertexts minted before the upgrade', async () => { + const encrypted = unwrapResult( + await v2Client.bulkEncrypt( + [ + { id: '1', plaintext: SECRET }, + { id: '2', plaintext: 'grace@example.com' }, + ], + { table: usersV2, column: usersV2.email }, + ), + ) + + const decrypted = unwrapResult(await v3Client.bulkDecrypt(encrypted)) + expect(decrypted).toEqual([ + { id: '1', data: SECRET }, + { id: '2', data: 'grace@example.com' }, + ]) + }, 30000) + + it('decrypts a stored v2 DynamoDB item through a v3-configured adapter', async () => { + const encryptedModel = unwrapResult( + await v2Client.encryptModel({ pk: 'a', email: SECRET }, usersV2), + ) + const storedV2Item = toEncryptedDynamoItem(encryptedModel, ['email'], false) + + // The adapter is built on the v3 client; only the TABLE argument still + // describes the legacy v2 shape, because that is what the item on disk is. + const dynamo = encryptedDynamoDB({ encryptionClient: v3Client }) + const decrypted = unwrapResult( + await dynamo.decryptModel(storedV2Item, usersV2), + ) + + expect(decrypted).toMatchObject({ pk: 'a', email: SECRET }) + }, 30000) +}) diff --git a/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts b/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts index d6d8b8b3..4f1f4f21 100644 --- a/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts +++ b/packages/stack/src/dynamodb/operations/bulk-decrypt-models.ts @@ -4,6 +4,7 @@ import { logger } from '@/utils/logger' import { buildReadContext, handleError, + isV3Table, resolveDecryptResult, throwPreservingCode, toItemWithEqlPayloads, @@ -53,9 +54,13 @@ export class BulkDecryptModelsOperation< const client = this.encryptionClient as CallableEncryptionClient const decryptResult = await resolveDecryptResult[]>( - // The second argument is required by the typed client and ignored by - // the nominal one, which derives the table from the payloads. - client.bulkDecryptModels(itemsWithEqlPayloads, this.table), + // Conditional for the same reason as `decryptModel` — see the note + // there. A v2 table forwarded to a v3-configured typed client is + // rejected by its reconstructor lookup, breaking the v2 read path + // this adapter documents as supported. + isV3Table(this.table) + ? client.bulkDecryptModels(itemsWithEqlPayloads, this.table) + : client.bulkDecryptModels(itemsWithEqlPayloads), this.getAuditData(), ) diff --git a/packages/stack/src/dynamodb/operations/decrypt-model.ts b/packages/stack/src/dynamodb/operations/decrypt-model.ts index aea746b3..1a725e56 100644 --- a/packages/stack/src/dynamodb/operations/decrypt-model.ts +++ b/packages/stack/src/dynamodb/operations/decrypt-model.ts @@ -3,6 +3,7 @@ import type { Decrypted, EncryptedValue } from '@/types' import { logger } from '@/utils/logger' import { handleError, + isV3Table, resolveDecryptResult, throwPreservingCode, toItemWithEqlPayloads, @@ -47,9 +48,15 @@ export class DecryptModelOperation< const client = this.encryptionClient as CallableEncryptionClient const decryptResult = await resolveDecryptResult>( - // The second argument is required by the typed client and ignored by - // the nominal one, which derives the table from the payloads. - client.decryptModel(withEqlPayloads, this.table), + // The typed client REQUIRES the table; the nominal one derives it + // from the payloads and needs no second argument. Forwarding a v2 + // table unconditionally breaks this adapter's documented v2 read + // path: a v3-configured typed client looks the table up in its own + // reconstructor map, does not find it, and fails. Only a v3 table is + // ever meaningful to that lookup, so only a v3 table is passed. + isV3Table(this.table) + ? client.decryptModel(withEqlPayloads, this.table) + : client.decryptModel(withEqlPayloads), this.getAuditData(), ) diff --git a/packages/stack/src/dynamodb/types.ts b/packages/stack/src/dynamodb/types.ts index bee2e952..cfe37bb6 100644 --- a/packages/stack/src/dynamodb/types.ts +++ b/packages/stack/src/dynamodb/types.ts @@ -170,9 +170,9 @@ type Simplify = { [K in keyof T]: T[K] } * * A declared column `email` does NOT survive as `email`: the adapter deletes it * and writes `email__source` (plus `email__hmac` for equality domains). Typing - * the result as the input model — what the removed v2 write overload did — is a - * lie that type-checks `result.data.email` (always `undefined` at runtime) and - * rejects `result.data.email__source` (the value you actually want). + * the result as the input model is a lie that type-checks `result.data.email` + * (always `undefined` at runtime) and rejects `result.data.email__source` (the + * value you actually want). * * Keys that name no column pass through untouched — partition/sort keys, GSI * attributes, anything else on the item. diff --git a/packages/stack/src/encryption/index.ts b/packages/stack/src/encryption/index.ts index 45ffe5a9..5a47a930 100644 --- a/packages/stack/src/encryption/index.ts +++ b/packages/stack/src/encryption/index.ts @@ -30,6 +30,7 @@ import type { KeysetIdentifier, Plaintext, ScalarQueryTerm, + V3ClientConfig, } from '@/types' import { hasBuildColumnKeyMap } from '@/types' import { logger } from '@/utils/logger' @@ -864,9 +865,15 @@ export function __resetStrategyDeprecationWarningForTests(): void { // Overload 1 — v3-typed: an array literal of concrete EQL v3 tables (from // `@cipherstash/stack/v3`) yields the strongly-typed {@link TypedEncryptionClient}, // the collapse of the former `EncryptionV3`. The wire format is forced to v3. -export function Encryption(config: { +// +// The schema tuple is constrained NON-EMPTY: `readonly AnyV3Table[]` admits +// `readonly []`, so `Encryption({ schemas: [] })` type-checked and then threw at +// runtime. The nominal overload has always required at least one table. +export function Encryption< + const S extends readonly [AnyV3Table, ...AnyV3Table[]], +>(config: { schemas: S - config?: ClientConfig + config?: V3ClientConfig }): Promise> // Overload 2 — nominal: loose/dynamic schemas (introspection-derived, e.g. // stack-supabase) or EQL v2 tables yield the generation-neutral diff --git a/packages/stack/src/encryption/v3.ts b/packages/stack/src/encryption/v3.ts index 070bce61..3ba2f19c 100644 --- a/packages/stack/src/encryption/v3.ts +++ b/packages/stack/src/encryption/v3.ts @@ -14,6 +14,7 @@ import type { LockContextInput } from '@/identity' import type { BulkDecryptPayload, BulkEncryptPayload, + Decrypted, Encrypted, EncryptedReturnType, EncryptOptions, @@ -116,12 +117,32 @@ export interface TypedEncryptionClient { lockContext?: LockContextInput, ): AuditableDecryptModelOperation> + /** + * Table-less form, mirroring the nominal {@link EncryptionClient}: decrypt + * whatever encrypted fields the model carries, with no `Date` reconstruction + * (there is no `cast_as` to reconstruct from) and no precise plaintext shape. + * + * This is the read path for rows that predate this client's schemas — legacy + * **EQL v2** models above all, whose table is not, and cannot be, a member of + * `S`. The runtime has always accepted the one-arg call; without this + * signature the type layer forbids the very compatibility the client + * promises. Prefer the two-arg form whenever the table IS registered. + */ + decryptModel>( + input: T, + ): AuditableDecryptModelOperation> + bulkDecryptModels
>( input: Array, table: Table, lockContext?: LockContextInput, ): AuditableDecryptModelOperation>> + /** Table-less bulk form — see the one-arg {@link decryptModel} overload. */ + bulkDecryptModels>( + input: Array, + ): AuditableDecryptModelOperation>> + // Parity passthroughs — not v3-strengthened, delegated as-is. bulkEncrypt( plaintexts: BulkEncryptPayload, @@ -258,6 +279,80 @@ export function typedClient( return client.encryptQuery(plaintextOrTerms as never, opts as never) } + // Overloaded declarations for the same reason as `encryptQuery` above: the + // table-ful and table-less forms have different return types, and a single + // arrow in the object literal cannot present both. + function decryptModel< + Table extends S[number], + T extends Record, + >( + input: T, + table: Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation> + function decryptModel>( + input: T, + ): AuditableDecryptModelOperation> + function decryptModel( + input: Record, + table?: AnyV3Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation { + // `table` is absent on a nominal-style one-arg call (see `passthroughRow`). + // Given a table: reconstruct dates from its cast_as, or — if it was never + // registered — leave `map` undefined so the mapped op resolves to + // `unknownTableFailure` on execute. + const reconstruct = table + ? reconstructors.get(table.tableName) + : passthroughRow + const op = client.decryptModel(input) + const base = lockContext ? op.withLockContext(lockContext) : op + return new MappedDecryptOperation( + base, + reconstruct, + unknownTableFailure, + ) as never + } + + function bulkDecryptModels< + Table extends S[number], + T extends Record, + >( + input: Array, + table: Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation>> + function bulkDecryptModels>( + input: Array, + ): AuditableDecryptModelOperation>> + function bulkDecryptModels( + input: Array>, + table?: AnyV3Table, + lockContext?: LockContextInput, + ): AuditableDecryptModelOperation { + const op = client.bulkDecryptModels(input) + const base = lockContext ? op.withLockContext(lockContext) : op + // No table → pass rows through (nominal behaviour). Registered table → + // reconstruct each row. Unregistered table → `undefined` map → + // `unknownTableFailure` on execute. + let mapRows: + | (( + rows: Array>, + ) => Array>) + | undefined + if (!table) { + mapRows = passthroughRows + } else { + const reconstruct = reconstructors.get(table.tableName) + mapRows = reconstruct ? (rows) => rows.map(reconstruct) : undefined + } + return new MappedDecryptOperation( + base, + mapRows, + unknownTableFailure, + ) as never + } + return { encrypt: (plaintext, opts) => client.encrypt(plaintext as never, opts as never), @@ -267,53 +362,45 @@ export function typedClient( bulkEncryptModels: (input, table) => client.bulkEncryptModels(input as never, table as never) as never, decrypt: (encrypted) => client.decrypt(encrypted), - decryptModel: (input, table, lockContext) => { - // `table` is absent on a nominal-style one-arg call (see `passthroughRow`). - // Given a table: reconstruct dates from its cast_as, or — if it was never - // registered — leave `map` undefined so the mapped op resolves to - // `unknownTableFailure` on execute. - const maybeTable = table as AnyV3Table | undefined - const reconstruct = maybeTable - ? reconstructors.get(maybeTable.tableName) - : passthroughRow - const op = client.decryptModel(input as never) - const base = lockContext ? op.withLockContext(lockContext) : op - return new MappedDecryptOperation( - base, - reconstruct, - unknownTableFailure, - ) as never - }, - bulkDecryptModels: (input, table, lockContext) => { - const maybeTable = table as AnyV3Table | undefined - const op = client.bulkDecryptModels(input as never) - const base = lockContext ? op.withLockContext(lockContext) : op - // No table → pass rows through (nominal behaviour). Registered table → - // reconstruct each row. Unregistered table → `undefined` map → - // `unknownTableFailure` on execute. - let mapRows: - | (( - rows: Array>, - ) => Array>) - | undefined - if (!maybeTable) { - mapRows = passthroughRows - } else { - const reconstruct = reconstructors.get(maybeTable.tableName) - mapRows = reconstruct ? (rows) => rows.map(reconstruct) : undefined - } - return new MappedDecryptOperation( - base, - mapRows, - unknownTableFailure, - ) as never - }, + decryptModel, + bulkDecryptModels, bulkEncrypt: (plaintexts, opts) => client.bulkEncrypt(plaintexts, opts), bulkDecrypt: (payloads) => client.bulkDecrypt(payloads), getEncryptConfig: () => client.getEncryptConfig(), } satisfies TypedEncryptionClient } +/** + * The client type {@link Encryption} resolves to for the schema tuple `S`. + * + * **Use this instead of `Awaited>`.** `Encryption` + * is overloaded, and TypeScript's `ReturnType` reads the LAST overload — the + * nominal one — so that expression yields `EncryptionClient` even for an all-v3 + * schema set, and assigning the real (typed) client to it is an error: + * + * ``` + * Type 'TypedEncryptionClient<…>' is missing the following properties + * from type 'EncryptionClient': client, encryptConfig, init + * ``` + * + * Overload order cannot fix that — whichever signature is last wins, so one of + * the two forms is always mis-resolved. Name the schema tuple instead: + * + * ```typescript + * const users = encryptedTable("users", { email: types.TextSearch("email") }) + * let client: EncryptionClientFor + * client = await Encryption({ schemas: [users] }) + * ``` + * + * The equivalent inline workaround — inferring through a single-signature + * helper, `Awaited>` — also works, and is what + * `packages/bench` does. + */ +export type EncryptionClientFor = + S extends readonly [AnyV3Table, ...AnyV3Table[]] + ? TypedEncryptionClient + : EncryptionClient + /** * @deprecated Use {@link Encryption} instead — it is now overloaded so an array * of concrete EQL v3 tables yields the same strongly-typed client this used to. diff --git a/packages/stack/src/types-public.ts b/packages/stack/src/types-public.ts index 2553d3d3..bd1ec8b8 100644 --- a/packages/stack/src/types-public.ts +++ b/packages/stack/src/types-public.ts @@ -45,6 +45,7 @@ export type { QueryTypeName, ScalarQueryTerm, SearchTerm, + V3ClientConfig, } from '@/types' // Runtime values diff --git a/packages/stack/src/types.ts b/packages/stack/src/types.ts index 6eee471b..7cca6820 100644 --- a/packages/stack/src/types.ts +++ b/packages/stack/src/types.ts @@ -203,6 +203,22 @@ export type ClientConfig = { eqlVersion?: 2 | 3 } +/** + * {@link ClientConfig} for a client that authors EQL v3 — the same options + * minus the legacy `eqlVersion: 2` escape hatch. + * + * `Encryption` accepts this (not the full `ClientConfig`) alongside an all-v3 + * schema set. Forcing v2 wire over v3 schemas returns the NOMINAL client at + * runtime, so admitting `2` there typed the call as `TypedEncryptionClient` + * while handing back a client that silently ignores the typed client's extra + * `decryptModel` arguments. Adapters that are v3-only (`@cipherstash/prisma-next`, + * `@cipherstash/stack-drizzle`) should take this type for their pass-through + * config for the same reason. + */ +export type V3ClientConfig = Omit & { + eqlVersion?: 3 +} + type AtLeastOneCsTable = [T, ...T[]] /** Structural contract for a column builder the client can consume for STORAGE diff --git a/packages/wizard/src/__tests__/post-agent.test.ts b/packages/wizard/src/__tests__/post-agent.test.ts index 0ee7fc51..c13f83ce 100644 --- a/packages/wizard/src/__tests__/post-agent.test.ts +++ b/packages/wizard/src/__tests__/post-agent.test.ts @@ -1,3 +1,6 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' import { beforeEach, describe, expect, it, vi } from 'vitest' import { runPostAgentSteps } from '../lib/post-agent.js' import type { DetectedPackageManager } from '../lib/types.js' @@ -5,7 +8,25 @@ import type { DetectedPackageManager } from '../lib/types.js' // Mock the child_process module vi.mock('node:child_process') +// Only `confirm` is replaced — the log/spinner calls stay real so the module's +// output paths still execute. +vi.mock('@clack/prompts', async (importOriginal) => { + const actual = await importOriginal() + return { ...actual, confirm: vi.fn(async () => false) } +}) + +// Wraps the REAL sweep, so every test below still exercises it for free. Only +// the empty-message case overrides it, because no real filesystem error is +// reachable with a blank `message`. +vi.mock('../lib/rewrite-migrations.js', async (importOriginal) => { + const actual = + await importOriginal() + return { ...actual, sweepMigrationDirs: vi.fn(actual.sweepMigrationDirs) } +}) + import * as childProcess from 'node:child_process' +import * as p from '@clack/prompts' +import { sweepMigrationDirs } from '../lib/rewrite-migrations.js' const bun: DetectedPackageManager = { name: 'bun', @@ -99,3 +120,103 @@ describe('runPostAgentSteps execution commands', () => { expect(commands).toContain('bunx stash eql install') }) }) + +// The sweep's own warning says "do NOT run the migration" on a populated table. +// Defaulting the very next prompt to Yes invites the mistake the warning exists +// to prevent — so a sweep that touched anything flips the default to No. +describe('drizzle migrate prompt after a destructive rewrite', () => { + let cwd: string + + const runDrizzle = () => + runPostAgentSteps({ + cwd, + integration: 'drizzle', + packageManager: bun, + gathered: { + installCommand: 'bun add @cipherstash/stack', + hasStashConfig: true, + usesProxy: false, + } as never, + }) + + beforeEach(() => { + cwd = fs.mkdtempSync(path.join(os.tmpdir(), 'wizard-post-agent-')) + vi.mocked(p.confirm).mockClear() + }) + + it('defaults to Yes when the sweep changed nothing', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0000_init.sql'), + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(true) + }) + + it('defaults to No, and says why, when a file was rewritten', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0001_encrypt.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('DESTROYS data') + }) + + it('defaults to No when a statement was flagged rather than rewritten', async () => { + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.writeFileSync( + path.join(cwd, 'drizzle', '0001_using.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + }) + + // A directory whose sweep threw contributes 0 to both totals, so a failed + // sweep used to be indistinguishable from a clean one: prompt defaulting to + // Yes over migrations nobody checked. Unknown is not the same as safe. + it('defaults to No when a directory could not be swept at all', async () => { + // A directory named `*.sql` makes readFile throw EISDIR mid-sweep. + fs.mkdirSync(path.join(cwd, 'drizzle')) + fs.mkdirSync(path.join(cwd, 'drizzle', '0001_alter.sql')) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + // Nothing is known about that directory, so the prompt must not claim the + // migration destroys data — only that it went unchecked. + expect(String(options?.message)).not.toContain('DESTROYS data') + expect(String(options?.message)).toContain('drizzle/') + expect(String(options?.message)).toContain('could not check 1 directory') + }) + + // `error` is built as `err instanceof Error ? err.message : String(err)`, and + // `new Error()` has an empty message — so a thrown error can arrive as `''`. + // Testing it for truthiness rather than presence would drop that directory + // back into the fail-open default, which is the exact bug above wearing a + // different hat. + it('treats an empty error message as a failed sweep, not a clean one', async () => { + vi.mocked(sweepMigrationDirs).mockResolvedValueOnce([ + { dir: 'drizzle', rewritten: [], skipped: [], error: '' }, + ]) + + await runDrizzle() + + const [options] = vi.mocked(p.confirm).mock.calls.at(-1) ?? [] + expect(options?.initialValue).toBe(false) + expect(String(options?.message)).toContain('could not check 1 directory') + }) +}) diff --git a/packages/wizard/src/__tests__/rewrite-migrations.test.ts b/packages/wizard/src/__tests__/rewrite-migrations.test.ts index 41507fe8..f5ef3ee4 100644 --- a/packages/wizard/src/__tests__/rewrite-migrations.test.ts +++ b/packages/wizard/src/__tests__/rewrite-migrations.test.ts @@ -438,6 +438,369 @@ describe('rewriteEncryptedAlterColumns', () => { expect(skipped[0].statement).toBe(statement) }) + // A multi-line replacement inherits the author's `-- ` on line 1 ONLY, so + // rewriting a commented-out ALTER turns lines 2+ — including DROP COLUMN — + // into live SQL. Commented SQL never runs; leave it exactly as written. + describe('commented-out statements', () => { + it.each([ + ['a line comment', '-- '], + ['an indented line comment', ' -- '], + ['a drizzle statement-breakpoint style prefix', '--> '], + ])('leaves an ALTER behind %s untouched', async (_label, prefix) => { + const original = `${prefix}ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n` + const filePath = path.join(tmpDir, '0030_commented.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + it('leaves an ALTER inside a block comment untouched', async () => { + const original = [ + '/* superseded by 0031', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_block.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('leaves an ALTER inside a NESTED block comment untouched', async () => { + const original = [ + '/* outer /* inner */', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '*/', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0030_nested.sql') + fs.writeFileSync(filePath, original) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + + it('does not report a commented-out near-miss', async () => { + const filePath = path.join(tmpDir, '0030_commented-using.sql') + fs.writeFileSync( + filePath, + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search USING (email)::eql_v3_text_search;\n', + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toEqual([]) + }) + + // The comment scan must not be fooled by `--` inside a string literal, or + // it would skip a live statement and leave broken SQL to fail at migrate. + it('still rewrites an ALTER that follows a "--" inside a string literal', async () => { + const filePath = path.join(tmpDir, '0030_literal.sql') + fs.writeFileSync( + filePath, + [ + `INSERT INTO "notes" ("body") VALUES ('a -- b');`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(fs.readFileSync(filePath, 'utf-8')).toContain( + 'ALTER TABLE "users" DROP COLUMN "email";', + ) + }) + + // An apostrophe inside a DOUBLE-QUOTED identifier is not a string + // delimiter. Reading it as one opens a phantom literal whose "closing" + // quote is the apostrophe in the SAME identifier further down the file — + // PAST the commented-out ALTER — so the scan concludes the ALTER is live + // and rewrites it into a real DROP COLUMN. The CREATE that declared the + // column always sits above the ALTER, so a real corpus produces exactly + // this shape. + it('leaves a commented-out ALTER untouched when an earlier identifier holds an apostrophe', async () => { + const original = [ + 'CREATE TABLE "users" (', + '\t"id" serial PRIMARY KEY NOT NULL,', + '\t"o\'brien_data" text', + ');', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "o\'brien_data" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0031_apostrophe.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // A statement inside a single-quoted literal is DATA, not SQL. Rewriting it + // splices `--> statement-breakpoint` markers INSIDE the literal, so + // splitting the file the way drizzle's migrator does yields a bare, live + // `ALTER TABLE ... DROP COLUMN ...;` as a chunk of its own. + it('leaves an ALTER inside a string literal untouched', async () => { + const original = [ + `INSERT INTO "audit_log" ("note") VALUES ('the reverted migration read:`, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + `(do not run it again)');`, + '', + ].join('\n') + const filePath = path.join(tmpDir, '0032_string-literal.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + expect(updated).not.toContain('--> statement-breakpoint') + }) + + // An UNTERMINATED quoted identifier must fail the same way an unterminated + // string literal does — by swallowing the rest of the file as inert. If it + // instead runs the scan cursor to the end, the loop exits and every + // commented-out ALTER below it is reported live and rewritten: the same + // destructive outcome as the apostrophe case above, one branch over. + it('leaves a commented-out ALTER untouched after an unterminated quoted identifier', async () => { + const original = [ + 'CREATE TABLE "users" ("id" serial PRIMARY KEY NOT NULL, "email" text);', + '--> statement-breakpoint', + 'SELECT "unclosed FROM users;', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n') + const filePath = path.join(tmpDir, '0033_unterminated-identifier.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toBe(original) + expect(updated).not.toContain('DROP COLUMN') + }) + + // Regression pin, not a bug fix — this already behaves. A commented-out + // ALTER in a CRLF file must come back byte-identical. + it('leaves a commented-out ALTER with CRLF line endings byte-identical', async () => { + const original = [ + 'CREATE TABLE "users" ("id" integer PRIMARY KEY);', + '--> statement-breakpoint', + '-- ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\r\n') + const filePath = path.join(tmpDir, '0033_crlf.sql') + fs.writeFileSync(filePath, original) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toEqual([]) + expect(fs.readFileSync(filePath, 'utf-8')).toBe(original) + }) + }) + + // ADD+DROP+RENAME on a column that is ALREADY encrypted drops CIPHERTEXT, and + // unlike the plaintext case there is nothing left anywhere to backfill from. + describe('columns that are already encrypted', () => { + it('refuses to rewrite a domain change on a column created encrypted', async () => { + const create = path.join(tmpDir, '0000_create.sql') + fs.writeFileSync( + create, + [ + 'CREATE TABLE "users" (', + '\t"id" integer PRIMARY KEY,', + '\t"email" "public"."eql_v3_text_eq"', + ');', + '', + ].join('\n'), + ) + const alterSql = + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;' + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync(alter, `${alterSql}\n`) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(fs.readFileSync(alter, 'utf-8')).toBe(`${alterSql}\n`) + expect(skipped).toEqual([ + { file: alter, statement: alterSql, reason: 'already-encrypted' }, + ]) + }) + + it('refuses to rewrite a domain change on a column ADDed encrypted', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // A previous sweep of this directory leaves ADD tmp + RENAME behind. The + // column it renamed onto is encrypted, so a later domain change on it is + // just as destructive. + it('follows a RENAME from a previous sweep', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_swept.sql'), + [ + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_eq";', + '--> statement-breakpoint', + 'ALTER TABLE "users" DROP COLUMN "email";', + '--> statement-breakpoint', + 'ALTER TABLE "users" RENAME COLUMN "email__cipherstash_tmp" TO "email";', + '', + ].join('\n'), + ) + const alter = path.join(tmpDir, '0001_domain-change.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([]) + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + it('reports the destructive statement once, not twice', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + [ + '-- Custom SQL migration file, put your code below! --', + '', + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;', + '', + ].join('\n'), + ) + + const { skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(skipped).toHaveLength(1) + expect(skipped[0].reason).toBe('already-encrypted') + }) + + // The scoping matters: encrypting `contacts.email` must not be blocked by + // an unrelated `users.email` that happens to share a column name. + it('scopes the check to the table', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_contacts.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "contacts" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // The ordinary case this rewrite exists for: plaintext today, encrypted + // after the ALTER. Nothing to preserve, so rewrite it. + it('still rewrites a plaintext column created in an earlier migration', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_create.sql'), + 'CREATE TABLE "users" (\n\t"id" integer PRIMARY KEY,\n\t"email" text\n);\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + expect(skipped).toEqual([]) + }) + + // A commented-out ADD never ran, so it says nothing about the live schema. + it('ignores an encrypted ADD COLUMN that is commented out', async () => { + fs.writeFileSync( + path.join(tmpDir, '0000_add.sql'), + '-- ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + const alter = path.join(tmpDir, '0001_encrypt.sql') + fs.writeFileSync( + alter, + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([alter]) + }) + + // `options.skip` excludes a file from being EDITED, not from describing the + // schema the other files are altering. + it('honours an encrypted column defined in the skipped file', async () => { + const skipPath = path.join(tmpDir, '0000_install.sql') + fs.writeFileSync( + skipPath, + 'ALTER TABLE "users" ADD COLUMN "email" eql_v3_text_eq;\n', + ) + fs.writeFileSync( + path.join(tmpDir, '0001_domain-change.sql'), + 'ALTER TABLE "users" ALTER COLUMN "email" SET DATA TYPE eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns( + tmpDir, + { + skip: skipPath, + }, + ) + + expect(rewritten).toEqual([]) + expect(skipped[0]?.reason).toBe('already-encrypted') + }) + }) + it('handles multiple ALTER statements in one file', async () => { const original = [ 'ALTER TABLE "a" ALTER COLUMN "x" SET DATA TYPE eql_v3_text_search;', @@ -455,6 +818,27 @@ describe('rewriteEncryptedAlterColumns', () => { // Non-matching statement preserved expect(updated).toContain('CREATE INDEX "a_z" ON "a" ("z");') }) + + // Regression pin, not a bug fix — the matchers carry `/gi`, so a + // hand-lowercased migration is rewritten just like drizzle-kit's output. + it('rewrites a lowercase alter table ... set data type', async () => { + const filePath = path.join(tmpDir, '0034_lowercase.sql') + fs.writeFileSync( + filePath, + 'alter table "users" alter column "email" set data type eql_v3_text_search;\n', + ) + + const { rewritten, skipped } = await rewriteEncryptedAlterColumns(tmpDir) + + expect(rewritten).toEqual([filePath]) + expect(skipped).toEqual([]) + const updated = fs.readFileSync(filePath, 'utf-8') + expect(updated).toContain( + 'ALTER TABLE "users" ADD COLUMN "email__cipherstash_tmp" "public"."eql_v3_text_search";', + ) + expect(updated).toContain('ALTER TABLE "users" DROP COLUMN "email";') + expect(updated).not.toMatch(/set data type/i) + }) }) describe('sweepMigrationDirs', () => { diff --git a/packages/wizard/src/lib/post-agent.ts b/packages/wizard/src/lib/post-agent.ts index e66af3b1..e4d4e215 100644 --- a/packages/wizard/src/lib/post-agent.ts +++ b/packages/wizard/src/lib/post-agent.ts @@ -8,7 +8,7 @@ import { execSync } from 'node:child_process' import * as p from '@clack/prompts' import type { GatheredContext } from './gather.js' -import { sweepMigrationDirs } from './rewrite-migrations.js' +import { describeSkipReason, sweepMigrationDirs } from './rewrite-migrations.js' import type { DetectedPackageManager, Integration } from './types.js' interface PostAgentOptions { @@ -80,11 +80,42 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { // drizzle-kit just produced — those fail in Postgres (no cast from // text/numeric to an EQL domain). Covers the EQL v3 family the wizard now // scaffolds, and legacy eql_v2_encrypted. CIP-2991 + CIP-2994 + #693. - await rewriteEncryptedMigrations(cwd) + const sweep = await rewriteEncryptedMigrations(cwd) + + // A rewritten file is a DROP+ADD in disguise, and a flagged statement is one + // the sweep could not make safe at all. Either way the next keystroke can + // destroy data, so the prompt says so and defaults to NO — an + // `initialValue: true` immediately under a "do NOT run the migration" + // warning invites exactly the mistake the warning is about. + const unsafe = sweep.rewritten > 0 || sweep.skipped > 0 + + // A directory whose sweep threw contributes 0 to both totals, so on its own + // it is indistinguishable from a clean sweep — except that it means the + // opposite: those migrations may still hold unrepaired `SET DATA TYPE` + // statements and nobody has looked. `stash eql migration` / `db install` + // treat "sweep failed outright" and "sweep left near-misses" as the same + // state for the same reason; unknown is not safe, so the default is NO here + // too. The wording differs from the destructive case on purpose: nothing is + // known about that directory, so claiming it destroys data would be a guess. + const unverifiedDirs = sweep.failedDirs + const unverified = unverifiedDirs.length > 0 + const unverifiedList = unverifiedDirs.map((dir) => `${dir}/`).join(', ') + const unverifiedCount = `${unverifiedDirs.length} director${ + unverifiedDirs.length === 1 ? 'y' : 'ies' + }` + if (unverified) { + p.log.warn( + `The ALTER COLUMN sweep did not fully complete — review the sibling migrations in ${unverifiedList} before running drizzle-kit migrate, or you may apply broken/unsafe SQL.`, + ) + } const shouldMigrate = await p.confirm({ - message: `Run the migration now? (${runner} drizzle-kit migrate)`, - initialValue: true, + message: unsafe + ? `Run the migration now? (${runner} drizzle-kit migrate) — see the warnings above: this migration DESTROYS data on any table that already holds rows` + : unverified + ? `Run the migration now? (${runner} drizzle-kit migrate) — the sweep could not check ${unverifiedCount} (${unverifiedList}); review those migrations before migrating, or you may apply broken/unsafe SQL` + : `Run the migration now? (${runner} drizzle-kit migrate)`, + initialValue: !unsafe && !unverified, }) if (!p.isCancel(shouldMigrate) && shouldMigrate) { @@ -114,12 +145,37 @@ export async function runPostAgentSteps(opts: PostAgentOptions): Promise { } } -async function rewriteEncryptedMigrations(cwd: string): Promise { +/** + * Sweep the candidate migration directories, reporting what happened, and + * return the totals so the caller can decide how dangerous "run it now" is. + * + * `failedDirs` names the directories that exist but whose sweep threw. It is a + * third state, not a variant of "nothing to do": those migrations may still + * contain unrepaired `SET DATA TYPE` statements and went unchecked, which the + * `rewritten`/`skipped` counts cannot express — both stay 0 for such a + * directory, exactly as they do for a clean one. + */ +async function rewriteEncryptedMigrations(cwd: string): Promise<{ + rewritten: number + skipped: number + failedDirs: string[] +}> { const results = await sweepMigrationDirs(cwd, DRIZZLE_OUT_DIRS) + const totals = { rewritten: 0, skipped: 0, failedDirs: [] as string[] } for (const { dir, rewritten, skipped, error } of results) { - if (error) { - p.log.warn(`Could not rewrite migrations in ${dir}: ${error}`) + totals.rewritten += rewritten.length + totals.skipped += skipped.length + + // Presence, not truthiness: `error` is `err.message` for a thrown `Error`, + // and `new Error()` has an empty message. Testing `if (error)` would put a + // blank-message failure back on the fail-open path this whole branch exists + // to close. + if (error !== undefined) { + totals.failedDirs.push(dir) + p.log.warn( + `Could not rewrite migrations in ${dir}: ${error || 'unknown error'}`, + ) continue } @@ -135,11 +191,16 @@ async function rewriteEncryptedMigrations(cwd: string): Promise { if (skipped.length > 0) { p.log.warn( - `${skipped.length} statement(s) look like an ALTER-to-encrypted the rewrite could not safely repair (e.g. a hand-authored SET DATA TYPE ... USING ...). Review them before migrating:`, + `${skipped.length} statement(s) look like an ALTER-to-encrypted that the rewrite left alone. Review them before migrating:`, ) - for (const s of skipped) p.log.step(` - ${s.file}: ${s.statement}`) + for (const s of skipped) { + p.log.step(` - ${s.file}: ${s.statement}`) + p.log.step(` ${describeSkipReason(s.reason)}`) + } } } + + return totals } async function runStep( diff --git a/packages/wizard/src/lib/rewrite-migrations.ts b/packages/wizard/src/lib/rewrite-migrations.ts index ce3ce1ab..73e1908b 100644 --- a/packages/wizard/src/lib/rewrite-migrations.ts +++ b/packages/wizard/src/lib/rewrite-migrations.ts @@ -124,12 +124,242 @@ function trimStatementPreamble(statement: string): string { return statement.replace(STATEMENT_PREAMBLE_RE, '').trim() } +/** + * True when the character at `index` is INERT — it sits inside a SQL comment (a + * `--` line comment, or a nestable block comment) or inside a single-quoted + * string literal. Either way it is not a statement the database will execute, + * so the rewrite must leave it exactly as written. + * + * **Why the rewrite needs this:** {@link ALTER_COLUMN_TO_ENCRYPTED_RE} is + * comment-blind and {@link renderSafeAlter} returns MULTIPLE lines. Rewriting a + * commented-out `-- ALTER TABLE … SET DATA TYPE …;` therefore leaves the + * author's `-- ` prefix on line 1 only — lines 2+, including `DROP COLUMN`, + * become live executable SQL and destroy the column. Commented SQL is inert by + * definition, so the only correct move is to leave it exactly as written. + * + * **Why string literals count as inert too:** an ALTER quoted inside an + * `INSERT … VALUES ('…')` is DATA. Rewriting it splices `--> statement-breakpoint` + * markers INSIDE the literal, so splitting the file the way drizzle's migrator + * does yields a bare, live `ALTER TABLE … DROP COLUMN …;` as a chunk of its own. + * Escaping the injected text's own apostrophe (`@cipherstash/stack's`) would fix + * only the syntax error, not that live DROP COLUMN — so the statement is left + * as written, exactly like a commented one. + * + * Double-quoted identifiers are tokenised BEFORE `'` is considered: an + * apostrophe inside `"o'brien_data"` must not open a phantom literal, or the + * scan runs to the NEXT apostrophe — typically the same identifier in a + * commented-out ALTER further down — decides that ALTER is live, and rewrites + * it into a real `DROP COLUMN`. A doubled delimiter (`''` in a literal, `""` in + * an identifier) is an escape and does not close the token. + * + * Dollar-quoted bodies are NOT tracked: a `--` or `'` inside one reads as a + * comment/literal here, which can only make us skip a rewrite (the statement + * then fails loudly at migrate time), never perform a destructive one. + */ +function isInsideCommentOrString(sql: string, index: number): boolean { + let i = 0 + while (i < index) { + if (sql.startsWith('--', i)) { + const eol = sql.indexOf('\n', i) + if (eol === -1 || eol >= index) return true + i = eol + 1 + } else if (sql.startsWith('/*', i)) { + // Postgres block comments nest, so track depth rather than stopping at + // the first `*/` — a nested close would otherwise end the comment early + // and let the text after it read as live SQL. + let depth = 1 + let j = i + 2 + while (j < sql.length && depth > 0) { + if (sql.startsWith('/*', j)) { + depth += 1 + j += 2 + } else if (sql.startsWith('*/', j)) { + depth -= 1 + j += 2 + } else { + j += 1 + } + } + if (j > index) return true + i = j + } else if (sql[i] === '"') { + // A quoted identifier is live SQL, but its body is not: consuming it here + // — before the `'` branch below — is what stops an apostrophe inside one + // from opening a string literal that never really existed. + const end = endOfQuoted(sql, i, '"') + // An unterminated identifier swallows the rest of the file, so treat it + // as inert exactly like the unterminated literal below. Running the + // cursor to the end instead would exit the loop and report `false` — + // "live" — which is how the apostrophe bug destroyed a column in the + // first place, one branch over. + if (end > index) return true + i = end + } else if (sql[i] === "'") { + const end = endOfQuoted(sql, i, "'") + // Unterminated, or the literal runs past `index`: `index` is inside a + // string literal, which is every bit as inert as a comment. + if (end > index) return true + i = end + } else { + i += 1 + } + } + return false +} + +/** + * The index just past the `quote`-delimited token that opens at `open`, or + * `sql.length` when it is never closed. A doubled delimiter inside the token is + * an escaped one (`''`, `""`) and does not end it. + */ +function endOfQuoted(sql: string, open: number, quote: "'" | '"'): number { + let i = open + 1 + while (i < sql.length) { + if (sql[i] !== quote) { + i += 1 + } else if (sql[i + 1] === quote) { + i += 2 + } else { + return i + 1 + } + } + return sql.length +} + +/** A table reference, bare (`"users"`) or schema-qualified (`"app"."users"`). */ +const TABLE_REF = String.raw`"([^"]+)"(?:\."([^"]+)")?` + +/** + * An encrypted type in any of the {@link MANGLED_TYPE_FORMS}, pinned to end at a + * delimiter so a bare domain cannot match a prefix of a longer identifier. + */ +const ENCRYPTED_TYPE_REF = String.raw`(?:${MANGLED_TYPE_FORMS})(?=[\s,;)]|$)` + +/** `ALTER TABLE … ADD COLUMN "col" ` — $1/$2 table, $3 column. */ +const ADD_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+ADD COLUMN\s+(?:IF NOT EXISTS\s+)?"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** `ALTER TABLE … RENAME COLUMN "a" TO "b"` — $1/$2 table, $3 from, $4 to. */ +const RENAME_COLUMN_RE = new RegExp( + String.raw`ALTER TABLE\s+${TABLE_REF}\s+RENAME COLUMN\s+"([^"]+)"\s+TO\s+"([^"]+)"`, + 'gi', +) + +/** `CREATE TABLE … ( … );` — $1/$2 table, $3 the column-definition body. */ +const CREATE_TABLE_RE = new RegExp( + String.raw`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?${TABLE_REF}\s*\(([\s\S]*?)\)\s*;`, + 'gi', +) + +/** `"col" ` inside a CREATE TABLE body — $1 column. */ +const CREATE_TABLE_ENCRYPTED_COLUMN_RE = new RegExp( + String.raw`"([^"]+)"\s+${ENCRYPTED_TYPE_REF}`, + 'gi', +) + +/** Splits a `TABLE_REF` capture pair into its schema and table halves. */ +function tableOf( + first: string, + second: string | undefined, +): { schema?: string; table: string } { + // When schema-qualified (`"app"."users"`) the first capture is the schema and + // the second is the table; otherwise the first is the table. + return second === undefined + ? { table: first } + : { schema: first, table: second } +} + +/** Identity of a column across the corpus, for {@link indexEncryptedColumns}. */ +function columnKey(table: string, column: string, schema?: string): string { + return JSON.stringify([schema ?? '', table, column]) +} + +/** + * Index every column the migration corpus gives an ENCRYPTED type, so the + * rewrite can tell the change it exists for (plaintext → encrypted) from one it + * must never touch (encrypted → encrypted). + * + * **Why (#772 review, W-3):** the strict matcher captures only the TARGET type. + * A column whose encrypted domain merely changes (`types.TextEq` → + * `types.TextSearch`) matches just as well as a plaintext column, and the + * ADD+DROP+RENAME then drops a column full of CIPHERTEXT — with no plaintext + * left anywhere to backfill from, so unlike the plaintext case the data is not + * recoverable from the application at all. Changing an encrypted column's domain + * changes its index terms, so the data has to be re-encrypted through the client + * regardless; the sweep flags the statement and leaves it for the user. + * + * The index is corpus-wide rather than ordered by migration: over-detecting + * "encrypted" costs a flagged statement the user must handle by hand, while + * under-detecting costs irrecoverable ciphertext. Only comment-free statements + * count, for the same reason {@link isInsideCommentOrString} exists. + */ +function indexEncryptedColumns(contents: readonly string[]): Set { + const encrypted = new Set() + + for (const sql of contents) { + for (const created of sql.matchAll(CREATE_TABLE_RE)) { + if (isInsideCommentOrString(sql, created.index)) continue + const { schema, table } = tableOf(created[1], created[2]) + for (const column of created[3].matchAll( + CREATE_TABLE_ENCRYPTED_COLUMN_RE, + )) { + encrypted.add(columnKey(table, column[1], schema)) + } + } + + for (const added of sql.matchAll(ADD_ENCRYPTED_COLUMN_RE)) { + if (isInsideCommentOrString(sql, added.index)) continue + const { schema, table } = tableOf(added[1], added[2]) + encrypted.add(columnKey(table, added[3], schema)) + } + + // A rename carries the column's type with it — and `__cipherstash_tmp` + // renamed onto the real name is exactly what a previous sweep of this very + // directory emitted. Run after ADD so that tmp column is already indexed. + for (const renamed of sql.matchAll(RENAME_COLUMN_RE)) { + if (isInsideCommentOrString(sql, renamed.index)) continue + const { schema, table } = tableOf(renamed[1], renamed[2]) + if (encrypted.has(columnKey(table, renamed[3], schema))) { + encrypted.add(columnKey(table, renamed[4], schema)) + } + } + } + + return encrypted +} + +/** Why a recognised ALTER-to-encrypted statement was left alone. */ +export type SkipReason = + /** Outside the strict matcher — hand-authored `USING`, or an unknown form. */ + | 'unrecognised-form' + /** The column already holds an encrypted domain; rewriting drops ciphertext. */ + | 'already-encrypted' + /** A statement the sweep recognised as ALTER-to-encrypted but did NOT rewrite. */ export interface SkippedAlter { /** Absolute path of the migration file the statement lives in. */ file: string /** The offending statement, verbatim (trimmed), for the user to review. */ statement: string + /** Why it was left alone — the caller turns this into user-facing guidance. */ + reason: SkipReason +} + +/** + * One-line explanation of a {@link SkipReason}, for the CLI/wizard to print + * next to the statement. Lives here so every caller says the same thing — the + * two reasons need very different action from the user, and a single generic + * "could not rewrite automatically" hides that. + */ +export function describeSkipReason(reason: SkipReason): string { + switch (reason) { + case 'already-encrypted': + return "the column is ALREADY encrypted, so the ADD+DROP+RENAME rewrite would DROP the ciphertext with no plaintext left to backfill from. Changing an encrypted column's domain changes its index terms, so the data must be re-encrypted through the staged `stash encrypt` lifecycle" + case 'unrecognised-form': + return 'it falls outside the strict matcher (a hand-authored `SET DATA TYPE ... USING ...`, or a drizzle-kit form the sweep does not recognise) and an in-place cast to an encrypted domain fails at migrate time' + } } /** Outcome of a sweep: the files rewritten, and near-misses left for review. */ @@ -162,27 +392,58 @@ export interface RewriteResult { * which keeps both columns alive across deploys. Each rewritten file carries a * header comment saying exactly this. * - * Returns {@link RewriteResult}: the files rewritten, plus `skipped` near-misses - * — statements that look like an ALTER-to-encrypted but fall outside the strict - * matcher (a hand-authored `SET DATA TYPE … USING …;`, or a future drizzle-kit - * form). Near-misses are left untouched on disk and surfaced non-fatally so the - * caller can tell the user to review them, rather than silently shipping broken - * SQL. + * Returns {@link RewriteResult}: the files rewritten, plus `skipped` statements + * left for a human — ones outside the strict matcher (a hand-authored + * `SET DATA TYPE … USING …;`, or a future drizzle-kit form), and ones targeting + * a column that is ALREADY encrypted, where the rewrite would drop ciphertext. + * Both are left untouched on disk and surfaced non-fatally so the caller can + * tell the user to review them, rather than silently shipping broken SQL or + * destroying data. Statements sitting inside a SQL comment — or inside a + * single-quoted string literal, where they are data rather than SQL — are inert + * and are neither rewritten nor reported. */ export async function rewriteEncryptedAlterColumns( outDir: string, options: { skip?: string } = {}, ): Promise { - const entries = await readdir(outDir).catch(() => []) + const entries = await readdir(outDir).catch( + (error: NodeJS.ErrnoException) => { + // A missing directory is simply nothing to sweep. Anything else — EACCES + // above all — is a sweep that did NOT happen, and the caller reports it + // rather than letting the user believe their migrations were checked. + if (error.code === 'ENOENT') return [] as string[] + throw error + }, + ) const rewritten: string[] = [] const skipped: SkippedAlter[] = [] + const seen = new Set() + + /** Record a skip once — the strict pass and the broad scan can both find it. */ + const skip = (file: string, statement: string, reason: SkipReason): void => { + // Keyed on collapsed whitespace: the two passes trim the same statement + // by slightly different rules, and the strict pass runs first so its + // more specific reason is the one kept. + const key = `${file} :: ${statement.replace(/\s+/g, ' ')}` + if (seen.has(key)) return + seen.add(key) + skipped.push({ file, statement, reason }) + } - for (const entry of entries) { - if (!entry.endsWith('.sql')) continue + const sqlFiles = entries.filter((entry) => entry.endsWith('.sql')).sort() + const contents = new Map() + for (const entry of sqlFiles) { const filePath = join(outDir, entry) - if (options.skip && filePath === options.skip) continue + contents.set(filePath, await readFile(filePath, 'utf-8')) + } + + // Built from the WHOLE corpus, including `options.skip`: a column's current + // type comes from the migrations that ran before this one, not just the files + // we are allowed to edit. + const encryptedColumns = indexEncryptedColumns([...contents.values()]) - const original = await readFile(filePath, 'utf-8') + for (const [filePath, original] of contents) { + if (options.skip && filePath === options.skip) continue // Reset the regex's lastIndex — it's stateful on /g ALTER_COLUMN_TO_ENCRYPTED_RE.lastIndex = 0 @@ -195,11 +456,21 @@ export async function rewriteEncryptedAlterColumns( second: string | undefined, column: string, mangledType: string, + offset: number, ) => { - // When schema-qualified (`"app"."users"`) the first capture is the - // schema and the second is the table; otherwise the first is the table. - const schema = second === undefined ? undefined : first - const table = second === undefined ? first : second + // Commented-out SQL never runs, and a multi-line replacement would only + // inherit the `-- ` on its first line — leaving the rest live. + if (isInsideCommentOrString(original, offset)) return match + + const { schema, table } = tableOf(first, second) + + // Already encrypted: the ADD+DROP+RENAME would drop the ciphertext and + // there is no plaintext left to backfill from. Flag, never guess. + if (encryptedColumns.has(columnKey(table, column, schema))) { + skip(filePath, match.trim(), 'already-encrypted') + return match + } + const domain = DOMAIN_RE.exec(mangledType)?.[0]?.toLowerCase() // Unreachable — the outer regex only matches when a domain is present — // but leave the statement alone rather than emit a broken rewrite. @@ -218,10 +489,16 @@ export async function rewriteEncryptedAlterColumns( // matcher. Flag it — non-fatally — rather than leave the user shipping SQL // that fails at migrate time. for (const nearMiss of updated.matchAll(NEAR_MISS_RE)) { - skipped.push({ - file: filePath, - statement: trimStatementPreamble(nearMiss[0]), - }) + const statement = trimStatementPreamble(nearMiss[0]) + // Anchor the comment test on the `SET DATA TYPE` itself: the match starts + // at the previous `;`, so its own offset sits before any preamble. + const keyword = nearMiss[0].search(/\bSET\s+DATA\s+TYPE\b/i) + if ( + isInsideCommentOrString(updated, nearMiss.index + Math.max(keyword, 0)) + ) { + continue + } + skip(filePath, statement, 'unrecognised-form') } } diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 7cf99621..b35c9be4 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -352,9 +352,9 @@ Gets a project from zero to installed EQL. It loads an existing `stash.config.ts | `--force` | Reinstall even if EQL is present | | `--dry-run` | Show what would happen | | `--supabase` | Supabase-compatible install (no operator families; grants `anon`, `authenticated`, `service_role`) | -| `--drizzle` | Generate a Drizzle migration (`--name`, `--out` tune it — `--name` accepts letters, numbers, `-`, `_` only; `--out` is passed to `drizzle-kit --out`, so set it to match your `drizzle.config.ts`) | -| `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly | -| `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`) | +| `--drizzle` | Generate a Drizzle migration (**v2 only — requires `--eql-version 2`**; for v3 use `eql migration --drizzle`). `--name`, `--out` tune it — `--name` accepts letters, numbers, `-`, `_` only; `--out` is passed to `drizzle-kit --out`, so set it to match your `drizzle.config.ts` | +| `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly. `--migration` is **v2 only — requires `--eql-version 2`**; for a v3 install as a migration use `eql migration` | +| `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`). **v2 only — requires `--eql-version 2`** | | `--exclude-operator-family` | Skip operator families (non-superuser roles) | | `--eql-version <2\|3>` | EQL generation. **Default `3`** (the native `public.eql_v3_*` domain schema — the documented approach). `2` is the legacy composite schema. | | `--latest` | Fetch latest EQL from GitHub instead of the bundled copy (**v2 only**) | @@ -433,7 +433,7 @@ For AI-guided integration that edits your existing schema files in place, prefer The database-side toolset that takes an existing plaintext column the rest of the way, **after** the rollout PR is deployed and dual-writes are live. It drives `@cipherstash/migrate`, recording every transition in `cipherstash.cs_migrations` (installed by `eql install`) and reading intent from `.cipherstash/migrations.json`. -The phase ladder depends on the column's EQL version, which the commands detect from the column's **domain type** (EQL v3 types are self-describing; the `_encrypted` naming is a convention only, never relied upon): +The phase ladder depends on the column's EQL version, which the commands read off the column's **domain type** — never off the `_encrypted` naming, which is a convention only. Detection is one-sided: a `public.eql_v3_*` domain is recognised as v3, and everything else (including a legacy `eql_v2_encrypted` column) falls through to the v2 ladder: - **EQL v3 (the default):** `schema-added → dual-writing → backfilling → backfilled → dropped`. There is no cut-over — the application switches to the encrypted column by name, then the plaintext column is dropped. - **EQL v2:** `schema-added → dual-writing → backfilling → backfilled → cut-over → dropped`, where cut-over renames the encrypted twin into the original column name. @@ -451,7 +451,7 @@ stash encrypt backfill --table users --column email --chunk-size 5000 Chunked, resumable, idempotent. Walks the table in keyset-pagination order, encrypts each chunk via `bulkEncryptModels`, and writes one `UPDATE ... FROM (VALUES ...)` per chunk in a transaction that also checkpoints to `cs_migrations`. SIGINT/SIGTERM finishes the current chunk and exits cleanly; re-running resumes. The `IS NOT NULL AND _encrypted IS NULL` guard makes concurrent runners and re-runs converge. -Backfill **auto-detects the target column's EQL version** from its Postgres domain type and records it (plus the version-appropriate target phase) in `.cipherstash/migrations.json`. On an EQL v3 column it finishes by printing the v3 next steps: switch the application to `_encrypted` by name, then `stash encrypt drop` — there is no cut-over. +Backfill **detects a `public.eql_v3_*` target column as EQL v3** from its Postgres domain type and records it (plus the target phase) in `.cipherstash/migrations.json`. A column that is not a v3 domain — including a legacy `eql_v2_encrypted` one — does not classify and takes the v2 lifecycle, recording no version. On an EQL v3 column it finishes by printing the v3 next steps: switch the application to `_encrypted` by name, then `stash encrypt drop` — there is no cut-over. **Dual-write precondition.** The application must already write both `` and `_encrypted` on every insert and update. Otherwise rows written *during* the backfill land in plaintext only, silently. The first run prompts (interactive) or requires `--confirm-dual-writes-deployed` (non-interactive), then records `dual_writing`. Resumes don't re-prompt. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 05728057..f52025bb 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -421,9 +421,9 @@ Run `ANALYZE
` after the migration applies — an expression index gather The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (If using CipherStash Proxy, the rollout also includes `stash db push` to register the encryption config with EQL.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and a **cutover step** (backfill + switch reads + drop — under EQL v2 the switch is a rename, under v3 it is an application-side change). (On a legacy EQL v2 + CipherStash Proxy database, the rollout also includes `stash db push` to register the encryption config in `eql_v2_configuration`; EQL v3 ships no configuration table, so on the v3-only database the schema below assumes, `db push` reports "Nothing to do." and a v3 rollout never needs it.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. -> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and auto-detects a column's version from its Postgres domain type — there is no flag. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +> **EQL version note.** The CLI rollout tooling (`stash encrypt *`, and the underlying `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what the schema below uses) is `schema-add → dual-write → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which Drizzle tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. @@ -473,19 +473,19 @@ const usersEncryptionSchema = extractEncryptionSchema(users) export const encryptionClient = await Encryption({ schemas: [usersEncryptionSchema] }) ``` -Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted public.eql_v3_text_search;`. Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) +Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN "email_encrypted" "eql_v3_text_search";` — drizzle-kit emits the **bare** domain name, which resolves to the `public.eql_v3_text_search` domain via `search_path` (a schema-qualified custom type would be quoted as one identifier and fail, so the bare name is deliberate). Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) > **Using CipherStash Proxy?** > -> If your app queries encrypted data through CipherStash Proxy, register the new encryption config with EQL: +> `stash db push` registers the encryption config in `eql_v2_configuration`, which only exists on a database that has EQL v2 installed. The Database Setup above installs EQL v3 only, and `types.TextSearch('email_encrypted')` is a `public.eql_v3_text_search` column — v3 keeps a column's config in its domain type and ships no configuration table, so on that database `stash db push` prints "Nothing to do." and exits 0. There is nothing to push for this schema. > > ```bash -> stash db push +> stash db push # EQL v2 + Proxy databases only > ``` > -> If this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected. The pending row will be promoted to active by `stash encrypt cutover` in the cutover step. +> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected, and `stash db activate` promotes the pending row to active. > -> SDK-only users can skip this step. +> SDK-only users can skip this step on EQL v3 (this section's case) — there is nothing to push. On a legacy EQL v2 column they cannot: `stash encrypt cutover` requires a pending EQL config, so an SDK-only v2 rollout must still run `stash db push` once before cutover (see the SDK-only note under Backfill below). #### Dual-writing: write to both columns from app code @@ -576,9 +576,14 @@ stash encrypt drop --table users --column email ``` The CLI emits a Drizzle migration file with the drop. For a v3 column it drops -the original plaintext column, `ALTER TABLE users DROP COLUMN email;` — there was -no rename, so no `email_plaintext` exists. Requires the column to be in the -`backfilled` phase, plus a live coverage check. +the original plaintext column, `email` — there was no rename, so no +`email_plaintext` exists. The generated SQL is not a bare `ALTER TABLE`: it is a +`DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, +re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply +time*, `RAISE EXCEPTION`s if any remain, and only then executes the +`ALTER TABLE ... DROP COLUMN`. So a row written after generation can't be +silently destroyed. Requires the column to be in the `backfilled` phase, plus a +live coverage check at generation time. Review and apply with `drizzle-kit migrate`, then update the schema to its final shape — the encrypted column is the only one left: diff --git a/skills/stash-dynamodb/SKILL.md b/skills/stash-dynamodb/SKILL.md index 2369d75b..af53d7fc 100644 --- a/skills/stash-dynamodb/SKILL.md +++ b/skills/stash-dynamodb/SKILL.md @@ -517,10 +517,11 @@ const result = await encryptionClient.encryptQuery( if (result.failure) throw new Error(result.failure.message) const hmac = result.data.hm // Use this in DynamoDB key conditions -// EQL v2 — pass queryType explicitly: +// EQL v2 — pass queryType explicitly (`usersV2` is the legacy table declared +// in the v2 read section above; `users` is the v3 one and infers its queryType): const v2Result = await encryptionClient.encryptQuery( "search-value", - { table: users, column: users.email, queryType: "equality" } + { table: usersV2, column: usersV2.email, queryType: "equality" } ) if (v2Result.failure) throw new Error(v2Result.failure.message) const v2Hmac = v2Result.data.hm diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 31f93990..3c780c1c 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -182,7 +182,7 @@ The SDK never logs plaintext data. | Import Path | Provides | |---|---| -| `@cipherstash/stack/v3` | `EncryptionV3` (a deprecated alias of `Encryption`), `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3 schema authoring. | +| `@cipherstash/stack/v3` | `Encryption` (the client factory), `EncryptionV3` (a deprecated alias of it), `typedClient`, `TypedEncryptionClient`, `EncryptionClientFor` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3 schema authoring. | | `@cipherstash/stack/eql/v3` | `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, `V3ModelInput`, ...) | | `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the `Encryption` function (typed for an all-v3 schema set; nominal for v2/loose schemas), legacy v2 re-exports | | `@cipherstash/stack/identity` | `LockContext` class and identity types | @@ -807,7 +807,7 @@ try { ## Rolling Encryption Out to Production -> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and auto-detects the column's version from its Postgres domain type — no flag. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). +> **EQL version note.** The rollout tooling (`stash encrypt *`, `@cipherstash/migrate`) works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v2** finishes with `stash encrypt cutover` (a rename swap plus a config promotion in `eql_v2_configuration`), then drops `_plaintext`. **v3 has no cut-over and no configuration table** — after backfill you point the application at `_encrypted` *by name*, verify reads, then `stash encrypt drop` generates the drop of the original plaintext ``. Running `encrypt cutover` on a v3 column safely reports "not applicable" with the next step. `stash db push`/`db activate` remain v2-only (they manage `eql_v2_configuration`). Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. @@ -838,7 +838,7 @@ Everything that lands in the repo and ships in **one** PR: | Schema-add | Migration adds `_encrypted` (nullable `jsonb`) alongside the existing plaintext column. Plaintext column unchanged; application still writes only plaintext. | | Dual-write code | Application now writes both `` and `_encrypted` on every persistence path that mutates the row, in the same transaction, on every code branch. Reads still come from the plaintext column. | -> **If you use CipherStash Proxy:** After the schema-add, run `stash db push` to register the new column in `eql_v2_configuration`. With no active config yet it writes directly to `active`; with an existing active config it writes `pending` (cutover will promote it). Required for Proxy-based queries. +> **If you use CipherStash Proxy (the EQL v2 path):** `stash db push` and `stash db activate` manage `eql_v2_configuration`, so they only apply to a database that has EQL v2 installed — on a v3-only database (the default) `db push` reports "Nothing to do." and exits 0, and `db activate` errors out. EQL v3 ships no configuration table, so a v3 rollout has nothing to push. On a v2 + Proxy database, run `stash db push` after the schema-add to register the new column. With no active config yet it writes directly to `active`; with an existing active config it writes `pending`, which `stash db activate` promotes to active (the v2 `stash encrypt cutover` also promotes it as part of its rename). Required for Proxy-based queries. **The dual-write definition matters.** "Writes both columns" is not enough. The rule is: every persistence path that mutates this row writes both columns, in the same transaction, on every code branch. A single missed branch — a CSV import, an admin action, a background job, a third-party webhook handler — means rows inserted in production after deploy land in plaintext only, and backfill won't catch them. Grep for every site that writes the plaintext column before declaring rollout complete. @@ -857,11 +857,11 @@ Once dual-writes are recorded as live in `cs_migrations`: | Action | What changes | |---|---| | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | -| Schema rename | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. | -| `stash encrypt cutover` | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. | -| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | -| Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | -| `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | +| Schema rename (**v2 only**) | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. **v3:** there is no rename — leave `_encrypted` under its own name and point the schema/queries at that name instead. | +| `stash encrypt cutover` (**v2 only**) | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes the `eql_v2_configuration` row `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. **v3:** cutover does not apply — on a backfilled v3 column it reports "not applicable" and exits 0 without changing anything; skip this row. | +| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw EQL payloads to end users (a `public.eql_v3_*` jsonb document on v3; an `eql_v2_encrypted` composite on a legacy v2 column). | +| Remove dual-write code | The plaintext column is no longer authoritative — **v2:** it is now `_plaintext` (the cutover renamed it); **v3:** it is still the original ``, since nothing was renamed. Either way, delete the dual-write logic once reads are served from the encrypted column. | +| `stash encrypt drop` | Emits a migration that drops the plaintext column — and *which* column that is depends on the generation. **v2** (precondition: phase `cut-over`): a plain `ALTER TABLE … DROP COLUMN "_plaintext"`. **v3** (precondition: phase `backfilled`): it drops the **original ``** — there is no `_plaintext` — and the generated SQL is a `DO` block that takes `ACCESS EXCLUSIVE` on the table, re-counts rows with `` set and `_encrypted` NULL *at apply time*, and raises instead of dropping if any remain. Apply with the project's normal migration tooling. | **Create the functional indexes between backfill and the read switch** (EQL v3 columns). After `stash encrypt backfill` completes and before reads move to the encrypted column, create the `eql_v3.*` extractor indexes for every queried capability (and `ANALYZE`) — one bulk build instead of per-row maintenance during backfill, and the switched reads engage an index from the first query. Recipes in the `stash-indexing` skill. (Legacy v2 rollouts have no extractor indexes to create — skip this step.) @@ -870,7 +870,7 @@ Once dual-writes are recorded as live in `cs_migrations`: Three sources of truth, kept separate on purpose: - **`.cipherstash/migrations.json`** (repo) — *intent*. Which columns the developer wants to encrypt and at which phase, code-reviewable. -- **`eql_v2_configuration`** (DB, EQL-managed) — *EQL intent*. Which columns are encrypted and with which indexes; drives the CipherStash Proxy. +- **`eql_v2_configuration`** (DB, EQL-managed) — *EQL intent*. **EQL v2 + Proxy only.** Which columns are encrypted and with which indexes; drives the CipherStash Proxy. EQL v3 encodes a column's config in its Postgres domain type and ships no configuration table, so a v3-only database has just the other two. - **`cipherstash.cs_migrations`** (DB, CipherStash-managed) — *runtime state*. Append-only event log: phase transitions, backfill cursors, error rows. Latest row per `(table, column)` is the current state. `stash encrypt status` shows all three side-by-side and flags drift (e.g. EQL says registered, the physical `_encrypted` column is missing). `stash status` (the quest log) rolls them up into the per-column "what's the next move" view used during a rollout. @@ -879,6 +879,51 @@ Three sources of truth, kept separate on purpose: ### CLI sequence for a single column +#### EQL v3 (the default) + +```bash +# Run this often — it's the canonical "where am I?" command. +stash status + +# ---- ENCRYPTION ROLLOUT (one PR, one deploy) ---- +# 1. Add the encrypted twin column via your normal migration tooling +# (drizzle-kit / supabase migrations / etc.). +# 2. Edit application code so every persistence path writes both +# `` and `_encrypted` in the same transaction, on every +# code branch. +# 3. Ship the PR to production. + +# ---- ⛔ DEPLOY GATE ---- +# Verify dual-writes are live, then redraft the plan for cutover work: +stash status +stash plan + +# ---- ENCRYPTION CUTOVER ---- +stash encrypt backfill --table users --column email +# Prompts to confirm dual-writes are live (or pass +# --confirm-dual-writes-deployed in CI). Resumable; SIGINT-safe. + +# Recovery — if dual-writes weren't actually live when backfill ran, +# re-run with --force to encrypt every plaintext row regardless. +stash encrypt backfill --table users --column email --force + +# Create the `eql_v3.*` extractor indexes for every queried capability +# and ANALYZE the table, via your normal migration tooling. Do this +# after backfill and before reads move over. Recipes: `stash-indexing`. + +# Point the application at the encrypted column BY NAME — +# `email_encrypted`. There is no rename and no `stash encrypt cutover` +# on v3. Wire the read paths through the encryption client so they +# decrypt, deploy, and verify reads return plaintext. + +# Then remove the dual-write code and drop the plaintext column. +# The generated migration re-checks coverage under a lock at apply +# time and refuses to drop if any plaintext-only row remains: +stash encrypt drop --table users --column email +``` + +#### EQL v2 (legacy) + > **Known limitation (v2):** `stash encrypt cutover` requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked separately. (EQL v3 columns never hit this — cutover doesn't apply to them.) ```bash @@ -919,9 +964,9 @@ stash encrypt cutover --table users --column email stash encrypt drop --table users --column email ``` -#### If you use CipherStash Proxy +#### EQL v2 + CipherStash Proxy -Register and promote encryption config at each phase: +Register and promote encryption config at each phase. This applies only to a database with EQL v2 installed — `eql_v2_configuration` is where `stash db push` writes, and a v3-only database has no such table (`db push` reports "Nothing to do."). A v3 rollout uses the v3 sequence above whether or not Proxy is in front of it. ```bash # Run this often — it's the canonical "where am I?" command. @@ -933,8 +978,9 @@ stash status # 2. Register the new encryption config with EQL: stash db push # First push (no active config yet) → writes directly to active. -# Subsequent push (active already exists) → writes pending; cutover -# will promote it. +# Subsequent push (active already exists) → writes pending, which +# `stash db activate` promotes — as does the v2 `stash encrypt +# cutover` below, as part of its rename. # 3. Edit application code so every persistence path writes both # `` and `_encrypted` in the same transaction, on every # code branch. diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index fdbb2d62..2cb45f9a 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -598,11 +598,11 @@ alias of its unsuffixed counterpart above. The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + switch reads to the encrypted column + drop — under EQL v3, which this section's schema uses, the switch is an application-side change with no rename; under legacy EQL v2 it is a rename). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. -> **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and auto-detects a column's version from its Postgres domain type — there is no flag. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). +> **EQL version note.** The `stash encrypt *` tooling works with **both EQL versions** and detects a column's generation from its Postgres domain type — there is no flag. Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else — a plaintext column, or a legacy `eql_v2_encrypted` one — classifies as *unknown* and falls through to the **v2** lifecycle, which is the correct default for a v2 column. Only a v3 column has its version recorded in the migration manifest. The lifecycles differ at the end: **v3** (the default, and what this section's schema uses) is `rollout → deploy gate → backfill → switch the app to the encrypted column by name → drop`, with **no cut-over rename**; **v2** finishes with `stash encrypt cutover` (a rename swap plus an `eql_v2_configuration` promotion) before the drop. Running `stash encrypt cutover` on a **backfilled** v3 column reports "not applicable" and exits 0 (it exits 1 if the backfill hasn't finished). -> **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. +> **Using CipherStash Proxy?** `stash db push` and `stash db activate` manage the `eql_v2_configuration` table, so they only apply to a database that has EQL v2 installed. EQL v3 ships no configuration table — on the v3-only database this section's schema assumes, `stash db push` reports "Nothing to do." and exits 0, and there is no cutover to run it before. If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) against a legacy EQL v2 database instead of the SDK, run `stash db push` after schema-add to register the encrypted column shape with EQL. > **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. @@ -658,13 +658,13 @@ export const users = encryptedTable('users', { }) ``` -> **Using CipherStash Proxy?** Register the new encryption config with EQL: +> **Using CipherStash Proxy?** `stash db push` registers the encryption config in `eql_v2_configuration` — an EQL v2 + Proxy artifact. The `public.eql_v3_text_search` column added above needs nothing registered: EQL v3 keeps a column's config in its domain type and ships no configuration table, so on a v3-only database `db push` prints "Nothing to do." and exits 0. > > ```bash -> stash db push +> stash db push # EQL v2 + Proxy databases only > ``` > -> If this is the project's first encrypted column, `db push` writes directly to the active EQL config. If an active config already exists, it writes the new config as `pending` — that's expected. Cutover (later) will promote it. +> On a legacy EQL v2 database: if this is the project's first encrypted column, `db push` writes directly to the active EQL config. If an active config already exists, it writes the new config as `pending` — that's expected, and `stash db activate` promotes it to active. > > **SDK users:** Skip this step. Your encryption config lives in app code. @@ -715,7 +715,7 @@ stash encrypt backfill --table users --column email # (CI: pass --confirm-dual-writes-deployed instead.) ``` -Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It auto-detects whether the column is EQL v2 or v3 and records that in `cs_migrations`. +Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. It detects a `public.eql_v3_*` column as EQL v3 and records that in `cs_migrations`; a legacy `eql_v2_encrypted` column does not classify and takes the v2 lifecycle with no version recorded. If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. @@ -732,10 +732,12 @@ finished). > supported.** `@cipherstash/stack-supabase` authors and queries EQL v3 only — > the v2 `encryptedSupabase({ encryptionClient, supabaseClient })` wrapper that > read the post-cutover `eql_v2_encrypted` column has been removed. The CLI -> lifecycle commands still auto-detect a v2 column, but the SDK read path for one -> is gone. If you have an in-flight v2 cutover, either pin the last -> `@cipherstash/stack-supabase` release that shipped the v2 wrapper to finish it, -> or (recommended) create an `eql_v3_*` twin and run the v3 rollout above. New +> lifecycle commands still handle a v2 column — detection is one-sided, so a v2 +> column classifies as *unknown* and falls through to the v2 lifecycle — but the +> SDK read path for one is gone. If you have an in-flight v2 cutover, either pin +> the last `@cipherstash/stack-supabase` release that shipped the v2 wrapper to +> finish it, or (recommended) create an `eql_v3_*` twin and run the v3 rollout +> above. New > encryption should always target an `eql_v3_*` domain. #### Drop: remove the plaintext column @@ -746,9 +748,9 @@ Once read paths are routing through the wrapper and you're confident reads are d stash encrypt drop --table users --column email ``` -The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version**, which the CLI auto-detects: +The CLI emits a Supabase migration file with the drop. **Which column it drops depends on the EQL version.** Detection is one-sided: a `public.eql_v3_*` domain classifies as **v3**; anything else classifies as *unknown* and falls through to the **v2** path: -- **v3** — drops the original plaintext column, `ALTER TABLE users DROP COLUMN email;`. There was no rename, so no `email_plaintext` exists. Requires the `backfilled` phase plus a live coverage check. +- **v3** — drops the original plaintext column, `email`. There was no rename, so no `email_plaintext` exists. The SQL is not a bare `ALTER TABLE`: it's a `DO $stash_drop$` block that takes `LOCK TABLE users IN ACCESS EXCLUSIVE MODE`, re-counts rows where `email IS NOT NULL AND email_encrypted IS NULL` *at apply time*, `RAISE EXCEPTION`s if any remain, and only then executes the `ALTER TABLE ... DROP COLUMN` — so a row written after generation can't be silently destroyed. Requires the `backfilled` phase plus a live coverage check at generation time. - **v2** — drops the post-rename leftover, `ALTER TABLE users DROP COLUMN email_plaintext;`. Requires the `cut-over` phase. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — the plaintext column is gone; only the encrypted column is written now, through the wrapper. @@ -782,7 +784,11 @@ Existing v2 deployments have two options: rollout in "Migrating an Existing Column to Encrypted" above. - **Stay on v2 for now:** pin the last `@cipherstash/stack-supabase` release that shipped the v2 wrapper. The CLI rollout tooling (`stash encrypt backfill` / - `cutover` / `drop`) still auto-detects a column's EQL generation. + `cutover` / `drop`) still drives a v2 column, but only because detection is + one-sided: a v2 column is never detected as v2, it classifies as *unknown* and + falls through to the v2 lifecycle. No EQL version is recorded for it in + `.cipherstash/migrations.json`, so `stash encrypt status` reports no version + for that column. For the removed v2 wrapper's historical API and semantics, see the docs at https://cipherstash.com/docs. diff --git a/vitest.shared.ts b/vitest.shared.ts index 56fee920..3b2c3554 100644 --- a/vitest.shared.ts +++ b/vitest.shared.ts @@ -33,6 +33,10 @@ export const sharedAlias: Record = { repoRoot, 'packages/test-kit/src/catalog.ts', ), + '@cipherstash/test-kit/install': resolve( + repoRoot, + 'packages/test-kit/src/install.ts', + ), '@cipherstash/test-kit/integration-clerk': resolve( repoRoot, 'packages/test-kit/src/integration/clerk.ts',