`test: $(TEST_DEPS)` unconditionally includes `install` in `TEST_DEPS` (`pgxntool/base.mk`), and `verify-results` depends on `test` in turn -- so both `make test` and `make verify-results` always filesystem-install the extension as a side effect, with no override flag (unlike `PGXNTOOL_ENABLE_TEST_BUILD`, which does have one for the test-build sanity check).
This is a real problem for "existing mode" testing (running the pgTAP suite via `pg_regress --use-existing` against a database whose extension was installed by something OTHER than `make install` -- e.g. binary `pg_upgrade`, or in our case, an extension registered via pg_tle instead of the filesystem). Calling into `test`/`verify-results` in that scenario silently filesystem-installs the extension as a side effect, which can defeat the entire point of a test that's specifically trying to prove the extension was deployed via a non-filesystem mechanism.
We worked around this in cat_tools (see `bin/test_existing`'s `TEST_EXISTING_DEPLOY=pgtle` mode, added in Postgres-Extensions/cat_tools#47) by calling `testdeps`+`installcheck` directly instead of `test` (skips the `install` prerequisite, since only `test` pulls it in) and invoking `verify-results-pgtap.sh` directly instead of `make verify-results` (skips the `test` prerequisite `verify-results` pulls in). It works, but it's a workaround that has to reimplement a chunk of what `test`/`verify-results` already do, and any pgxntool project hitting this same scenario has to rediscover and reimplement the same workaround.
Proposed fix: a new override flag, e.g. `PGXNTOOL_ENABLE_FS_INSTALL` (yes/no, default yes), mirroring the existing `PGXNTOOL_ENABLE_TEST_BUILD` pattern (`ifdef`/`override`/`pgxntool_validate_yesno`), gating just the `install` prerequisite in `test`'s `TEST_DEPS`. With that in place, existing-mode/pg_tle-style testing could just call `make test PGXNTOOL_ENABLE_FS_INSTALL=no ...`/`make verify-results PGXNTOOL_ENABLE_FS_INSTALL=no ...` directly, and the `testdeps`+`installcheck`+`verify-results-pgtap.sh` workaround in cat_tools' `bin/test_existing` could be simplified/removed.
Note: the name `PGXNTOOL_ENABLE_TEST_INSTALL` is already taken (controls the unrelated `test/install/*.sql` fixture-schedule feature) -- don't reuse it.
`test: $(TEST_DEPS)` unconditionally includes `install` in `TEST_DEPS` (`pgxntool/base.mk`), and `verify-results` depends on `test` in turn -- so both `make test` and `make verify-results` always filesystem-install the extension as a side effect, with no override flag (unlike `PGXNTOOL_ENABLE_TEST_BUILD`, which does have one for the test-build sanity check).
This is a real problem for "existing mode" testing (running the pgTAP suite via `pg_regress --use-existing` against a database whose extension was installed by something OTHER than `make install` -- e.g. binary `pg_upgrade`, or in our case, an extension registered via pg_tle instead of the filesystem). Calling into `test`/`verify-results` in that scenario silently filesystem-installs the extension as a side effect, which can defeat the entire point of a test that's specifically trying to prove the extension was deployed via a non-filesystem mechanism.
We worked around this in cat_tools (see `bin/test_existing`'s `TEST_EXISTING_DEPLOY=pgtle` mode, added in Postgres-Extensions/cat_tools#47) by calling `testdeps`+`installcheck` directly instead of `test` (skips the `install` prerequisite, since only `test` pulls it in) and invoking `verify-results-pgtap.sh` directly instead of `make verify-results` (skips the `test` prerequisite `verify-results` pulls in). It works, but it's a workaround that has to reimplement a chunk of what `test`/`verify-results` already do, and any pgxntool project hitting this same scenario has to rediscover and reimplement the same workaround.
Proposed fix: a new override flag, e.g. `PGXNTOOL_ENABLE_FS_INSTALL` (yes/no, default yes), mirroring the existing `PGXNTOOL_ENABLE_TEST_BUILD` pattern (`ifdef`/`override`/`pgxntool_validate_yesno`), gating just the `install` prerequisite in `test`'s `TEST_DEPS`. With that in place, existing-mode/pg_tle-style testing could just call `make test PGXNTOOL_ENABLE_FS_INSTALL=no ...`/`make verify-results PGXNTOOL_ENABLE_FS_INSTALL=no ...` directly, and the `testdeps`+`installcheck`+`verify-results-pgtap.sh` workaround in cat_tools' `bin/test_existing` could be simplified/removed.
Note: the name `PGXNTOOL_ENABLE_TEST_INSTALL` is already taken (controls the unrelated `test/install/*.sql` fixture-schedule feature) -- don't reuse it.