#269 Implement W4: KindAdapter + unified loader (extension catalog rearchitecture)

Problem

W1a (#1292), W1b (#1295), LockfileRepository prequel (#1298), W2 (#231 / closing swamp-club#201), and W3 (#252 / structurally closing the rebundle-loop bug class) put the foundation in place: domain aggregate, ExtensionRepository, lifecycle services, asymmetric unit-of-work, ReconcileFromDisk, freshness as aggregate query.

But the audit's second compounding cause — "the five loaders are duplicated by string substitution, not separated by abstraction" — remains. Every fix to bundleWithCache, importBundleByPath, findStaleFiles wrappers, bundleAndIndexOne, or buildIndex shells has a five-fold blast radius and a five-fold chance of being missed. PR #128 was "we forgot to do this in four places." PR #209/#1286 was again. swamp-club#214 is the latest instance, currently rippled and deferred to this workstream.

W4 collapses the five user_*_loader.ts files into one ExtensionLoader parameterized by KindAdapter. Per-kind concerns live in five KindAdapter implementations. The unified loader exists once; the bug class "we forgot in four places" becomes structurally impossible.

W4 also removes the W1b legacyStore escape hatch by migrating every remaining callsite to typed ExtensionRepository APIs. The field is explicitly flagged "REMOVE IN W4" in its JSDoc.

Full architectural context: design/extension-rearchitecture.md ("W4 — KindAdapter and unified loader" section) — referenced from #211.

Scope

Phase 1 — Audit + define KindAdapter interface

Audit the 5 user_*_loader.ts files; enumerate every per-kind concern. New src/domain/extensions/kind_adapter.ts interface captures:

• Datastore base path resolution
• Source directories configuration
• Kind-specific Zod schema for extension validation
• Registry registration callback (modelRegistry, vaultRegistry, etc.)
• Any other genuinely per-kind concern surfaced during audit

Audit-driven: don't lock in the interface before reading all 5 loaders. If a kind has truly unique behavior that doesn't fit the abstraction, surface BEFORE Phase 2 — do not shoehorn.

Phase 2 — Implement the unified ExtensionLoader

New src/domain/extensions/extension_loader.ts taking KindAdapter as a constructor field. Methods: buildIndex, loadSingleType, attachPendingExtensionsForType, bundleAndIndexOne, importBundleByPath.

The importBundleByPath ENOENT fallback (added for the model loader in PR #1288 for swamp-club#212) is baseline behavior across all kinds. The bug class swamp-club#214 names becomes structurally impossible because there is now ONE code path.

Phase 3 — Implement 5 KindAdapters

Five files in src/domain/extensions/:

• model_kind_adapter.ts
• vault_kind_adapter.ts
• driver_kind_adapter.ts
• datastore_kind_adapter.ts
• report_kind_adapter.ts

Each captures the previously-duplicated per-kind code. Each implements KindAdapter.

Phase 4 — Delete the 5 user_*_loader.ts files

After Phase 3, all 5 are dead. Delete them. Update every construction site in cli/mod.ts and auto_resolver_adapters.ts (post-W2's a-2 wiring established 8 loader construction + 2 repository construction sites — verify count in audit) to instantiate ExtensionLoader with the appropriate KindAdapter.

Phase 5 — Remove legacyStore escape hatch

Every remaining .legacyStore callsite migrates to a typed ExtensionRepository API. Methods added as needed to replace each callsite class. Once all migrated, delete the legacyStore: ExtensionCatalogStore field. Add a CI guard: grep .legacyStore in src/ → zero occurrences.

Pre-work decisions to pin in the PR description

1. KindAdapter interface scope. Audit-driven; if a per-kind concern doesn't fit, surface before locking in.
2. KindAdapter file layout. One file per kind. Keeps each kind's specifics isolated and W3-style "find all the X for kind Y" work mechanical.
3. legacyStore migration scope. Apply the LockfileRepository-prequel threshold pattern: audit .legacyStore callsite count; ≤ 7 callsites bundle into W4; > 7 callsites split as a refactor-prequel PR before W4 lands. Same mechanical-decision pattern that worked for LockfileRepository.
4. Construction-site migration. Wholesale substitution, not parallel implementations.
5. Test migration shape. Recommend: single shared test suite that runs against all 5 KindAdapters in turn, instead of 5 near-duplicate test files. Cuts ~80% of test code by line.

Out of scope (deferred to later workstreams)

• Per-fingerprint import URLs + subprocess test harness → W5
• swamp doctor extensions aggregate-state rendering → W6
• Bundle cache eviction (orphaned bundle files + Tombstoned catalog rows) → swamp-club#267

Success criteria

• 5 user_*_loader.ts files deleted; one ExtensionLoader + 5 KindAdapter implementations exist.
• legacyStore field removed from ExtensionRepository; CI guard asserts zero .legacyStore callsites in src/.
• importBundleByPath ENOENT fallback fires uniformly across all 5 kinds (closes swamp-club#214 structurally).
• swamp-uat self-recovery scenario (swamp-club#215, moved to swamp-uat — "missing cached bundle, intact catalog → self-recover") passes for all 5 kinds. Pre-W4 it passes for model only; post-W4 must pass for all five.
• Performance: cold-start time on 50-extension benchmark ≤ 1.2x post-W3 baseline (re-baseline against the W3-shipped numbers).
• All existing tests pass on Linux + macOS (Windows not a merge gate per W-series precedent), with the test migration consolidated per pre-work decision #5.
• Auto-ship-on-merge readiness verified via diversity-matrix soak.

Suggested test additions

• ENOENT fallback uniformity (the swamp-club#214 structural-fix verification): parameterized test running the same scenario over all 5 kinds — "bundle file deleted, catalog intact" → assert importBundleByPath rebundles + recovers. If this test exists post-W4, the bug class cannot regress.
• KindAdapter contract compliance: each KindAdapter implementation passes the same compliance test suite. No kind-specific quirks leak.
• Concurrent-kind loading: parallel loadSingleType calls across kinds don't race on shared state.
• Cold-start guard parity regression: post-W4 invalidationGuards behavior across kinds matches post-W1b/W3 baseline.
• .legacyStore grep guard: CI test or lint rule asserting zero .legacyStore occurrences in src/ outside extension_repository.ts.

Auto-ship-on-merge constraint

Same gates as W2/W3:

• CI green (all new + existing tests + type-check + lint + fmt)
• CI guard: grep .legacyStore in src/ → zero occurrences
• Author smoke on real repo: install + rm + upgrade exercised across multiple kinds; cold-start works
• Reviewer smoke on different real repo
• Diversity-matrix soak (multiple machines × OS × install shape × kind exercised)
  • Specifically watch for: any kind-specific behavior regression; cold-start performance changes; bundle-import edge cases on non-model kinds (the long tail this workstream addresses)
• swamp-uat self-recovery scenario passes for all 5 kinds
• Forward-only revert posture documented

Push-back encouraged

If the design doesn't fit the ground, surface before implementation. Specific watch list:

• KindAdapter interface might be too narrow. If a kind has genuinely unique behavior, do NOT shoehorn — surface, expand the interface, or carve the kind out. The audit confirmed the 5 loaders are mostly copy-paste, but small per-kind divergences may exist.
• legacyStore callsite count might surprise. If the audit reveals significantly more callsites than expected, apply the LockfileRepository threshold rule and split as a refactor-prequel.
• W3's reconcile path depends on per-kind APIs. If consolidating loaders changes calling shapes (e.g., loader.bundleAndIndexOne becomes loader.bundleAndIndexOne(kind, ...)), reconcile needs a small update. Verify before locking in.
• Construction sites might have hidden assumptions. Each loader is constructed with kind-specific deps today. If KindAdapter doesn't capture all of them, construction-site migration breaks. Audit cli/mod.ts construction sites in Phase 1.
• Test migration consolidation may be ambitious. If the per-loader tests have significant kind-specific assertions, shared parameterized tests may not cleanly absorb them. Pin the consolidation strategy in the audit.

The two most expensive misses to watch for

1. KindAdapter abstraction too narrow. A kind's behavior gets shoehorned, latent bug ships. Catch by: per-kind-concern audit in Phase 1 BEFORE interface design.
2. legacyStore callsite migration misses one. Field stays alive, W4's "deletion" claim is partial. Catch by: CI guard grepping .legacyStore in src/.

Inlined acceptance criteria (referenced from other issues in the body above)

The success criteria mention swamp-club#214 and swamp-club#215. Spelling those out so this issue is self-contained.

Folded-in: swamp-club#214 — importBundleByPath ENOENT fallback parity

Current bug shape:

• The model loader's importBundleByPath catches ENOENT when the cached bundle file is missing on disk, calls bundleAndIndexOne to regenerate the bundle, retries the import. Added in PR #1288 for swamp-club#212.
• The four sibling loaders (user_vault_loader.ts, user_driver_loader.ts, user_datastore_loader.ts, user_report_loader.ts) do NOT have this fallback. When their importBundleByPath hits a missing bundle file, it throws ENOENT and the user sees the type as unavailable until manual intervention.

Expected W4 behavior (baseline of the unified ExtensionLoader):

• One importBundleByPath code path. On ENOENT for the bundle file:
  1. Log a structured info-level message ("bundle missing, regenerating from source")
  2. Call bundleAndIndexOne for the source path
  3. Retry the dynamic import against the regenerated bundle
  4. Return the imported module
• Same code, same behavior, all 5 kinds.

Acceptance test (parameterized over 5 kinds): For each kind in [model, vault, driver, datastore, report]:

1. Build an extension of kind, install via the lifecycle service so a catalog row + bundle file exist.
2. Delete the bundle file on disk while leaving the catalog row intact.
3. Call extensionLoader.importBundleByPath(catalogRow.bundlePath).
4. Assert:
   • Call returns successfully (no ENOENT thrown)
   • Bundle file regenerated on disk
   • Imported module is functional (e.g., for a model, the model.run method is callable)
   • Structured log line emitted

If this parameterized test exists post-W4, the bug class cannot regress in any future copy-paste scenario.

Folded-in: swamp-club#215 — bundle-cache self-recovery UAT scenario

Test location: lives in ~/code/systeminit/swamp-uat, not in this repo. swamp-club#215 should be closed in Lab and moved to swamp-uat (per prior triage); this workstream's success criteria reference the swamp-uat scenario.

The user journey the UAT scenario covers:

1. User has a swamp repo with extensions installed across multiple kinds (e.g. @swamp/aws/ec2, @swamp/aws/s3, plus a local model under extensions/models/).
2. An external event removes a cached bundle file from .swamp/<kind>-bundles/ — could be filesystem corruption, accidental deletion, OS cleanup process, or a user troubleshooting attempt.
3. The catalog row remains intact, pointing at the now-missing bundle path.
4. User runs a swamp command that needs that extension's type (e.g. swamp model run @swamp/aws/ec2 list).

Expected behavior:

• swamp's import path detects the missing bundle on the affected extension.
• Automatically regenerates the bundle via bundleAndIndexOne.
• Completes the original operation successfully without surfacing the ENOENT to the user.
• No user intervention needed.
• Optionally surfaces a structured log line (info level) noting the recovery happened.

Acceptance:

• swamp-uat scenario passes for all 5 extension kinds.
• Pre-W4 the scenario passes for model only and fails for vault | driver | datastore | report (the swamp-club#214 bug).
• Post-W4 the scenario passes for all 5 kinds (the unified loader makes it impossible to fail for any single kind).

Performance baseline reference

The success criterion "≤ 1.2x post-W3 baseline" needs an inlined number for the planning agent. From W3's shipping benchmark:

• W3 post-merge cold-start time: 1.2s for 50 local models × 1 source each (per the implementer's reconcile_from_disk_bench.ts measurement)
• W3 post-merge warm-start no-op: 7ms

W4 acceptance:

• Cold-start (50 models): ≤ 1.44s (1.2 × 1.2s)
• Warm-start no-op: ≤ 8.4ms (1.2 × 7ms)

If either threshold is blown, optimize before shipping (per W3's pre-committed counter-strategy: fingerprint caching, mtime fast-path) or surface for redesign discussion.

References

• Predecessors: #211 (W1 tracking), #223 (W1b), #231 (W2), #252 (W3)
• Structurally closed by W4: swamp-club#214 (importBundleByPath ENOENT parity — folded into Phase 2's baseline behavior)
• UAT scenario this workstream must pass: swamp-club#215 (moved to swamp-uat as the self-recovery
• Bundle file naming may be affected by W5 (per-fingerprint URLs); coordinate if W5 starts before W4 ships
• Design doc: design/extension-rearchitecture.md