#224 S3 datastore stale-lock retry loop never evicts orphan locks left by swamp processes that panic at shutdown

Thanks @bixu for the detailed report. With the wider context across lab #213, #219, and the deploy CI hits, this turns out to be three independent problems compounding — let me break them apart.

Problem 1: stale-lock recovery isn't atomic — already fixed

The "infinite retry loop" / "Global lock released, proceeding with per-model locks" cycle is exactly what swamp-club#218 described, and it was fixed in commit 60829024 (PR #1291), which landed shortly before you filed this. Your version 20260504.140403.0-sha.d4c9188f is the commit immediately before that fix.

That commit wires the existing DistributedLock.forceRelease(expectedNonce) breakglass primitive into both wait loops in acquireModelLocks (src/cli/repo_context.ts) via a new tryForceReleaseStaleLock helper. Behaviour now:

• When the wait loop observes a stale global lock, the helper performs a CAS-style delete keyed on the lock's nonce. forceRelease re-verifies the nonce immediately before the deleteObject, so there's zero risk of clobbering a live re-acquire.
• The post-acquire TOCTOU re-check sees null instead of the same orphan, breaking the recursion that previously triggered "Global lock acquired by ... during per-model lock acquisition — releasing and retrying" forever.
• A regression test (acquireModelLocks - force-releases stale global lock instead of infinite-looping) was added to src/cli/repo_context_test.ts to lock this in.

Verified end-to-end against MinIO in the original PR: planted a stale lock object, ran a model method, observed clean exit in ~1.9s with .datastore.lock deleted from the bucket. Once you upgrade past 60829024 you'll stop hitting the loop, regardless of what's holding the orphan.

This addresses your suggested fixes #1 (atomic eviction via CAS) and #2 (process B no longer spins).

Problem 2: the Deno tls_wrap.rs panic — waiting on Deno 2.7.15

The Rust panic at ext/node/ops/tls_wrap.rs:2018:31 (called Option::unwrap() on a None value) is the same upstream Deno bug hitting lab #213, lab #219, our swamp-club deploy CI, and your kubevirt-dev run. Triggered when a detached or zero-length ArrayBuffer reaches the TLSWrap _writev / write_buffer code path — different swamp call sites tickle it (background update fetch, datastore push, kubectl-using extensions), but it's the same unwrap().

Already fixed upstream in denoland/deno#33737, merged 2026-05-01. The PR replaces the three unwrap() calls (lines 2000, 2018, 2068) with let-else that returns/continues gracefully, plus a regression test under tests/specs/node/tls_write_detached_buffer/. We're waiting on the next Deno release (likely v2.7.15, expected this week) and will rebundle swamp on top of it. That release closes out problem 2 across every trigger including yours.

This addresses your suggested fix #3 (the actual Deno-side investigation).

Problem 3: process stays alive after non-unwinding panic — residual concern

The "binary remained alive ~33 minutes" observation is independent of problems 1 and 2 and isn't fixed by either of the patches above. Rust prints "thread caused non-unwinding panic. aborting." but the swamp process stays in ps, holding whatever it was holding at the moment of panic. Even after Deno 2.7.15 lands and the specific tls_wrap panic is gone, anything else that panics during teardown could surface the same pattern. This sits at the Deno-runtime / event-loop boundary rather than in any extension.

This needs its own swamp-side investigation. We'll file a separate tracking issue for it once we've confirmed (post-2.7.15) that the panic-during-teardown class of bugs still leaves the process resident — that lets us scope the fix without conflating it with the upstream tls_wrap issue.

Closing this out

Going to close #224 as a duplicate — problem 1 is resolved on tip via #218, problem 2 is resolved by the pending Deno 2.7.15 bump, and problem 3 will get its own issue once we have a clean repro on a 2.7.15 build.

Immediate action for you: upgrade past commit 60829024 and the lock retry-loop will stop. Until Deno 2.7.15 ships, the panic itself can still fire against the kubevirt extension's TLS traffic, but with the lock-recovery fix in place it will no longer wedge subsequent invocations the way it did for you here.

Stronger repro of behavior #2 (the retry loop): once an orphan lock exists in S3, killing every local swamp process does not clear it. With ps -ax | grep swamp | grep -v grep returning empty (zero live swamp PIDs system-wide), the next invocation still spins:

datastore·lock Global lock held by "[email protected]" — waiting for structural command to finish
datastore·lock Global lock held by "[email protected]" appears stale (exceeded TTL of 30000ms) — proceeding
datastore·lock Global lock released, proceeding with per-model locks
datastore·lock Global lock acquired by "[email protected]" during per-model lock acquisition — releasing and retrying
[loops indefinitely]

So "kill the wedged process" is not a reliable workaround — the orphan lock persists in S3 even when no holder is alive anywhere. The only escape is whatever forces eviction of the S3 lock object itself.

This reinforces fix suggestion #1 in the original report: stale-lock detection must atomically replace the holder (CAS swap), not "proceed" while leaving the stale entry intact. Right now stale detection has no actual side effect on the S3 state — the entry is read, declared stale, and left in place.

What are the supported procedures for clearing an orphaned S3 lock? Is there a swamp datastore lock clear or similar, or do users need to manually aws s3 rm the lock object?

Found the actual mechanism behind the infinite retry loop: it's AWS SSO credential expiration during/before lock acquisition being misclassified as lock contention.

With fresh SSO creds, the same swamp model method run works normally. With expired SSO creds, the first S3.putObjectConditional call inside S3Lock.acquire throws CredentialsProviderError: Token is expired. That exception is bubbled up — but the calling layer in datastore_sync_coordinator.ts / repo_context.ts does not distinguish "auth failure" from "lock held by another caller", so it goes into the retry loop.

Fatal trace from a single, currently-failing invocation (no concurrent swamp processes anywhere on the host):

FTL error S3OperationError [CredentialsProviderError]: S3 putObjectConditional failed CredentialsProviderError — Token is expired. To refresh this SSO session run 'aws sso login' with the corresponding profile.
  at S3Client2.wrapError (.swamp/datastore-bundles/7c811b12/s3.js:55300:12)
  at S3Client2.run (.swamp/datastore-bundles/7c811b12/s3.js:55273:18)
  at async S3Client2.putObjectConditional (.swamp/datastore-bundles/7c811b12/s3.js:55392:7)
  at async S3Lock.acquire (.swamp/datastore-bundles/7c811b12/s3.js:55504:23)
  at async registerDatastoreSyncNamed (src/infrastructure/persistence/datastore_sync_coordinator.ts:269:7)
  at async acquireModelLocks (src/cli/repo_context.ts:810:5)
  ...
  [cause]: _CredentialsProviderError: Token is expired. To refresh this SSO session run 'aws sso login' with the corresponding profile.

Updated theory of behavior #2 (the retry loop, from the original report):

• When SSO creds expire at startup → fast fatal exit (above stack), good behavior.
• When SSO creds expire mid-execution (long-running model run, or token aged out between processes that share a daemon/cache) → S3Lock.acquire keeps throwing CredentialsProviderError, the lock layer treats those failures as transient lock contention, and the loop runs forever. The "stale lock" log lines we saw earlier weren't evidence of contention — they were the lock layer reading old state and then failing to write the replacement, indistinguishable in its logs from a real held lock.

Suggested fix narrows: in the lock-acquire retry path, distinguish CredentialsProviderError (and other auth/permission errors) from PreconditionFailed (the only condition that should mean "another caller holds the lock"). Auth errors should be fatal and surfaced immediately. The "Global lock released, proceeding" log line should also only fire when an actual eviction succeeded, not just when a read showed the entry as TTL-expired.

Workaround for users: when you see the retry loop, first check aws sso login status — odds are the credentials are expired, not that there's a real lock holder.