#520 Per-model LockTimeoutError at 60s causes cascading failures under concurrent access

Summary

Under concurrent access (25 parallel subagents hitting the same model instance), a single long-running model method or workflow step holds the per-model datastore lock for >60s, causing all other concurrent callers to receive LockTimeoutError. This cascades — once one process times out, the next queued caller also hits the 60s limit, creating a chain of failures.

Environment

• swamp version: 20260530.005533.0-sha.1c117111
• OS: Linux (WSL2)
• Single model instance of type @webframp/gitlab with ~45 extension methods
• Local filesystem datastore (no Postgres/S3)

Steps to Reproduce

1. Create a model instance and register ~45 extension methods on it.
2. Spawn 10+ concurrent swamp model method run <model> <method> calls targeting the same model with varying methods (some fast at ~1s, some slow at ~3.5s).
3. Meanwhile, spawn 3+ concurrent swamp workflow run calls that also target the same model across multiple steps.
4. Observe lock contention within 1-2 minutes.

Observed Behavior

WRN datastore·lock Waiting for lock "data/@webframp/gitlab/.../.lock"
  held by "user@host" (pid XXXXX, acquired XXXms ago)

FTL error LockTimeoutError: Lock "data/@webframp/gitlab/.../.lock"
  held by user@host (pid XXXXX) — timed out after 60130ms

A single workflow process (pid 4130092 in our test) held the lock across multiple minutes (several consecutive lock acquisitions), causing every other concurrent process to time out at 60s. The lock holder would release after ~3 rounds of lock acquisition (~3 minutes), at which point another queued process would take over and repeat the pattern.

Workflow runs wrapped this as:

FTL error Error: 'Workflow execution failed: Lock ".../.lock" held by user@host (pid XXXXX) — timed out after 60135ms'

Impact

• ~80% of concurrent model method calls failed during contention periods
• Entire workflow runs failed because any single step couldn't acquire the lock
• No automatic recovery — the contention only resolved when the lock-holding process naturally completed its cycle

Expected Behavior

• Lock acquisition should have a fair queue (FIFO) rather than everyone contending and timing out
• Workflow steps should share or release the lock between steps, not hold it across the entire workflow job
• A configurable timeout or retry mechanism would help
• Consider a read/write lock pattern for read-only methods