Datastore lock: allow concurrent reads across different models
Opened by swampadmin · 10/13/2025
Problem
Currently the datastore uses a single global file lock. Running swamp model method run on one model blocks all other model method runs, even on completely unrelated models. This makes it impossible to run operations on different models in parallel (e.g. querying prod-vms while also running a workflow on a different model).
Context
When running fleet operations across 100+ VMs, each method call can take 30-90 seconds. Being unable to run a quick query on a different model while a long operation is in progress feels unnecessarily rigid.
Proposed Solution
Consider a more granular locking strategy:
- Per-model locks — each model instance gets its own lock, so unrelated models don't block each other
- Read/write locks — read-only operations (like
get,list) could run concurrently, with exclusive locks only for writes - Per-method locks — even within a model, independent methods on different resources could potentially run concurrently
The simplest improvement would be per-model locking, so that running a method on model A doesn't block an unrelated method on model B.
Current behavior
# Terminal 1: starts a 90-second healthCheck on prod-vms
swamp model method run prod-vms healthCheck --json
# Terminal 2: tries to check a different model — blocked until Terminal 1 finishes
swamp model method run dev-vms checkService --input service=docker --json
# ^ waits for datastore lock, eventually times outExpected behavior
Operations on different model instances should not block each other.
Closed
No activity in this phase yet.
Sign in to post a ripple.