Race condition in concurrent vault put: refreshSecretsIndex ENOTEMPTY
Opened by swampadmin · 11/15/2024
Summary
Concurrent vault put operations can fail with ENOTEMPTY when SymlinkRepoIndexService.refreshSecretsIndex performs a non-atomic remove + recreate of the secrets directory.
Reproduction
Run 5 concurrent vault put processes writing different keys to the same vault:
swamp init
swamp vault create local_encryption test-vault
# Run these concurrently (not sequentially):
swamp vault put test-vault KEY_1=val1 &
swamp vault put test-vault KEY_2=val2 &
swamp vault put test-vault KEY_3=val3 &
swamp vault put test-vault KEY_4=val4 &
swamp vault put test-vault KEY_5=val5 &
waitThe failure is intermittent — it depends on process scheduling timing.
Error
FTL error Error: Directory not empty (os error 66): remove '.../vaults/test-vault/secrets'
at async Object.remove (ext:deno_fs/30_fs.js:258:3)
at async SymlinkRepoIndexService.refreshSecretsIndex (symlink_repo_index_service.ts:786:9)
at async SymlinkRepoIndexService.indexVault (symlink_repo_index_service.ts:765:7)
at async SymlinkRepoIndexService.handleVaultSecretUpdated (symlink_repo_index_service.ts:215:5)Root Cause
refreshSecretsIndex removes the secrets directory and recreates it. When two processes run concurrently:
- Process A removes the directory
- Process B creates new files in the directory (or vice versa)
- Process B's remove fails with
ENOTEMPTYbecause Process A already wrote into it
This is a TOCTOU (time-of-check-time-of-use) race on the directory contents.
Suggested Fix
Replace the remove + recreate pattern with an atomic approach, e.g.:
- Build the new index in a temp directory, then atomic rename
- Or use a retry loop around the remove + recreate
Found By
Adversarial concurrency UAT in swamp-uat — tests/adversarial/concurrency_test.ts, test B ("swamp vault put to different keys concurrently preserves all keys").
Closed
No activity in this phase yet.
Sign in to post a ripple.