Race condition: repo index concurrent with vault put crashes with ENOTEMPTY
Opened by swampadmin · 11/18/2024
Summary
Running swamp repo index concurrently with swamp vault put crashes with ENOTEMPTY when both processes attempt to rebuild the secrets symlink index simultaneously.
This is a separate trigger from #487 (which covered concurrent vault put only). The repo index path goes through rebuildAll → indexVault → refreshSecretsIndex, colliding with vault put's handleVaultSecretUpdated → indexVault → refreshSecretsIndex.
Reproduction
swamp init
swamp vault create local_encryption test-vault
swamp vault put test-vault SEED=x
# Run concurrently:
for i in 1 2 3 4 5; do
swamp vault put test-vault "KEY_${i}=val${i}" &
swamp repo index --json &
done
waitThe failure is intermittent but triggers frequently (first attempt in UAT).
Error
FTL error Error: Directory not empty (os error 66): remove '.../vaults/test-vault/secrets'
at async Object.remove (ext:deno_fs/30_fs.js:258:3)
at async SymlinkRepoIndexService.refreshSecretsIndex (symlink_repo_index_service.ts:786:9)
at async SymlinkRepoIndexService.indexVault (symlink_repo_index_service.ts:765:7)
at async SymlinkRepoIndexService.rebuildAll (symlink_repo_index_service.ts:360:9)
at async Command.<anonymous> (repo_index.ts:86:22)Stack Traces
Two code paths collide on refreshSecretsIndex:
vault put path:
handleVaultSecretUpdated → indexVault → refreshSecretsIndexrepo index path:
rebuildAll → indexVault → refreshSecretsIndexRoot Cause
refreshSecretsIndex performs a non-atomic remove + recreate of the secrets symlink directory. When repo index rebuilds the full index while vault put triggers its own index update, both enter refreshSecretsIndex and race on the directory contents.
Found By
Adversarial concurrency UAT in swamp-uat — tests/adversarial/concurrency_test.ts, test E ("swamp repo index concurrent with vault put does not crash or corrupt the index").
Closed
No activity in this phase yet.
Sign in to post a ripple.