Feat/git repo output sync queue#9790
Feat/git repo output sync queue#9790gatzjames wants to merge 8 commits intoKong:feat/git-repo-outputfrom
Conversation
- Replace NeDB client with a unified disk client for all file operations. - Introduce a serial queue to manage sync tasks and prevent race conditions. - Implement content-hash deduplication to avoid unnecessary file imports. - Add problem tracking for YAML files with conflicts or parse errors. - Streamline watcher start/stop logic and improve notification handling. - Ensure immediate DB→FS flush before git operations to maintain consistency. - Enhance import logic to handle workspace deletions and renames effectively.
…ers and consolidating logic
bd6a38e to
16472ba
Compare
| * pulls / checks out a branch that doesn't contain it. | ||
| */ | ||
| private async removeOrphanedWorkspaces(currentDiskFiles: string[]): Promise<void> { | ||
| const diskFileSet = new Set(currentDiskFiles.map(f => path.normalize(f))); |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
| const diskFileSet = new Set(currentDiskFiles.map(f => path.normalize(f))); | |
| const base = path.resolve(this.repoDir); | |
| const diskFileSet = new Set(currentDiskFiles.map(f => { | |
| const target = path.resolve(f); | |
| const rel = path.relative(base, target); | |
| if (rel.startsWith('..') || path.isAbsolute(rel)) { | |
| throw new Error('Invalid file path'); | |
| } | |
| return path.normalize(f); | |
| })); |
| const nested = await this.collectYamlFiles(absPath); | ||
| result.push(...nested); | ||
| } else if (entry.isFile() && entry.name.endsWith('.yaml')) { | ||
| result.push(path.normalize(absPath)); |
There was a problem hiding this comment.
Potential file inclusion attack via reading file - medium severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.
Show fix
| result.push(path.normalize(absPath)); | |
| const resolvedBase = path.resolve(dir); | |
| const resolvedTarget = path.resolve(absPath); | |
| const relative = path.relative(resolvedBase, resolvedTarget); | |
| if (relative.startsWith('..') || path.isAbsolute(relative)) { | |
| continue; | |
| } | |
| result.push(resolvedTarget); |
| if (this.lastWrittenHash.get(absPath) === hash) { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
A timing attack might allow hackers to bruteforce passwords - low severity
An insecure way to compare passwords to user input might allow hackers to bruteforce passwords.
Show fix
| if (this.lastWrittenHash.get(absPath) === hash) { | |
| continue; | |
| } | |
| const lastHash = this.lastWrittenHash.get(absPath); | |
| if (lastHash) { | |
| const actual = crypto.createHash('sha256').update(hash).digest(); | |
| const expected = crypto.createHash('sha256').update(lastHash).digest(); | |
| if (crypto.timingSafeEqual(actual, expected)) { | |
| continue; | |
| } | |
| } |
…roject settings form
Overview
Improves on the existing design of the repo-watcher with guarantees for data consistency and no race conditions when reading/writing to/from the db/fs.
Consistency
In order to achieve these guarantees we introduce a serial queue (not direction lock):
All sync work — both FS→DB and DB→FS — goes through one FIFO queue.
Tasks never run concurrently, eliminating all race conditions.
Throughput isn't a concern since sync events are infrequent.
No infinite loops
To ensure there are no write -> read -> write loops we use Content-hash dedup (not timing-based suppression):
After writing a YAML file to disk, record its SHA-256 hash.
When the watcher sees the file change, compare hashes — if it matches, skip (it's our own write).
This is writer-agnostic and handles external tools, git ops, and timing edge cases.
mtime as fast-path: Before computing content hash, we check the mtime first (cheap stat call). Only read + hash the file if mtime changed.
Other
project-nedb-clientand completely uses the new sync mechanism for all git operations which completely decouples thegitfunctionality from the app data storage.