Skip to content

Feat/git repo output sync queue#9790

Open
gatzjames wants to merge 8 commits intoKong:feat/git-repo-outputfrom
gatzjames:feat/git-repo-output-sync-queue
Open

Feat/git repo output sync queue#9790
gatzjames wants to merge 8 commits intoKong:feat/git-repo-outputfrom
gatzjames:feat/git-repo-output-sync-queue

Conversation

@gatzjames
Copy link
Copy Markdown
Contributor

Overview

Improves on the existing design of the repo-watcher with guarantees for data consistency and no race conditions when reading/writing to/from the db/fs.

Consistency

In order to achieve these guarantees we introduce a serial queue (not direction lock):

All sync work — both FS→DB and DB→FS — goes through one FIFO queue.
Tasks never run concurrently, eliminating all race conditions.
Throughput isn't a concern since sync events are infrequent.

No infinite loops

To ensure there are no write -> read -> write loops we use Content-hash dedup (not timing-based suppression):

After writing a YAML file to disk, record its SHA-256 hash.
When the watcher sees the file change, compare hashes — if it matches, skip (it's our own write).
This is writer-agnostic and handles external tools, git ops, and timing edge cases.

mtime as fast-path: Before computing content hash, we check the mtime first (cheap stat call). Only read + hash the file if mtime changed.

Other

  • Removes the project-nedb-client and completely uses the new sync mechanism for all git operations which completely decouples the git functionality from the app data storage.
  • The UI now refreshes when data on the repository (fs) change
  • Added an open folder button and copy repo path in the Project Settings view

@gatzjames gatzjames self-assigned this Apr 8, 2026
- Replace NeDB client with a unified disk client for all file operations.
- Introduce a serial queue to manage sync tasks and prevent race conditions.
- Implement content-hash deduplication to avoid unnecessary file imports.
- Add problem tracking for YAML files with conflicts or parse errors.
- Streamline watcher start/stop logic and improve notification handling.
- Ensure immediate DB→FS flush before git operations to maintain consistency.
- Enhance import logic to handle workspace deletions and renames effectively.
@gatzjames gatzjames force-pushed the feat/git-repo-output-sync-queue branch from bd6a38e to 16472ba Compare April 8, 2026 20:23
* pulls / checks out a branch that doesn't contain it.
*/
private async removeOrphanedWorkspaces(currentDiskFiles: string[]): Promise<void> {
const diskFileSet = new Set(currentDiskFiles.map(f => path.normalize(f)));
Copy link
Copy Markdown
Contributor

@aikido-pr-checks aikido-pr-checks bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - high severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix
Suggested change
const diskFileSet = new Set(currentDiskFiles.map(f => path.normalize(f)));
const base = path.resolve(this.repoDir);
const diskFileSet = new Set(currentDiskFiles.map(f => {
const target = path.resolve(f);
const rel = path.relative(base, target);
if (rel.startsWith('..') || path.isAbsolute(rel)) {
throw new Error('Invalid file path');
}
return path.normalize(f);
}));

More info

const nested = await this.collectYamlFiles(absPath);
result.push(...nested);
} else if (entry.isFile() && entry.name.endsWith('.yaml')) {
result.push(path.normalize(absPath));
Copy link
Copy Markdown
Contributor

@aikido-pr-checks aikido-pr-checks bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion attack via reading file - medium severity
If an attacker can control the input leading into the ReadFile function, they might be able to read sensitive files and launch further attacks with that information.

Show fix
Suggested change
result.push(path.normalize(absPath));
const resolvedBase = path.resolve(dir);
const resolvedTarget = path.resolve(absPath);
const relative = path.relative(resolvedBase, resolvedTarget);
if (relative.startsWith('..') || path.isAbsolute(relative)) {
continue;
}
result.push(resolvedTarget);

More info

Comment on lines +278 to 280
if (this.lastWrittenHash.get(absPath) === hash) {
continue;
}
Copy link
Copy Markdown
Contributor

@aikido-pr-checks aikido-pr-checks bot Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A timing attack might allow hackers to bruteforce passwords - low severity
An insecure way to compare passwords to user input might allow hackers to bruteforce passwords.

Show fix
Suggested change
if (this.lastWrittenHash.get(absPath) === hash) {
continue;
}
const lastHash = this.lastWrittenHash.get(absPath);
if (lastHash) {
const actual = crypto.createHash('sha256').update(hash).digest();
const expected = crypto.createHash('sha256').update(lastHash).digest();
if (crypto.timingSafeEqual(actual, expected)) {
continue;
}
}

More info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant