Add system(command; args) operator (disabled by default)

[Copilot]

Adds a system operator that executes an external command and returns its stdout as a string value. Disabled by default for security — requires explicit opt-in via --security-enable-system-operator.

Behaviour

• Disabled (default): returns an error ("system operations are disabled, use --security-enable-system-operator to enable"), consistent with --security-disable-env-ops and --security-disable-file-ops
• Enabled: runs the command, pipes the current node's YAML representation to stdin, returns stdout with one trailing newline stripped as a !!str

# Without flag: returns an error
yq '.country = system("/usr/bin/echo"; "test")' data.yml
# => Error: system operations are disabled, use --security-enable-system-operator to enable

# With flag: executes the command
yq --security-enable-system-operator '.country = system("/usr/bin/echo"; "test")' data.yml
# => country: test

# Array arguments
yq --security-enable-system-operator '.value = system("/bin/echo"; ["hello", "world"])' data.yml
# => value: hello world

# No arguments (stdin-only)
yq --security-enable-system-operator '.result = system("/usr/local/bin/my-filter")' data.yml

Note: When enabled, the system operator can replicate the functionality of env and load operators via external commands. Enabling it effectively overrides --security-disable-env-ops and --security-disable-file-ops.

Changes

• security_prefs.go — adds EnableSystemOps bool (default false)
• operator_system.go — operator implementation; supports system("cmd"), system("cmd"; "arg"), and system("cmd"; ["arg1", "arg2"]); captures stderr in error messages; validates command is a non-empty !!str scalar (rejects !!null, !!int, !!bool, etc.); evaluates command/args per matched node using SingleReadonlyChildContext; returns error when disabled (consistent with env/file operators); resolveSystemArgs returns ([]string, error) — errors on invalid arg types; logs debug when args expression returns multiple results
• operation.go — registers systemOpType
• lexer_participle.go — registers simpleOp("system", systemOpType)
• cmd/root.go — adds --security-enable-system-operator persistent flag (consistent with --security-disable-env-ops / --security-disable-file-ops)
• operator_system_test.go — covers disabled (error) state, enabled state, array args, per-node expression evaluation, null/integer command errors, command failure, and multi-document iteration; uses exec.LookPath for portability
• doc/operators/headers/system-operators.md — documentation header with security note about overriding env/file restrictions and corrected description of the ; separator

---

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

[Copilot]

Pull request overview

Adds a new system(...) operator to yqlib that can execute external commands (disabled by default) and wires it through parsing, CLI configuration, tests, and operator documentation.

Changes:

• Introduces SecurityPreferences.EnableSystemOps (default false) and a --enable-system-operator CLI flag to opt in.
• Implements system operator parsing/registration and execution logic (stdin = current node YAML, stdout -> string).
• Adds operator scenarios + generated documentation pages/headers.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pkg/yqlib/security_prefs.go	Adds security preference toggle for system operator.
pkg/yqlib/operator_system.go	Implements system operator execution and result mapping.
pkg/yqlib/operation.go	Registers SYSTEM operation type.
pkg/yqlib/lexer_participle.go	Adds system to the lexer/operator table.
cmd/root.go	Adds --enable-system-operator persistent flag.
pkg/yqlib/operator_system_test.go	Adds scenarios for disabled/enabled behavior and failures.
pkg/yqlib/doc/operators/system-operators.md	Adds generated operator docs with examples/output.
pkg/yqlib/doc/operators/headers/system-operators.md	Adds doc header snippet included by generator.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

command/argsExpression are resolved once using context.ReadOnlyClone() before iterating over context.MatchingNodes, and only the first match is used. This means system(<expr>) and system(<expr>; <expr>) cannot vary per matched node (e.g. system(.cmd; .args)), and may pick an unrelated first result when multiple nodes are matched. Consider evaluating the command/args expressions inside the per-candidate loop using context.SingleReadonlyChildContext(candidate) (and failing if they don't yield a single scalar/sequence) so parameter expressions behave consistently with other operators like load.

[mikefarah]

@copilot apply changes based on this feedback

[Copilot]

Done in da611f7. Command and args expressions are now evaluated inside the per-candidate loop using context.SingleReadonlyChildContext(candidate), matching the pattern used by loadOperator. This allows expressions like system(.cmd; .arg) to resolve .cmd and .arg against each matched node individually. Also added a test case for per-node evaluation, used exec.LookPath for portability, and fixed CRLF-aware newline trimming.

[Copilot]

These tests depend on hard-coded absolute paths like /usr/bin/echo and /bin/echo, which can fail on some Linux distros/containers and all non-POSIX platforms. Consider resolving the executable at runtime (e.g. via exec.LookPath) or skipping these scenarios when the required commands aren't available, to keep the test suite portable and reliable.

[Copilot]

SecurityPreferences currently uses a Disable* naming pattern (DisableEnvOps, DisableFileOps), but this introduces EnableSystemOps. For consistency and to reduce confusion when adding future prefs, consider renaming so all fields use the same polarity/pattern (e.g. DisableSystemOps with an inverted default, or rename the existing fields).

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

command is taken from the first matched node’s .Value without validating type/tag. If the command expression yields !!null or a non-scalar node, exec.Command will be invoked with an empty/invalid command (leading to confusing errors like “exec: no command”). Consider validating that the command node is a non-null scalar (and possibly erroring if multiple results are returned) and returning a clearer system operator: command must be a string scalar error.

[mikefarah]

@copilot apply changes based on this feedback If there are multiple results, just use the first (and log a debug, not a warning, of all the matches).

[Copilot]

Done in 53abbba. Extracted a resolveCommandNode helper that validates the command is a non-null scalar (returning "system operator: command must be a string scalar" if not) and logs a Debugf when multiple results are returned before using the first.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The PR description says stdout is "trimmed", but the implementation only strips a single trailing \n/\r\n. If the intent is to trim all trailing whitespace/newlines, use strings.TrimRight/TrimSpace; otherwise update the description/docs to match the current behavior ("strip one trailing newline").

[Copilot]

The doc-generating enabled scenarios embed echoPath from exec.LookPath, which can vary across environments (e.g. /usr/bin/echo vs /bin/echo) and cause non-deterministic generated docs. Consider hardcoding a stable command path/name for scenarios that are included in docs (or marking these scenarios skipDoc: true and adding a separate deterministic doc scenario).

[mikefarah]

@copilot no need to add enable-system-operator in this PR only to deprecate it. Also MarkPersistentFlagDeprecated doesn't exist anyway

[Copilot]

Done in b3b4478. Removed the deprecated --enable-system-operator alias and the MarkPersistentFlagDeprecated call entirely. The flag is now only --security-enable-system-operator, consistent with --security-disable-env-ops and --security-disable-file-ops. All references in the operator warning message, test yqFlags, and docs are updated accordingly.

[mikefarah]

@jandubois - I don't suppose you could run your deep review tool against this PR as well? That report was really neat..what are you using? Is this something you made?

[jandubois]

I don't suppose you could run your deep review tool against this PR as well?

I did run it: https://jandubois.github.io/yq/20260406-103045-pr-2640.html

There is also a Markdown version at https://jandubois.github.io/yq/20260406-103045-pr-2640.md if you want to copy some of it into a comment here, to feed back to Copilot. Be aware that the Markdown version includes the individual agent responses, so is quite verbose; you don't want to copy all of it. The HTML version is post-processed and has added code blocks and some other minor things.

Since this was another multi-pass review I now ran out of quota for Gemini for today. Unfortunately I only have a limited subscription for it and it looks like running 2 multi-pass reviews use up all of it. 😢

I can still run the tool, but have to run it with just Claude and Codex until 11pm...

That report was really neat..what are you using? Is this something you made?

Yes, it is my own tool running on "Zilicon Island", my off-leash sandbox (like Jurassic Park, except there are roaming AI agents instead of dinosaurs).

It has a lead agent preparing prompts for review agents using the top reasoning models, and then combining their findings. In a --thorough review it will run the review agents multiple times after adding the combined earlier findings to the prompt ("I already know about these; find me something else").

It takes a bit of time (you can see this latest review ran for 90 minutes wall time).

It also generates metrics and feedback that I use to continuously improve it.

You can see all reviews I ran against the yq repo at https://jandubois.github.io/yq/.

You can see that I ran the tool (single-pass only) against my oss-fuzz branches. It found issues in each of them, that I fixed before creating the PRs.

Those PRs were also all made with AI, but of course with some steering.

PS: Just realized that some of the code blocks have off-by-one errors in the line numbers and highlighted lines. So added a script to the tool to verify the code blocks, so the agent can check and fix it by themselves for future reviews. This is perpetual work-in-progress...

[jandubois]

Regarding I6. system operator subsumes env/file security restrictions I think it would be best not just to add the section to the docs, but to actually make --security-enable-system-operator mutually exclusive with either of --security-disable-env-ops or --security-disable-file-ops. It should throw an error when you try to use the former with either of the latter ones.

[mikefarah]

@copilot please fix according to the following feedback:

I1. Disabled mode silently injects nulls instead of erroring — pkg/yqlib/operator_system.go:63-71 (important, regression) [Claude Opus 4.6, Gemini 3.1 Pro]

if !ConfiguredSecurityPreferences.EnableSystemOps {
    log.Warning("system operator is disabled, use --security-enable-system-operator flag to enable")
    results := list.New()
    for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
        candidate := el.Value.(*CandidateNode)
        results.PushBack(candidate.CreateReplacement(ScalarNode, "!!null", "null"))
    }
    return context.ChildContext(results), nil
}

When the system operator is disabled (the default), it returns null per matched node with only a warning on stderr. The existing env and file operators return errors when their security gates are active (operator_env.go:20-22, operator_load.go:66-68). In a pipeline like yq '.field = system("cmd")' file.yml | kubectl apply -f -, a user who forgets the flag gets null values silently embedded in output with exit code 0. The warning goes to stderr, which pipelines routinely discard.

Fix: Return an error, matching the codebase convention:

 if !ConfiguredSecurityPreferences.EnableSystemOps {
-    log.Warning("system operator is disabled, use --security-enable-system-operator flag to enable")
-    results := list.New()
-    for el := context.MatchingNodes.Front(); el != nil; el = el.Next() {
-        candidate := el.Value.(*CandidateNode)
-        results.PushBack(candidate.CreateReplacement(ScalarNode, "!!null", "null"))
-    }
-    return context.ChildContext(results), nil
+    return Context{}, fmt.Errorf("system operations are disabled, use --security-enable-system-operator to enable")
 }

If the null-return behaviour is intentional, document the divergence from the env/file pattern in a code comment.

I2. Multi-result arg expressions silently truncated to first match — pkg/yqlib/operator_system.go:101-103 (important, regression) [Claude Opus 4.6, Codex GPT 5.4, Gemini 3.1 Pro]

if argsNodes.MatchingNodes.Front() != nil {
    args = resolveSystemArgs(argsNodes.MatchingNodes.Front().Value.(*CandidateNode))
}

When the args expression produces multiple results (e.g. system("echo"; .args[]) on {"args":["a","b"]}), only the first result is used; the rest are silently dropped. The command side (resolveCommandNode at line 50-51) logs a debug message for multiple results, but the args side does not. A user writing system("cmd"; .items[]) gets only the first item with no indication that the rest were dropped.

Fix: Add a debug log matching the command-resolution pattern, or flatten all matching arg nodes into argv:

+if argsNodes.MatchingNodes.Len() > 1 {
+    log.Debugf("system operator: args expression returned %d results, using first", argsNodes.MatchingNodes.Len())
+}
 if argsNodes.MatchingNodes.Front() != nil {
     args = resolveSystemArgs(argsNodes.MatchingNodes.Front().Value.(*CandidateNode))
 }

I3. Command validation accepts non-string scalars — pkg/yqlib/operator_system.go:53-55 (important, regression) [Codex GPT 5.4]

cmdNode := commandNodes.MatchingNodes.Front().Value.(*CandidateNode)
if cmdNode.Kind != ScalarNode || cmdNode.Tag == "!!null" {
    return "", fmt.Errorf("system operator: command must be a string scalar")
}

The check rejects !!null and non-scalars but allows !!int, !!bool, and other scalar tags through. system(123) tries to exec "123" rather than failing validation, contradicting both the error text and the documented contract. In practice the OS returns a "not found" error, but the validation should catch this before reaching exec.

Fix: Check the tag explicitly:

-if cmdNode.Kind != ScalarNode || cmdNode.Tag == "!!null" {
+if cmdNode.Kind != ScalarNode || cmdNode.guessTagFromCustomType() != "!!str" {
    return "", fmt.Errorf("system operator: command must be a string scalar")
 }

I4. Misleading documentation about semicolons for multiple args — pkg/yqlib/doc/operators/headers/system-operators.md:15 (important, regression) [Gemini 3.1 Pro]

- An argument or array of arguments separated by `;` (optional)

This suggests multiple arguments can be semicolon-separated: system("cmd"; "arg1"; "arg2"). In yq, ; is the block operator, which evaluates expressions sequentially and yields the right-hand result. Writing system("cmd"; "arg1"; "arg2") silently drops "arg1" and passes only "arg2". Multiple arguments require an array: system("cmd"; ["arg1", "arg2"]).

Fix:

-- An argument or array of arguments separated by `;` (optional)
+- An argument (or an array of arguments), separated from the command by `;` (optional)

I5. Argument resolution silently ignores invalid node types — pkg/yqlib/operator_system.go:23-26 (important, regression) [Gemini 3.1 Pro]

if child.Kind != ScalarNode || child.Tag == "!!null" {
    log.Warningf("system operator: argument must be a non-null scalar; got kind=%v tag=%v - ignoring", child.Kind, child.Tag)
    continue
}

When a sequence argument contains a non-scalar or null element, resolveSystemArgs logs a warning and skips it. Silently dropping command arguments alters execution: system("rm"; ["-rf", {bad: node}]) resolves to rm -rf, losing the invalid node without halting.

Fix: Return an error instead of continuing:

 if child.Kind != ScalarNode || child.Tag == "!!null" {
-    log.Warningf("system operator: argument must be a non-null scalar; got kind=%v tag=%v - ignoring", child.Kind, child.Tag)
-    continue
+    return nil, fmt.Errorf("system operator: argument must be a non-null scalar; got kind=%v tag=%v", child.Kind, child.Tag)
 }

This requires changing resolveSystemArgs to return ([]string, error) and updating its callers.

I6. system operator subsumes env/file security restrictions — undocumented — pkg/yqlib/doc/operators/headers/system-operators.md, cmd/root.go:213-215 (important, gap) [Claude Opus 4.6] (pass 2)

When --security-enable-system-operator is active, --security-disable-env-ops and --security-disable-file-ops become ineffective: system("printenv"; "SECRET") reads environment variables and system("cat"; "/path/to/file") reads arbitrary files. The documentation does not mention this.

Fix: Add a note to headers/system-operators.md:

 **Security warning**: The system operator is disabled by default. You must explicitly pass `--security-enable-system-operator` to use it.
+
+**Note:** When enabled, the system operator can replicate the functionality of `env` and `load`
+operators via external commands. Enabling it effectively overrides `--security-disable-env-ops`
+and `--security-disable-file-ops`.

I7. Multi-node execution path untested — pkg/yqlib/operator_system_test.go (important, gap) [Claude Opus 4.6] (pass 2)

All 7 test scenarios exercise the per-node loop (operator_system.go:79-145) with exactly one matched node. Result accumulation across iterations, per-node context isolation, and failure-on-Nth-node behaviour are untested.

Fix: Add a multi-document or multi-element test:

{
    description: "System operator processes multiple documents",
    skipDoc:     true,
    document:    "a: first",
    document2:   "a: second",
    expression:  `.a = system("` + echoPath + `"; "replaced")`,
    expected: []string{
        "D0, P[], (!!map)::a: replaced\n",
        "D1, P[], (!!map)::a: replaced\n",
    },
},

[mikefarah]

@jandubois it found some really good important issues! Thanks :)

[Copilot]

@copilot please fix according to the following feedback:

I1. Disabled mode silently injects nulls instead of erroring — pkg/yqlib/operator_system.go:63-71 (important, regression) [Claude O...

Done in 62d28d5. All seven issues addressed:

• I1: Disabled mode now returns fmt.Errorf("system operations are disabled, use --security-enable-system-operator to enable"), consistent with operator_env.go and operator_load.go.
• I2: Added log.Debugf when the args expression returns more than one result, mirroring the command-side debug log.
• I3: Command validation now uses cmdNode.guessTagFromCustomType() != "!!str" — rejects !!int, !!bool, !!null, and non-scalars with a clear error.
• I4: Updated header doc: "An argument (or an array of arguments), separated from the command by ; (optional)" — clarifies semicolons are not for listing multiple args.
• I5: resolveSystemArgs now returns ([]string, error). Invalid arg types (non-scalar, null elements in sequences, non-scalar single args) return errors instead of being silently ignored.
• I6: Added a note to headers/system-operators.md that enabling the system operator effectively overrides --security-disable-env-ops and --security-disable-file-ops.
• I7: Added a skipDoc: true test with two input documents verifying per-node loop accumulation across both matched nodes.