Skip to content

OPL/Columnar Query Engine Support some TextExpression variants#2586

Open
albertlockett wants to merge 10 commits intoopen-telemetry:mainfrom
albertlockett:albert/2578
Open

OPL/Columnar Query Engine Support some TextExpression variants#2586
albertlockett wants to merge 10 commits intoopen-telemetry:mainfrom
albertlockett:albert/2578

Conversation

@albertlockett
Copy link
Copy Markdown
Member

@albertlockett albertlockett commented Apr 7, 2026

Change Summary

OPL / Columnar Query Engine support the Concat, Join and Replace variants of the TextExpression. In all these cases, we parse from specially named functions:

logs | set attributes["x"] = concat("the", " attribute value ", "is: ", attributes["x"])
logs | set event_name = join(" ", "event happened:", event_name)
logs | set event_name = replace(event_name, "otel", "otap")

In each of these cases, we use the equivalent datafusion scalar function concat, concat_ws (for join) and replace.

Note: concat_ws is also used as an alias for join. I was thinking this'd be helpful for folks coming from a datafusion/SQL background. So it's equally possibly to write an expression like:

logs | set event_name = concat_ws(" ", "event happened:", event_name)

In planner.rs, I refactored the planning of function arguments into a reusable helper function.

What issue does this PR close?

How are these changes tested?

Unit

Are there any user-facing changes?

These new expression types are now supported via the transform processor.

Future work

Will add support for the TextExpression::Capture variant of this expression in future PR.

@github-actions github-actions bot added rust Pull requests that update Rust code query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches opl-parser Work items related to OPL Parser labels Apr 7, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 7, 2026

Codecov Report

❌ Patch coverage is 93.79433% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.47%. Comparing base (a20798a) to head (0edff6e).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2586      +/-   ##
==========================================
+ Coverage   88.35%   88.47%   +0.11%     
==========================================
  Files         613      618       +5     
  Lines      222680   226631    +3951     
==========================================
+ Hits       196752   200502    +3750     
- Misses      25404    25605     +201     
  Partials      524      524              
Components Coverage Δ
otap-dataflow 90.36% <93.79%> (+0.10%) ⬆️
query_abstraction 80.61% <ø> (ø)
query_engine 90.74% <ø> (ø)
syslog_cef_receivers ∅ <ø> (∅)
otel-arrow-go 52.45% <ø> (ø)
quiver 92.27% <ø> (+0.35%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@albertlockett albertlockett marked this pull request as ready for review April 7, 2026 20:57
@albertlockett albertlockett requested a review from a team as a code owner April 7, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

opl-parser Work items related to OPL Parser query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant