[jaideeppyne]

Which issue does this PR close?

• Closes Incorrect results: leaf expression pushdown duplicates volatile expressions (random() evaluated twice) #24678.

Rationale for this change

A column alias denotes one value per row, and WHERE p may only return rows for which p held for that row. Today the leaf-expression extraction passes break both:

SELECT s, s['a'] AS field
FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 3));

s['a']	field
0.5159112071865757	0.4514238291986653
0.0029104680074608646	0.28979983332288195
0.48542729227457915	0.04499392663566881

field is defined as s['a'] but differs from it on every row, because the plan is

Projection: named_struct(Utf8("a"), random()) AS s, random() AS field

The Filter form is worse — it returns rows that fail their own predicate:

SELECT bool_and(s['a'] > 0.5)
FROM (SELECT s FROM (SELECT named_struct('a', random()) AS s FROM generate_series(1, 1000))
      WHERE s['a'] > 0.5);
-- false; the predicate tested a different draw than the one in the returned `s`

Setting datafusion.optimizer.enable_leaf_expression_pushdown = false returns the correct answer in both cases, so the rewrite alone changes the meaning of the query.

Root cause. build_extraction_projection_impl merges an extraction into the input projection by resolving column references through build_projection_replace_map, i.e. by inlining each referenced column's defining expression. Inlining a volatile definition produces a second, independent evaluation. There was no volatility check in the file.

This is the same invariant FileScanConfig::try_swapping_with_projection already enforces for the physical projection-pushdown path via would_duplicate_costly_exprs (#23220) — the logical extraction path was missing it.

What changes are included in this PR?

• volatile_output_columns() — a projection's output columns whose definition is volatile.
• would_duplicate_volatile() — true when an extraction references one of them.
• The guard is applied at the three places that can merge into an input projection: extract_from_plan (pass 1: Filter/Sort/Limit/Aggregate/Join), split_and_push_projection (pass 2), and try_push_into_inputs (multi-input/Union routing). Each already had a "leave the plan alone" return path.

The guard is targeted rather than blanket: for ORDER BY s['a'] the extraction still happens, stacked above the volatile projection instead of merged into it, so the optimization is kept and the result is correct.

Relationship to #23691

@fornwall wondered on the issue whether #23691 already covers this. I checked out that branch and ran both shapes against it: SELECT s, s['a'] is incidentally fixed there, but WHERE s['a'] > 0.5 still duplicates random() and still returns rows failing the predicate. #23691 guards KeepInPlace compute cost in split_and_push_projection only; volatility is a separate concern (one duplication is already wrong, regardless of cost or placement) and pass 1 is a different code path. The two changes look independent to me and I believe they compose, but I'd appreciate a second opinion on that from whoever reviews #23691.

Are these changes tested?

Yes.

• datafusion/sqllogictest/test_files/projection_pushdown.slt — a new section beside the existing Projection pushdown into file scan duplicates non-deterministic functions (regression in 52.0.0) #23220 volatile section: two EXPLAINs pinning random() to a single occurrence, and two deterministic bool_and(...) correctness queries. All four fail on main and pass here.
• Two rule-level snapshot tests in extract_leaf_expressions.rs covering the pass-2 projection merge and the pass-1 Filter extraction, using a new test-only PlacementTestUDF::with_volatility().

cargo test -p datafusion-optimizer (796), cargo test -p datafusion --lib --tests (2049) and the full 504-file sqllogictest suite all pass; cargo fmt --all and cargo clippy --all-targets --all-features -- -D warnings are clean.

Are there any user-facing changes?

Queries that were silently returning wrong results now return correct ones. No API change. In the affected shapes the extraction is skipped, which can cost a small amount of column pruning — only when the referenced column is defined by a volatile expression.

---

_{Per the ASF generative-tooling policy and DataFusion's AI-assisted contribution guidance: this patch was prepared with AI assistance. The core idea is the one described above — the merge path inlines a referenced column's defining expression, which duplicates a volatile definition — and the open question about how this composes with #23691 is flagged deliberately rather than glossed over.}

[kosiew]

@jaideeppyne, thanks for working on this. The volatility guard looks well scoped, and the projection and filter regression coverage captures the original issue nicely. I have one non-blocking suggestion for covering the routed join-input path as well.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.07921% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.60%. Comparing base (a38bb10) to head (e07285a).
⚠️ Report is 66 commits behind head on main.

Files with missing lines	Patch %	Lines
...tafusion/optimizer/src/extract_leaf_expressions.rs	91.66%	0 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24720      +/-   ##
==========================================
+ Coverage   81.45%   81.60%   +0.14%     
==========================================
  Files        1120     1123       +3     
  Lines      401289   408997    +7708     
  Branches   401289   408997    +7708     
==========================================
+ Hits       326871   333758    +6887     
- Misses      55295    55627     +332     
- Partials    19123    19612     +489     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jaideeppyne]

I added the non-blocking join-side regression in e07285a and resolved the thread. The new sqllogictest asserts the extracted field still equals the same join side’s materialized s[a] when that side builds named_struct(a, random()) AS s. Both the optimizer unit target and focused projection_pushdown sqllogictest passed locally.

[kosiew]

🚀
@jaideeppyne
Thank you for your contribution.