[SPARK-59147][SQL] Preserve struct nullness when SELECT * EXCEPT drops a nested field

[Vivek1106-04]

What changes were proposed in this pull request?

UnresolvedStarExceptOrReplace.filterColumns rewrites a struct column when the EXCEPT list
names a nested field: it extracts the retained fields and wraps them back into a
CreateStruct. CreateStruct resolves to CreateNamedStruct, which is never nullable, so
the rebuilt struct is always non-NULL even when the original struct expression evaluated to
NULL.

This PR guards the reconstruction with If(IsNull(col), Literal(null, dataType), ...) when
the original column is nullable, the same way UpdateFields.evalExpr already does for
DropField.

Why are the changes needed?

Dropping a nested field silently turns a NULL struct into a non-NULL struct whose remaining
fields are NULL. This is a silent correctness bug: nullable structs commonly come from
parsed JSON and from the null-producing side of outer joins, and afterwards IS NULL
checks, filters, joins, COALESCE and serialized output all behave differently.

WITH input AS (
  SELECT id,
    CASE
      WHEN id = 0 THEN CAST(NULL AS STRUCT<a: INT, b: INT>)
      ELSE named_struct('a', id, 'b', id + 10)
    END AS s
  FROM VALUES (0), (1) AS t(id)
),
actual AS (
  SELECT * EXCEPT (s.a) FROM input
)
SELECT i.id, i.s IS NULL AS before_except, a.s IS NULL AS after_except, a.s.b
FROM input i JOIN actual a USING (id) ORDER BY id;

Before:

0  true   false  NULL
1  false  false  11

After:

0  true   true   NULL
1  false  false  11

The same path backs the pipe |> DROP <col>.<field> operator, which had the same problem.

Does this PR introduce any user-facing change?

Yes. SELECT * EXCEPT (col.field) (and |> DROP col.field) now returns NULL for a row whose
col is NULL, instead of a struct of NULL fields. The result type is unchanged. This
corrects wrong results; there is no behavior change for non-nullable structs or for rows
whose struct is not NULL.

How was this patch tested?

Added cases to selectExcept.sql covering a NULL top-level struct, a NULL nested struct, and
struct expansion (SELECT data.* EXCEPT (s2.c)), and regenerated the golden files. The new
cases fail on master and pass with the fix. pipe-operators.sql analyzer results were
regenerated to reflect the added null guard; its query results are unchanged.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[tdcmeehan]

LGTM, thanks!

[uros-b]

The fix seems correct, minimal, and well-tested to me. Thank you @Vivek1106-04
Please ping @viirya and/or @cloud-fan for additional review!

[Vivek1106-04]

@viirya ! and @cloud-fan - requesting for a review. Thank You

[viirya]

The null-preserving fix makes sense when some fields remain. Could we clarify the intended behavior when all fields are excluded?

For s of type STRUCT<a: INT, b: INT>, SELECT * EXCEPT (s.a, s.b) currently produces an empty struct even when s is NULL. This patch changes that result to NULL, while the documentation says excluding all fields produces an empty struct.

I think preserving the empty-struct result is also reasonable: with no fields remaining, the result can be constructed without knowing the original value. Could we keep that behavior and apply the null guard only when fields remain? If changing this behavior is intentional, could we discuss the semantics explicitly and add documentation and tests covering both NULL and non-NULL inputs?

[Vivek1106-04]

Thanks @viirya — agreed, changed the patch accordingly.

You are right that the UpdateFields analogy in my comment does not cover this case: UpdateFields rejects dropping all fields with CANNOT_DROP_ALL_FIELDS, so it never produces an empty struct and offers no precedent. And the reasoning holds — with no fields remaining the result is a constant that does not depend on the input value, so there is no information to lose.

The latest commit applies the null guard only when fields remain:

• SELECT * EXCEPT (s.a) on a NULL s → NULL (the fix).
• SELECT * EXCEPT (s.a, s.b) → {} for both NULL and non-NULL s (unchanged, matches the docs).

Docs now state both rules explicitly, and selectExcept.sql covers the all-excluded case for NULL and non-NULL inputs, plus a nested struct with every field excluded (outer struct still NULL-preserving, inner one empty). The pipe-operators golden file for drop col.i1, col.i2 goes back to its original output.