Skip to content

[data] Fix Parquet filename suffix detection - #67156

Open
viiccwen wants to merge 1 commit into
ray-project:masterfrom
viiccwen:fix-parquet-filename-suffix
Open

viiccwen wants to merge 1 commit into
ray-project:masterfrom
viiccwen:fix-parquet-filename-suffix

Conversation

@viiccwen

@viiccwen viiccwen commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fix Parquet filename suffix detection for custom FilenameProvider values. A filename that merely contains parquet in its stem is no longer treated as though it already has a .parquet extension.

Related issues

Closes #67155.

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

Signed-off-by: viiccwen <vicwen@apache.org>
@viiccwen
viiccwen requested a review from a team as a code owner October 19, 2026 07:21

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Parquet datasink to check if a filename ends with the ".parquet" extension rather than simply containing the substring "parquet", preventing incorrect behavior when the string is part of the filename stem. It also adds corresponding test cases to verify this behavior. The review feedback suggests making the extension check case-insensitive by converting the filename to lowercase before calling endswith, which prevents redundant extensions if a user provides an uppercase extension like ".PARQUET".

Comment thread python/ray/data/_internal/datasource/parquet_datasink.py
@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Oct 19, 2026

@Jadiouo Jadiouo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I reproduced #67155 locally (Ray nightly at 0fe2388, Python 3.13): with a filename like contains-parquet-{write_uuid}, the written file had no .parquet suffix (contains-parquet-<uuid>-0), and the same happened with an {i} template. With the parquet_datasink.py change applied, both get the .parquet suffix, and names that already end in .parquet are unchanged.

I also ran the updated test_parquet_write_uuid_handling_with_custom_filename_provider from this PR against 0fe2388, with and without the change:

  • without the change: the four new contains_parquet cases fail (4 failed, 16 passed)
  • with the change: 20 passed

So the new test cases catch the bug, and the fix resolves it. (I applied the change to 0fe2388 rather than checking out this branch.)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] Parquet writes treat parquet anywhere in a custom filename as a .parquet suffix

2 participants