Repository navigation
Conversation
Signed-off-by: viiccwen <vicwen@apache.org>
There was a problem hiding this comment.
Code Review
This pull request updates the Parquet datasink to check if a filename ends with the ".parquet" extension rather than simply containing the substring "parquet", preventing incorrect behavior when the string is part of the filename stem. It also adds corresponding test cases to verify this behavior. The review feedback suggests making the extension check case-insensitive by converting the filename to lowercase before calling endswith, which prevents redundant extensions if a user provides an uppercase extension like ".PARQUET".
Jadiouo
left a comment
There was a problem hiding this comment.
LGTM. I reproduced #67155 locally (Ray nightly at 0fe2388, Python 3.13): with a filename like contains-parquet-{write_uuid}, the written file had no .parquet suffix (contains-parquet-<uuid>-0), and the same happened with an {i} template. With the parquet_datasink.py change applied, both get the .parquet suffix, and names that already end in .parquet are unchanged.
I also ran the updated test_parquet_write_uuid_handling_with_custom_filename_provider from this PR against 0fe2388, with and without the change:
- without the change: the four new
contains_parquetcases fail (4 failed, 16 passed) - with the change: 20 passed
So the new test cases catch the bug, and the fix resolves it. (I applied the change to 0fe2388 rather than checking out this branch.)
Description
Fix Parquet filename suffix detection for custom
FilenameProvidervalues. A filename that merely containsparquetin its stem is no longer treated as though it already has a.parquetextension.Related issues
Closes #67155.
Additional information