ClickHouse: Add fuzz target comparing the tokenizer with ClickHouse's lexer

[s5dsn-eqee]

Closes: #2645

Dialects: ClickHouse

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.29%. Comparing base (1aabd12) to head (098c7ef).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2646      +/-   ##
==========================================
+ Coverage   81.25%   81.29%   +0.04%     
==========================================
  Files          42       42              
  Lines       33910    33950      +40     
  Branches    33910    33950      +40     
==========================================
+ Hits        27553    27601      +48     
+ Misses       2792     2785       -7     
+ Partials     3565     3564       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaCappelletti94]

FYI, the ClusterFuzzLite error is due to duckDB shenanigans. Hopefully will be patched soon. Will review the PR shortly.

[LucaCappelletti94]

I would suggest the objectively non trivial option to go for a crate for the ClickHouse lexer. fuzz/build.rs fails when c++ is GCC with reference to 'std' is ambiguous at Lexer.cpp:526, which breaks --all-features, and the C ABI returns token kinds as bare numbers. The other oracle targets each depend on a crate (libsqlite3-sys, pg_query, libduckdb-sys), and https://github.com/LucaCappelletti94/sqlcipher-src is an example of a crate that tracks an upstream C codebase.

Such a crate should ship clickhouse-lexer's upstream/ files unchanged, compile them with clang++, and expose a safe Lexer yielding tokens with a TokenKind enum, along with the ClickHouse commit the files come from. Every update should run clickhouse-lexer's own test suite (test/test.js) on the shipped sources before release, next to the crate's tests. A scheduled workflow, like clickhouse-lexer's sync.yml for its own releases, should open the update PR by itself whenever the lexer changes, so nobody has to remember to bump it.

The best home is ClickHouse/clickhouse-lexer itself, and you could offer the crate there on ClickHouse/clickhouse-lexer#2. That issue has had no answer since September, so publishing from your own repository as clickhouse-lexer-sys and offering to transfer it later avoids blocking on them. This PR then keeps only the target and the docs, and the lone $ check below becomes kind == TokenKind::DollarSign.

I understand it is quite a bit of a challenge, and it is a reasonable ask only in this brave new world of LLMs given every step I described is rather mechanic and fully testable & verifiable. Would you be up for it?

[LucaCappelletti94]

You should pin the compiler to clang++. When c++ is GCC (the default on Ubuntu), cargo +nightly fuzz build --features clickhouse fails with reference to 'std' is ambiguous at Lexer.cpp:526.

Suggested change

	cc::Build::new()
	.cpp(true)
	cc::Build::new()
	// LexerStandalone.h declares a `std` namespace that GCC's predeclared one makes ambiguous.
	.compiler("clang++")
	.cpp(true)

But this is only relevant if we go for git submodules instead of a crate.

[LucaCappelletti94]

You should count a lone $ as rejected. ClickHouse lexes it as DollarSign, which only its JSONPath parser consumes, so no statement holding one parses.

Furthermore, ClickHouse 26.9 rejects SELECT $$, SELECT [$], SELECT 1 AS $ and SELECT * FROM t WHERE x = $, yet the target reports SELECT $$ as a tokenizer bug.

I tested an 8-minute -fork=6 run, and 9190 of the 46504 crashes were this case. A one-byte $ token is always DollarSign, and that held for all 46504 crashes.

Suggested change

	if clickhouse_lexer_token_is_end(token) != 0 {
	return true;
	}
	if clickhouse_lexer_token_is_end(token) != 0 {
	return true;
	}
	// A lone `$` lexes as DollarSign, which no ClickHouse SQL statement accepts.
	if token_end.offset_from(token_begin) == 1 && *token_begin == b'$' as c_char {
	return false;
	}

[LucaCappelletti94]

With the lone $ check, the doc comment should name it as it is hardly obvious.

Suggested change

	/// Whether ClickHouse's lexer reads `sql` to the end without an error token.
	/// Whether ClickHouse's lexer reads `sql` to the end without an error token or a lone `$`.

[s5dsn-eqee]

OK, it makes sense. Let's go with the separate lexer crate, then.

[LucaCappelletti94]

I will also be reviewing the crate itself as soon as I get enough time.

[s5dsn-eqee]

@LucaCappelletti94, does it make sense to you to add a rule to AGENTS.md that every tokenizer/parser bugfix also drops its triggering SQL into fuzz/fuzz_seeds/?

Smt like:

• One seed file per fixed bug, as small as still triggers the original path.
• Name it <dialect>_<issue>_<topic>.sql, e.g. clickhouse_2557_hash_comment.sql. For a fix with no issue, use the PR number.