Repository navigation
fix(selection): end charwise and blockwise selections on a whole character - #367
Conversation
…ters A charwise or blockwise selection that ends on a multibyte character was cut at that character's first byte. selection_changed then carried invalid UTF-8, and Claude's WebSocket client dropped the connection. The new tests assert that the sent text is valid UTF-8 and equals the selection, including composing characters, for the live and the visual-exit paths. The six charwise and blockwise tests fail until the fix in the next commit. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
The fix in the next commit reads the end character of every charwise and blockwise selection with vim.fn.matchstr, so the existing selection tests need the mock as well. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…acter The '> mark, the "v" mark and the cursor point at a character's first byte, and the selection code cut the last line there. A selection that ended on a multibyte character then sent invalid UTF-8 in selection_changed, with an end column inside that character. A strict WebSocket client such as Node's ws closes the connection with 1006. Claude Code 2.1.312 stays connected but shows no selection. The end column now moves to the last byte of the end character, composing characters included, with vim.fn.matchstr. The sent text and the end column then match the selection that Neovim yanks. Columns stay 0-based byte offsets. Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…rule Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9742eaa30a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…cter With 'selection' set to exclusive, Neovim leaves out the character at the end of a selection of more than one position. The sent selection included that character. The two new tests for a backward live selection and a multi-line marks selection fail until the fix in the next commit. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
With 'selection' set to exclusive, Neovim leaves out the character at the end of a selection of more than one position. The end column now stops just before that character, which is also a character boundary. A selection of one position keeps its character, as in Neovim. Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
On Neovim 0.11 and later, a regex scan from the line start treats a Devanagari conjunct such as स्ते as one character, while `e` and `$` leave the cursor on its inner consonant त. matchstr with \%<col>c. then finds nothing at that column, so the selection was cut after the consonant's first byte and selection_changed carried invalid UTF-8. The mock now models that, and two new tests fail until the fix in the next commit. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…hend On Neovim 0.11 and later, `e` and `$` can leave the cursor on the inner consonant of a Devanagari conjunct, such as the त of स्ते. A regex scan from the line start treats स्ते as one character, so matchstr with \%<col>c. matched nothing there. The end column then stayed on that consonant's first byte, and selection_changed carried invalid UTF-8. The end now comes from vim.fn.matchend(line, "\_.", col - 1), which matches the character that starts at the end position. That is the character Neovim selects and yanks there, and the match always ends on a code point boundary. `\_.` also matches the NL that stands for a NUL byte. Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…er consonant With 'selection' set to exclusive, Neovim ends the selection after the character before the end position. On Neovim 0.11 and later, that character can be a whole cluster that reaches past the end position, such as स्ते before its त. The mock's matchend now models the scan from the line start, and the new test fails until the fix in the next commit. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…ore the end Neovim ends an exclusive selection after the character before the end position. On Neovim 0.11 and later, that character can be a cluster that reaches past the end position, such as स्ते before its त, and Neovim then yanks the whole cluster. The end column now comes from a match of the last character that starts before the end position, so it follows Neovim on every version. Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
nvim_buf_get_lines returns a NUL byte as "\0", and vim.fn passes such a Lua string as a Blob, so the matchend call raised E976 from the selection autocmds. The mock now raises the same way, and the new test fails until the fix in the next commit. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
nvim_buf_get_lines returns a NUL byte as "\0", and vim.fn passes such a Lua string as a Blob, so matchend raised E976 inside the selection autocmds. The line now goes to matchend with each NUL as NL, which is how Neovim stores NUL in a line. Both are one byte, so the byte offsets do not change. Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
… valid UTF-8 Part of #362. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…t is not valid UTF-8 A mark keeps its byte column when text before it on the line changes. After `gv`, the start or the end of the selection can then fall inside a multibyte character, and the selection text is not valid UTF-8. A buffer line can also hold bytes that are not valid UTF-8 (`:e ++bad=keep`). In both cases the client drops the connection. The vim mock now treats a byte that does not start a complete UTF-8 sequence as one character, as Neovim does. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
A selection position inside a multibyte code point, such as a stale mark after an edit and `gv`, now moves back to the code point's first byte. This applies to the start in get_visual_selection and get_visual_selection_from_marks, and to the end in selection_end_byte. The selection text then replaces each ill-formed UTF-8 sequence with U+FFFD, so no buffer content can make selection_changed or the selection tools send invalid UTF-8. Positions stay byte offsets into the buffer line. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
|
@codex review |
|
@codex security review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ed1fc6b1f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
On a line that is not valid UTF-8, Neovim treats a continuation byte that follows no lead byte as a character of its own. Selecting only that byte in `a<0x80>b` sent `a`, because the start and the end moved back to the `a`. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
…oint code_point_start now moves a column back only when the bytes before it start a well-formed UTF-8 sequence that covers the column. A lone continuation byte keeps its column, and valid_utf8 sends it as U+FFFD. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Re-recorded on the merge with main (#373, #367, #372). The facts and the model are unchanged by the merge, and check.sh passes. The only covered file main changed outside the record is selection.lua, which cannot reach diff state. Part of #363 _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_
Why
The goal is to clear the open follow-up issues (#362 now, #350, #326 and #335 earlier), one small reviewed PR per bug. Part of #362. This PR handles item 1.
A charwise or blockwise visual selection that ended on a multibyte character was cut at that character's first byte. The
'>mark, thevmark and the cursor all point at the first byte of a character, andselection.luaused that byte as the last byte of the selection.selection_changedthen carried invalid UTF-8, and its end column pointed inside the character.ws) closed with code 1006 at the issue's repro: the linenaïve café, then3G 0 v w e.The end column now comes from
vim.fn.matchend(line, "\_.", col - 1), the end of the character that starts at the end position. That is the character Neovim itself selects and yanks there, with its composing characters, and on Neovim 0.11+ with the rest of its grapheme cluster.matchstrwith a pattern anchored at the column (for example\%13c.), and that was not enough on Neovim 0.11+. There,eand$can leave the cursor on the inner consonant of a Devanagari conjunct (the त of स्ते), while a regex scan from the line start treats स्ते as one character and finds nothing at that column.selection=exclusive, a selection of more than one position ends after the character before the end position. On 0.11+ that character can be a cluster that reaches past the end position (स्ते before its त), as Neovim's own yank shows.matchendwith NUL as NL, becausevim.fnwould pass the Lua string as a Blob and raise E976.Linewise selections were already whole lines and do not change.
Two more inputs sent invalid UTF-8. The verifier saw both drop the harness client with code 1006 on Neovim 0.8.3, 0.10.4 and 0.12.3, and on main too:
cdonabcdef, press Esc, insertaéat the line start, then pressgv:'<and thevposition are now on the second byte ofé. With a backward selection, the stale position is the end instead. Such a position now moves back to the first byte of its code point, at both ends.:e ++enc=utf-8 ++bad=keepthat holds the byte 255. The selection text now replaces each invalid byte sequence with U+FFFD, the way a UTF-8 decoder does. The positions still count the bytes of the buffer line.What changed
lua/claudecode/selection.lua:selection_end_byte(line, start_coords, end_coords)gives the end column of charwise and blockwise selections.get_visual_selection(live updates) andget_visual_selection_from_marks(the flush on visual exit) use it.code_point_start(line, col)moves a column inside a well-formed multibyte code point back to that code point's first byte. A continuation byte outside such a sequence is a character of its own in Neovim and keeps its column. Both functions apply it to the start column, andselection_end_byteapplies it to the end column.valid_utf8(text)replaces each ill-formed UTF-8 sequence with U+FFFD (pure Lua). Both functions apply it to the text they return.selection_changed,getCurrentSelection,getLatestSelectionandgetOpenEditorsall send that text.tests/selection_test.lua: a base mock ofvim.fn.matchendfor the two patterns the helper uses. It models composing characters, a virama that joins the next consonant on 0.11+, E976 for a NUL byte, and a byte that does not start a complete UTF-8 sequence as one character. New tests assert that the sent text is valid UTF-8 and equals the selection, plus the columns. They cover:selection=exclusive, and a NUL byte日, a lone continuation byte, an overlong form and an encoded surrogatelua/claudecode/README.md: contract lines for the units, where a selection starts and ends, and the valid UTF-8 text.Scope
Only where a charwise or blockwise selection ends. The units do not change. The plugin sends 0-based UTF-8 byte offsets, and
PROTOCOL.mddoes not define the unit. VS Code positions count UTF-16 code units, so on a line with non-ASCII text the numbers differ from VS Code's. I did not check how Claude uses the columns.The start column comes from Neovim's marks and cursor, moved back to a code point's first byte. On Neovim 0.12.3 the start can be the inner consonant of a conjunct (
$vonनमस्ते). Neovim's own yank starts there too, so the plugin sendsते, the same as the yank. With a stale mark, the text differs from Neovim's yank on purpose:gvthenyyanks the raw bytes (\xa9ain the repro above), and the plugin sendséa.Known and not changed here: a selection that includes the line break (
v$, orlpast the last character) is sent without that line break, with end column = line length + 1. Main does the same. Blockwise selections are still sent as one contiguous span.Blast Radius
Users who select text that ends on a non-ASCII character: accented letters, CJK, emoji, Hindi and other scripts with conjuncts or combining marks. Their selection now reaches Claude whole. Before, a strict client dropped the connection, and the real CLI showed nothing. Users who press
gvafter an edit, or who select text in a file that is not valid UTF-8, keep their connection. Each charwise or blockwise update now makes onematchendcall. Each visual update also makes one pass over the selected bytes in Lua, which I measured at 0.3 ms for a 125,000-byte selection of accented text (item 8 below).Users with
selection=exclusivesee a change on plain ASCII text too. A selection now sends one character fewer than main did, the same text Neovim yanks. Main was wrong. For example,0vlonabcdefsentab(end 2) on main, while Neovim yanksa. This PR sendsa(end 1), on Neovim 0.8.3 and 0.12.3. In the verifier's matrix, about 530 cases per run differ from main for this reason.Verification
The verifier's grapheme matrix (its
hs.luaandanalyze.py): 65 cases onmb2.txt(accents, CJK, emoji,नमस्ते, ZWJ👩💻, flags🇩🇪🇫🇷, skin tone👍🏽,नमस्ते दुनिया). It runs real keys, live and on the flush, inclusive and exclusive, on Neovim 0.8.3, 0.9.5, 0.10.4 and 0.12.3, so 130 sends per row.879828add06671All 60 remaining fails on this PR fall in two groups. 56 are the line-break case above, where the text and the yank match except for the line break. 4 are
7G$von 0.12.3, where the text equals Neovim's yank (ते) but the analyzer's cluster split puts the start insideस्ते.verify-claudecode-nvim (agent-tty, real Neovim 0.12.3,
ws-client.mjsas Claude), with a typed buffer ofनमस्ते, ZWJ, flag, skin-tone andनमस्ते दुनियाlines.1G0vedrops the client, andws-client.logshowsclosed (code 1006).1G0veand the flush sendनमस्ते(end 18).1G$vsendsते, and1G$v0sendsनमस्ते. The ZWJ, flag and skin-tone selections all arrive whole.naïve café, CJK and emoji also passes.My headless matrix on Neovim 0.8.3, 0.9.5, 0.10.4, 0.11.6, 0.12.3 and 0.12.6: 28 of 28 checks pass on every version (main: 2 of 28). It compares each sent text with Neovim's
gvyyank, inclusive and exclusive.A NUL byte:
a^@b, then0vl, sendsa\0(end 2) inclusive andaexclusive on 0.8.3 and 0.12.3, without an error.Real Claude Code CLI 2.1.312 (earlier head
c9aff94, same inclusive code path for this line): main sends 11 bytes and the CLI shows no selection. This PR sends 12 bytes, and the CLI shows1 line selected.Unit tests: each test commit fails against the code before it, and the fix commit after it makes the test pass.
1b29be7: 6 of its 7 tests fail. Fixed byc9aff94.3b6853e: 2 of 3 fail. Fixed by8cc3080.2847498: 2 of 2 fail. Fixed by3eccc94.40bdfd0: 1 of 1 fails. Fixed byc786c9e.81073ad: 1 of 1 fails. Fixed by82d5197.17da7fa: 5 of 5 fail. Fixed by2ed1fc6.d8ede18: 1 of 1 fails. Fixed by445f73f.mise run check(0 warnings),mise run test(830 successes, 0 failures),stylua --check, andprettier --checkpass on the head commit.The verifier's break-it probes (its
hb.lua, plus four probes I added: a backward selection whose stale anchor falls on the middle byte of日, a forward one whose stale start does, and two selections that start on the lone0x80ofa<0x80>b). Real keys, live and on the flush, on Neovim 0.8.3 and 0.12.3, inclusive and exclusive, 44 records per run.a0c16cf0x80445f73fAfter the fix, the
aérepro sendséa(start 1, end 4), the backward stale case sendsx日(end 4),abc<255>defsendsabcU+FFFDdef(end 7), and the lone0x80alone sends U+FFFD (start 1, end 2).Cost of the UTF-8 pass, timed on the function itself, 50 runs, Neovim 0.8.3 and 0.12.3: 0.14 ms for 125,000 ASCII bytes, 0.27 ms for 125,000 bytes of accented text, and about 2 ms for 125,000 bytes where every fifth byte is 255. The long-line timings of
get_visual_selectioninhb.luastay within run-to-run noise.verify-claudecode-nvim, round 4 (agent-tty, real Neovim 0.12.3,
ws-client.mjsas Claude): theaéstalegv, the backward stalegvwithx日, and:e ++enc=utf-8 ++bad=keepon a file withabc<255>def.a0c16cf): the first stalegvdrops the client, andws-client.logshowsclosed (code 1006).doctorreports the client alive after each one.getCurrentSelectionreturnsabcU+FFFDdefwith end 7.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh