Skip to content

Find IVF cells through an index that a rename renames - #352

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ivf-rename-index
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:ivf-rename-index

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

After an IVF table is renamed, another IVF table cannot be created under its old name.

create virtual table t using vec0(v float[4] indexed by ivf(nlist=2));
alter table t rename to t2;
-- sqlite_master still has the index t_ivf_cells00_centroid, now on t2_ivf_cells00
create virtual table t using vec0(v float[4] indexed by ivf(nlist=2));
-- Could not create IVF shadow tables for column 0

ivf_create_shadow_tables() indexes _ivf_cellsNN by centroid with CREATE INDEX "<table>_ivf_cellsNN_centroid". vec0Rename() renames the shadow tables, but ALTER TABLE RENAME leaves an index's name as it was, and the index cannot be dropped and recreated from inside xRename: a DROP INDEX there fails with "database table is locked". The new table's CREATE INDEX then collides with the old one.

The cells table now declares cell_id INTEGER PRIMARY KEY, the rowid every statement already addresses cells by, and UNIQUE (centroid_id, cell_id). SQLite names that constraint's index after its table and renames it with the table, and the index serves the lookups by centroid as the old one did (the same query plans). Every statement on the cells names its columns, so a table created before this change opens and works here, and one created here opens and works on main. A table created before this change keeps its old index after a rename, but that index no longer blocks a new table, which creates none.

Test: test_rename_ivf_then_create_under_old_name in test-ivf-rename.py checks that no index on the renamed table's shadows is named for the old table, then creates a new table under the old name and reads each table's rows back. It fails on main and passes here. The Python suite passes (with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1, 302 passed, 4 skipped; without it, 239 passed, 67 skipped), under ASan and UBSan as well, and both builds' warnings are main's.

It merges cleanly with every open PR that touches the IVF code, #313, #343, #347 and #349 to #351 included.

Origin. Found in the IVF port to photostructure/sqlite-vec, photostructure#12 (65817f4), which lists it among the port's differences from upstream, with the same UNIQUE (centroid_id, cell_id) constraint; the change and its test are written here for asg017's main. No open or closed PR or issue here makes the same change.

ivf_create_shadow_tables() indexed _ivf_cellsNN by centroid with CREATE
INDEX "<table>_ivf_cellsNN_centroid", which ALTER TABLE RENAME leaves
under the old table's name, and xRename cannot drop it (DROP INDEX
there fails with "database table is locked"). After a rename, creating
an IVF table under the old name failed with "Could not create IVF
shadow tables for column 0". The cells table now declares cell_id as
its rowid and UNIQUE (centroid_id, cell_id), whose index SQLite renames
with the table and which serves the same lookups by centroid. Every
statement on the cells names its columns, so tables created before this
open as they did.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant