Skip to content

fix(memorize): 丢弃已存在于 memcell 的重复转发消息 - #495

Open
flyto168 wants to merge 1 commit into
EverMind-AI:mainfrom
flyto168:feature/memorize-dedup-reposted
Open

flyto168 wants to merge 1 commit into
EverMind-AI:mainfrom
flyto168:feature/memorize-dedup-reposted

Conversation

@flyto168

@flyto168 flyto168 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

问题

同一条消息被重复投递时(聊天平台重发/重试),memorize 流程会把它当作新消息再次处理,产生重复的 memcell 内容。

改法

  • 新增 sqlite memorized_message 表与仓库:已在某 memcell 中的重复转发消息在进入 memorize 前被识别并丢弃。
  • 边界检测(boundary detection)接入该去重判断。

验证

  • memory 套件 750 passed(含新增 test_memorize_window_segmentation 覆盖);Ruff check/format 通过。
  • 基于上游 main @ a345d08,仅 8 文件。

范围

仅含通用修复与测试,不含本地环境文件。

The OpenClaw plugin re-posts a session's whole visible transcript on every
agent_end. A message that was already carved into a memcell has left
unprocessed_buffer, so the buffer's message_id PK no longer dedupes it, and
message_id is not stable anyway (its _idx is the position in the request).
Since 2026-10-07 each message landed in 5-18 memcells (35x for the main
session on 2026-10-13), atomic_fact grew from ~3k to ~34k rows a day, and
the derived load (unindexed rows, slow searches, cascade diffs) drove the
daemon into MemoryHigh repeatedly.

A new memorized_message table records (app, project, session, timestamp_ms)
for every item written into a memcell. buffer_messages and prepare_cells
drop fresh messages whose timestamp is already buffered or recorded before
boundary detection, so duplicates cost no LLM call either. Matching on the
timestamp keeps late routed messages that carry an older, never-memorized
timestamp (a high-water mark would have dropped 74 of them since 10-07).
Replaying 10-07..10-13 history: 84% of memcell items dropped, 0 duplicates
left among the kept ones.

scripts/backfill_memorized_messages.py seeds the table from existing
memcells (14,119 memcells -> 134,496 timestamps in ~9 s; idempotent).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flyto168 flyto168 changed the title fix(memorize): drop re-posted messages that are already in a memcell fix(memorize): 丢弃已存在于 memcell 的重复转发消息 Oct 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant