Repository navigation
Conversation
dpitera
marked this pull request as draft
September 23, 2026 23:28
When CouchDB and Clouseau get out of sync, ken_server enters an infinite resubmit loop. Every 5 seconds it resubmits the affected shard, which respawns the dreyfus index updater, which fails again, causing sustained text_search_error 500s for all _find queries against that database with no recovery path. To fix this, we limit the max number of retries to a configurable number, defaulting to 5 in this commit. Note: the retry limit applies to all index types and not just search. Note: we use a new table with an independent lifecycle to ensure we preserve the intent of the code beforehand: skip_job should not block a live job. This commit does not address the root cause of the "out of sync" behavior between Couch and Clouseau; rather it simply mitigates customer impact and does not infinitely retry. Issues: https://github.ibm.com/cloudant/dbcore/issues/724
dpitera
force-pushed
the
prevent-infinite-retry
branch
from
September 24, 2026 04:10
d1948fb to
1253a4b
Compare
Member
|
"When CouchDB and Clouseau get out of sync, the current behavior is to retry infinitely" Please describe how this happens? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When CouchDB and Clouseau get out of sync,
ken_server enters an infinite resubmit loop.
Every 5 seconds it resubmits the affected shard, which respawns the dreyfus index updater, which fails again, causing sustained text_search_error 500s for all _find queries against that database with
no recovery path.
To fix this, we limit the max number of retries to a configurable number, defaulting to 5 in this commit.
This commit does not address the root cause of the "out of sync" behavior between Couch and Clouseau; rather it simply mitigates customer impact and does not infinitely retry.
Issues: https://github.ibm.com/cloudant/dbcore/issues/724
Overview
Testing recommendations
Related Issues or Pull Requests
Checklist
rel/overlay/etc/default.inisrc/docsfolder