Skip to content

Isolate S3 CRT metric publishers from event loop - #7422

Open
jencymaryjoseph wants to merge 1 commit into
feature/master/S3CRTMetricPublisherfrom
jencyjos/s3-crt-metric-publisher-isolation
Open

jencymaryjoseph wants to merge 1 commit into
feature/master/S3CRTMetricPublisherfrom
jencyjos/s3-crt-metric-publisher-isolation

Conversation

@jencymaryjoseph

@jencymaryjoseph jencymaryjoseph commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation and context

S3CrtResponseHandlerAdapter.onTelemetry currently invokes customer-provided metric publishers on a CRT native callback thread. A slow or blocking publisher can therefore delay CRT event-loop processing and affect S3 request progress.

Modifications

CRT telemetry values are copied into an SDK MetricCollection while the callback is active, and publisher invocation is then moved to a client-scoped asynchronous dispatcher. The dispatcher uses one daemon worker and a bounded FIFO queue of 128 collections. This keeps publisher code off CRT callback threads, preserves collection acceptance order, and bounds memory usage. If publishers cannot keep up and the queue becomes full, new metric collections are dropped and a and warnings are logged at increasing drop-count thresholds to avoid log flooding.
The dispatcher follows the same general bounded single-worker queue pattern used by CloudWatchMetricPublisher, including the default queue capacity of 128. Its shutdown and drop-accounting behavior is specific to S3 CRT.

  • Added S3CrtMetricPublisherDispatcher with a single bounded queue and worker per S3 CRT client.
  • Submit completed metric collections without blocking the CRT callback thread.
  • Invoke configured publishers sequentially in their configured order.
  • Continue processing later collections if a publisher throws an exception.
  • Log queue-full, post-close, and shutdown-timeout metric drops at power-of-two thresholds.
  • Begin dispatcher shutdown after CRT native shutdown completes.
  • Drain accepted collections asynchronously for up to 10 seconds before discarding queued work.
  • Keep publisher lifecycle ownership with the caller; closing the S3 client does not close configured publishers.

Testing

Added unit tests covering callback-thread isolation, daemon execution, ordering, blocking publishers, bounded-queue drops, publisher exceptions, graceful and timed-out shutdown, asynchronous close, and post-close submissions.

Performance benchmarks are currently running. Will update once done.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All focused new and existing tests passed
  • I have added a changelog entry
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@jencymaryjoseph jencymaryjoseph changed the title Jencyjos/s3 crt metric publisher isolation Isolate S3 CRT metric publishers from event loop Oct 2, 2026
@jencymaryjoseph
jencymaryjoseph force-pushed the jencyjos/s3-crt-metric-publisher-isolation branch from e78ec3e to a96f3d6 Compare October 2, 2026 18:04
@jencymaryjoseph
jencymaryjoseph force-pushed the jencyjos/s3-crt-metric-publisher-isolation branch from a96f3d6 to 5790f46 Compare October 14, 2026 05:31
@jencymaryjoseph
jencymaryjoseph marked this pull request as ready for review October 14, 2026 18:47
@jencymaryjoseph
jencymaryjoseph requested a review from a team as a code owner October 14, 2026 18:47
@github-actions

Copy link
Copy Markdown

This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Oct 14, 2026

This branch has not been deployed

No deployments
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant