Skip to content

Improve rate-limit handling #1693

Open
Open
Improve rate-limit handling#1693
Labels
auto-triage-skipdiscussionM-T: An issue where more input is needed to reach a decision

Description

@kkerce
kkerce
opened on May 29, 2025

Reproducible in:

The Slack SDK version

slack-sdk==3.34.0
slackeventsapi==3.0.3

Python runtime version

Python 3.8.5

OS info

90~20.04.1-Ubuntu SMP Tue Apr 22 09:59:53 UTC 2025

Recently my Slack app continuously failed for a few hours in the following manner (US Eastern time).

2025-05-27 11:07:23,389 ERROR    slack_sdk.web.base_client Failed to decode Slack API response: Received a response in a non-JSON format: 
2025-05-27 11:07:23,389 ERROR    slack_sdk.socket_mode.builtin.client Failed to run a request listener: The request to the Slack API failed. (url: https://slack.com/api/team.info)
The server responded with: {'ok': False, 'error': 'Received a response in a non-JSON format: '}

The body of the response from the Slack API was empty.

Later that day, starting at 2025-05-27 14:16:56 and ending at 14:18:05 I ran the following commands seven times.

curl -vH "Authorization: Bearer <token-redacted>" https://slack.com/api/team.info; echo

The seventh run produced the following output.

*   Trying 54.92.199.186:443...
* TCP_NODELAY set
* Connected to slack.com (54.92.199.186) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=slack.com
*  start date: Mar 28 10:53:37 2025 GMT
*  expire date: Jun 26 10:53:36 2025 GMT
*  subjectAltName: host "slack.com" matched cert's "slack.com"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5632a8e5e0d0)
> GET /api/team.info HTTP/2
> Host: slack.com
> user-agent: curl/7.68.0
> accept: */*
> authorization: Bearer <token-redacted>
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 429
< x-edge-backend: envoy-www
< via: envoy-edge-iad-swgvnnyy
< x-envoy-ratelimited: true
< x-slack-edge-shared-secret-outcome: no-match
< date: Tue, 27 May 2025 18:18:05 GMT
< server: envoy
<
* Connection #0 to host slack.com left intact

Note in particular:

  • the empty response body
  • the x-envoy-ratelimited: true header

I described the problem to Slack support and in their prompt and helpful response they indicated the x-envoy-ratelimited: true header suggests this was handled via Slack's CDN or API gateway, which in rare cases may result in a 429 with an empty body instead of a JSON-formatted error. Requests are being throttled before reaching the Slack application layer.

The Python Slack SDK clearly doesn't handle this scenario well, e.g., the SDK's code is unable to honor any rate-limiting error handlers that are configured. Can the handling be improved?

Thanks.

Activity

  1. kkerce
    added
    untriaged
    on May 29, 2025
  2. kkerce
    changed the title [-](Set a clear title describing the issue)[/-] [+]Improve rate-limit handling[/+] on May 29, 2025
  3. mwbrooks
    added
    discussionM-T: An issue where more input is needed to reach a decision
    and removed
    untriaged
    on May 29, 2025

  4. mwbrooks commented on May 29, 2025

    @mwbrooks
    mwbrooks
    on May 29, 2025
    Member

    Hi @kkerce 👋🏻

    Thanks for the detailed issued and including the response from Slack support! 🙇🏻

    @kkerce Could you confirm that my understanding is correct: the Slack API response status code was 429 but the body was empty. This is because the CDN is handling the rate limit instead of the Slack API. Since the status code was 429, the SDK continually retried and always received an empty body. The correct approach would be for the SDK to error when the status code is 429 and there is a x-envoy-ratelimited: true header.

    @WilliamBergamin What do you think about extending the SDK's rate limit handler to support the CDN use-case that returns a x-envoy-ratelimited: true header?

  5. kkerce commented on May 30, 2025

    @kkerce
    kkerce
    on May 30, 2025
    Author

    Hi @mwbrooks and thanks for taking a look at the issue!

    Confirming that most of what you wrote above asking me to confirm is true, except I did not see any behavior from slack-sdk==3.34.0 to indicate it recognized the 429 and retried (despite that my client's retry_handlers is configured with RateLimitErrorRetryHandler(max_retry_count=2), and that I have historically seen retries from the SDK when the Slack API indicated 429 [as opposed to the CDN case]).

  6. WilliamBergamin commented on May 30, 2025

    @WilliamBergamin
    WilliamBergamin
    on May 30, 2025
    Contributor

    Interesting the RateLimitErrorRetryHandler should be retrying on every request with a status code 429 regardless of x-envoy-ratelimited: true 🤔

    @kkerce could you share how you are configuring the RetryHandler?

  7. kkerce commented on May 30, 2025

    @kkerce
    kkerce
    on May 30, 2025
    Author

    @WilliamBergamin In the following manner.

    from slack_sdk.web import WebClient
from slack_sdk.http_retry.builtin_handlers import RateLimitErrorRetryHandler

slackClient = WebClient(token="<redacted>")
slackClient.retry_handlers.append(RateLimitErrorRetryHandler(max_retry_count=2))

    Regardless of whether a retry handler was configured via the SDK, wouldn't we expect a different exception than the one I mentioned above?

    Failed to decode Slack API response: Received a response in a non-JSON format:

    As an aside but likely relevant for the near future: Because the circumstance seems relatively obscure and (my guess) doesn't occur often, ideally whoever ends up testing the SDK will need to somehow simulate rate-limiting at the level of Slack's CDN or API gateway, or otherwise convince Slack to configure a test scenario on their side at the CDN / API gateway level.

    Thanks for your help.

  8. seratch commented on Jun 2, 2025

    @seratch
    seratch
    on Jun 2, 2025
    Contributor

    Just my two cents: RateLimitErrorRetryHandler expects a "retry-after" response header, which tells you how long to wait before making the next request. However, in this case, the Envoy rate limiting situation does not provide that information, so I don't think the current retry logic works well. Additionally, I am not sure if retrying will help here, since it seems like a situation where Slack's backend infra (specifically Envoy proxy? or its backend?) is unable to handle a large number of requests.

  9. WilliamBergamin commented on Jun 2, 2025

    @WilliamBergamin
    WilliamBergamin
    on Jun 2, 2025
    Contributor

    We could attempt to implement a special case for this situation where we set a long retry after value such as 2 minutes when the response has a status code 429 and there is no "retry-after" response header 🤔

    But setting a long timeout like this as a default fallback could has unexpected behavior on tasks that are time sensitive, this value could also be surfaced as a configurable field

  10. github-actions commented on Jul 7, 2025

    @github-actions
    github-actionsBot
    on Jul 7, 2025 – with GitHub Actions · Hidden as outdated
  11. github-actions
    added
    auto-triage-stale
    on Jul 7, 2025

  12. kkerce commented on Jul 7, 2025

    @kkerce
    kkerce
    on Jul 7, 2025
    Author

    In the x-envoy-ratelimited: true header situation, can the Python Slack SDK at least simply recognize that it's a rate-limiting situation and not throw a Failed to decode Slack API response: Received a response in a non-JSON format: exception? The response in a non-JSON format exception will tend to lead developers down a path that's ultimately not helpful, e.g., the developer will need to investigate down to the "view all response headers" level (like I indicated here), only to find out it was a rate-limiting problem.

  13. github-actions
    removed
    auto-triage-stale
    on Jul 14, 2025

  14. github-actions commented on Aug 18, 2025

    @github-actions
    github-actionsBot
    on Aug 18, 2025 – with GitHub Actions · Hidden as outdated
  15. github-actions
    added
    auto-triage-stale
    on Aug 18, 2025

  16. kkerce commented on Aug 18, 2025

    @kkerce
    kkerce
    on Aug 18, 2025
    Author

    See this.

  17. zimeg
    added
    auto-triage-skip
    and removed
    auto-triage-stale
    on Aug 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    auto-triage-skipdiscussionM-T: An issue where more input is needed to reach a decision

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions