Repository navigation
Observability refactor - update merge in current origin main, image fix - #776
Merged
Merged
Conversation
…merge the Prometheus pages Move the Grafana Tempo, Honeycomb, New Relic, Dash0 and Chronosphere export pages from Resolve > Traces > Exporting to Integrations > Observability > Traces, with redirects from the old paths. Fold the Prometheus V2 page into the Prometheus page, keep the v1 content as a deprecated legacy section and redirect /integrations/observability/prometheus-v2 to it. Add a one-page Exporting router under Resolve > Traces, drop an unreferenced screenshot, repoint inbound links and regenerate the sitemap.
Document every v2 metric with its labels and window semantics from the current API code, including the hourly reset at 01:30, the +Inf-only histogram buckets, the tag parsing rule, token rotation, rate limit and error codes. Add an OpenTelemetry Collector receiver recipe and point Dashboards readers at the endpoint. Correct the legacy v1 section from code and a live scrape, and add the fallback scrape protocol Prometheus 3 needs for it. The scrape config, every PromQL example and the collector block were run against a live account. One new screenshot of the integration card replaces the two outdated activation screenshots.
…d the Datadog, ClickStack and Cribl Stream pages The overview states the model in two sentences, routes readers with a decision table and lists coverage and limits. The OTLP page documents the span schema, trace headers, tag rules, transport facts and troubleshooting from the runner code, and carries the OpenTelemetry Collector receiver configuration that received real spans during verification. The three new vendor pages follow the thin template. Nav entries and a fresh Export Traces screenshot are included.
Coralogix: current key location and names, region versus team name, the attribute list deferred to the OTLP page, the custom trace header behavior, RUM key naming, and a fresh form screenshot. Vendor pages: one template with the endpoint, the key location, the header rows and where the spans appear, with values checked against each vendor's documentation. Grafana's third-party base64 site is replaced by a shell one-liner, the stale New Relic and Honeycomb paths are fixed, Dash0's trailing slash is gone and Chronosphere notes the Cortex XCOR rename.
The Exporting router lists every export page, the Resolve cards link the OTLP page, how-it-works gets a vendor-neutral Mermaid diagram and the four-type coverage wording with the span schema deferred to the OTLP page, and the custom headers section notes when CHECKLY_TRACEPARENT is empty. Link text on the heartbeat, autoscaling and dashboards pages matches the new Prometheus page title.
Tighten conditions the code imposes: when checkly_check_status is absent, the 0 reading of the HTTP status gauge, the 400 for non-boolean parameters, the cache refresh wait, revoked tokens working for up to ten minutes, the hourly window wording, the private location queue series sources, and tag keys that collide with built-in labels. On the OTLP page: what the URL check covers, header templating with environment variables and secrets, the instrumentation scope names, when checkly_debug appears, and the Coralogix custom header on Browser and Multistep checks. Reader fixes from review: a Collector row and limits on the overview, a fallback for unlisted backends, TLS in front of the sample receiver, and a troubleshooting entry for credentials in the URL.
# Conflicts: # integrations/observability/coralogix.mdx # integrations/observability/prometheus-v2.mdx
stefanjudis
previously approved these changes
Nov 3, 2026
stefanjudis
left a comment
Collaborator
There was a problem hiding this comment.
I skimmed the pages and checkt the redirects. LGTM.
hlenke
reviewed
Nov 3, 2026
hlenke
left a comment
Member
There was a problem hiding this comment.
Needs changes · The last merge from main left docs.json invalid JSON, so the Mintlify deployment and every CI job fail. Fix the redirect block at docs.json:2482 first, then re-run CI.
The content holds up well. I checked the Prometheus v2 schema, limits and labels and the OTLP export behavior against the current source, and almost everything matched. Two statements need a fix. The rest of the items below are small.
Merge readiness
- CI is red because
docs.jsondoes not parse.static-docs-checksfails withSyntaxError: Expected ',' or '}' after property value in JSON at position 99108 (line 2483 column 7).Mintlify Deploymentfails for the same reason, andpreview-checksanddocs-gatefail because the deployment failed. Theprometheus-v2redirect object has lost its closing},and the next object's opening{, so it runs straight into the/platform/rocky-airedirect. There is a one-click suggestion inline. No preview was built, so nothing on this branch has been rendered yet. - Mergeable, no conflicts.
mergeStateStatusisBLOCKEDonly because of the failing checks. - Overlap. #446 and #449 (Mintlify bot) both edit
integrations/observability/prometheus-v2.mdx, which this PR deletes. Land this PR first, then close those two or rebase what is still useful ontoprometheus.mdx. #774 (draft, sentence case) touches the moved pages too. The PR description already covers that one. - Redirects and inbound links (
pr_impact.py, after patching the JSON locally so it could run): all six removed pages have a redirect indocs.json. No link in the head points at a removed page, and no new broken links or anchors were found.#heartbeat-metricssurvives on the merged page.
Outside the diff
- Important:
platform/private-locations/agent-metrics.mdx:51is now onmainand still links[Prometheus V2 integration](/integrations/observability/prometheus-v2/). The redirect covers it. The PR description says #755 is still open, but that page has since landed, so the follow-up you planned can go in this PR. Point the link at/integrations/observability/prometheusand add the one sentence on the Prometheus page that links to agent metrics.
Inline comments
1 blocker, 2 important, 1 wording.
What I checked
- Diff
54c23c8d..bd38402c(merge base = currentorigin/main), every text file read top to bottom. Sitemaps and PNGs were reviewed separately. - CI:
gh pr checks 776and the failed logs of runs 41640322922 (static-docs-checks), 41640322970 (docs-gate) and 41640322895 (preview-checks).gh pr viewfor mergeability. Overlap fromgh pr list --json number,files. pr_impact.py: it crashed on the head snapshot's invaliddocs.json. I added the missing},{to the cached snapshot only (not the repo) and re-ran it, which produced the redirect and link results above.- Prometheus v2, against monorepo
origin/main@28c905592a:- The endpoint's route definition: the three query flags, all default
false, validated as booleans. The hourly window offset is 90 s, which gives the HH:01:30 reset. - API config defaults: 50 requests per 60 s for v2 and 300 per 60 s for v1. The ClickHouse query timeout is 25 s.
- The v2 controller: 60 s cache, gzip only when the scraper asks for it, one cache entry per parameter combination.
- The v1 module: 30 s cache, plus a 600 s in-memory token cache, which supports "revoked token can work for up to 10 minutes".
- The token service caps an account at 2 tokens.
- The v2 golden output (
__goldens__/prometheus-v2.bb-spec.ts/active-account.txt):typevalues for web vitals, browser errors and API timings, base labels,groupand tag labels.
- The endpoint's route definition: the three query flags, all default
- OTLP export, same commit:
- Go API/URL runner:
checkly-tracerscope, the 5 s export timeout, OTLP/HTTP protobuf, header values rendered from environment variables, andcx.*resource attributes whenever the Coralogix integration is on. - Browser and PWCS runners: the
tracestatemembers includingcheckly_debug=trueon ad hoc and test-session runs, thecheckly-browser-tracerscope, span kindCLIENTwith status alwaysOK,service.name=checkly. - Playwright Check Suites build a span too, but only for the Checkly-owned integration (
CHECKLY_OTEL), not for the customer export (HOSTED_OTEL). The "four check types" statement holds.
- Go API/URL runner:
- UI labels against the webapp i18n and component tests: Create Prometheus endpoint, Bearer tokens, Add token, Add Coralogix integration, Enable sending traces to Coralogix, Custom trace header, Enable Coralogix RUM, RUM Region, RUM application name, RUM API key, Enable exporting traces, Add header, + Add more channels, New alert channel, Save Coralogix webhook. All match.
- Vendor docs:
- https://docs.datadoghq.com/opentelemetry/setup/otlp_ingest/traces/:
dd-api-key,compute_stats, http/protobuf. - https://docs.honeycomb.io/send-data/opentelemetry/: US and EU URLs,
x-honeycomb-team, dataset header only for Classic. - https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/: US, EU, JP and FedRAMP hosts,
api-key, 1 MB limit. - https://docs-xcor.paloaltonetworks.com/ingest/metrics-traces/otel/otlp-endpoints:
/data/opentelemetry/v1/tracespath andAPI-Tokenheader.
- https://docs.datadoghq.com/opentelemetry/setup/otlp_ingest/traces/:
- Screenshots: I read
cx_otel_int1.png,prometheus_integration_card.png,dash0-token.pngandotel_export_traces_settings.png(new or changed), plus the reusedprometheus_step1.pngandprometheus_v2_step2.png. Tokens are masked and I found no PII. One reused image is outdated (inline). - No end-to-end tests. The claims are about the production Prometheus endpoint, the trace export pipeline and third-party intakes, which the local app cannot show. I verified against source instead.
- Could not verify:
- Datadog's per-site OTLP hosts in the table (the Datadog page uses a placeholder).
- That Chronosphere hosts stay
<instance>.chronosphere.io. The XCOR page saysTENANTand does not mention the rename. - ClickStack, Cribl Stream and Dash0 setup steps.
- The private location queue sampling windows and the 250 s agent window.
- Coralogix's current API-key menu path.
Comment on lines
+2482
to
2483
| "destination": "/integrations/observability/prometheus" | ||
| "source": "/platform/rocky-ai", |
Member
There was a problem hiding this comment.
Blocker: The merge dropped the end of this redirect object and the start of the next one. docs.json no longer parses, which fails the Mintlify deployment and static-docs-checks (Expected ',' or '}' after property value ... line 2483 column 7).
Suggested change
| "destination": "/integrations/observability/prometheus" | |
| "source": "/platform/rocky-ai", | |
| "destination": "/integrations/observability/prometheus" | |
| }, | |
| { | |
| "source": "/platform/rocky-ai", |
|
|
||
| - **Caching and limits.** Responses are cached for 60 seconds, so a scrape interval below 60 seconds fetches the same data. Responses are gzip compressed when the scraper asks for it, which Prometheus does by default. The endpoint allows 50 requests per minute per account. Above that it returns `429` with `Retry-After`, `X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers. If you run multiple Prometheus instances, each instance scrapes independently. Align their scrape intervals where possible. | ||
| - **Errors.** An unknown token or a lapsed account returns `401`. A revoked token can keep working for up to 10 minutes. Tokens do not expire. A plan without the Prometheus integration returns `403`. | ||
| - **Timeouts.** A scrape fails as a whole if any part of the backing query fails. The query times out after 25 seconds, and a scrape that arrives while another one is refreshing the cache waits up to 20 seconds for it, which is why the sample sets `scrape_timeout: 30s`. |
Member
There was a problem hiding this comment.
Important: The 20-second wait no longer matches the code. On monorepo main, a scrape that arrives during a cache refresh waits on the regeneration lock for up to 45 seconds. That timeout was raised on 2026-10-28, see apps/backend/api/src/modules/public-api/prometheus-v2/PrometheusV2Controller.ts. So the wait does not explain the 30 s scrape_timeout anymore. I would drop the number rather than chase it:
Suggested change
| - **Timeouts.** A scrape fails as a whole if any part of the backing query fails. The query times out after 25 seconds, and a scrape that arrives while another one is refreshing the cache waits up to 20 seconds for it, which is why the sample sets `scrape_timeout: 30s`. | |
| - **Timeouts.** A scrape fails as a whole if any part of the backing query fails. The query times out after 25 seconds, and a scrape that arrives while another one is refreshing the cache waits for that refresh. For accounts with many checks, keep `scrape_timeout` at `30s` or higher. |
|  | ||
|  | ||
| 2. Under **Bearer tokens**, click **Add token** and copy the value. The full value is shown only once, right after you create it. An account can have at most two tokens, so you can rotate: add a token, point your scrape config at it, then revoke the old one. | ||
|  |
Member
There was a problem hiding this comment.
Important: This reused capture shows the old card with a single Bearer token field and no Bearer tokens list or Add token button, so it contradicts step 2 right above it. The new prometheus_integration_card.png in step 1 already shows the token list and Add token. Remove this image, or replace it with a capture of the token row right after Add token.
| --- | ||
|
|
||
| Before you start, read the prerequisites on the [OTLP export page](/integrations/observability/traces/otlp) and check [what is sent](/integrations/observability/traces/otlp#what-is-sent). | ||
| This page only covers the values Chronosphere needs. |
Member
There was a problem hiding this comment.
Wording (optional): The "Before you start…" lead-in, the "Find the spans… filter on checkly.check.status with the value FAILED" step and the closing "The OTLP export page has the span schema" line repeat word for word on all eight vendor pages. Under the repo's three-or-more-pages rule, that boilerplate belongs in a snippet under snippets/, so a later change to the schema link or the status values happens in one place. This can be a follow-up.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Contributor
Author
|
updated per your comments @hlenke. but things still seem to be problematic on the gates above. not clear to me why those three were cancelled however, perhaps a timeout thing? but i do not have permissions to try and trigger or dig deeper on the setup here from what I can tell cc @stefanjudis |
Contributor
Author
|
so strange. no change made on the commit but simply re-triggering the PR/GH checks above via the cli led to everything passing this time. beats me what happened but always down for the simplicity of did you turn it off and on solutions. re-requesting review @hlenke |
hlenke
reviewed
Nov 4, 2026
hlenke
left a comment
Member
There was a problem hiding this comment.
Ready to merge · The docs.json fix is in, CI is green, and the other four items from the last review are all addressed. The two wording notes below are optional.
This is a re-review of the one commit since my review at bd38402c (41ef019, "minor tweaks to copy and docs.json cleanup").
From the last review
- Blocker,
docs.json: fixed. Theprometheus-v2redirect is closed and the/platform/rocky-aiobject is restored. All checks pass:static-docs-checks,preview-checks,docs-gate,Mintlify Deploymentand link-rot. - Important, timeouts: fixed. The line no longer gives a wait number, and it still explains why
scrape_timeoutshould stay at30s. - Important, outdated screenshot: fixed.
prometheus_v2_step2.pngis removed from step 2. - Important, agent metrics link: fixed.
agent-metrics.mdxnow links/integrations/observability/prometheus, and the Prometheus page links back to Agent metrics. - Wording, repeated vendor boilerplate: done. The lead-in and the closing lines now come from
snippets/otlp-vendor-intro.mdxandsnippets/otlp-vendor-outro.mdxon all eight vendor pages.
Merge readiness
- Mergeable,
mergeStateStatusisCLEAN. - Staleness: since the merge base,
mainhas changed onlyadmin/team-management/single-sign-on.mdx(#786), which this PR does not touch. - Overlap: unchanged. #446 and #449 still edit the deleted
prometheus-v2.mdx. Land this PR first. - Nit, PR description: it still says #755 is open and lists its link as a follow-up. That page has landed and this PR now fixes the link, so the bullet can go.
Inline comments
0 blockers, 0 important, 2 wording (optional).
What I checked
- Incremental diff
bd38402c..41ef019d(13 files), read in full. Merge base withorigin/mainis still54c23c8d, andorigin/mainis at9bd2ca0f. gh pr checks 776: every check passes, and vale-spellcheck is skipped.gh pr viewshows the PR is mergeable andCLEAN.git log 54c23c8d..origin/main --stat: one commit, which touches onlyadmin/team-management/single-sign-on.mdx.pr_impact.pyon the full PR, now thatdocs.jsonparses:- All six removed pages have redirects, and no links in the head still point at them.
- It reports both new snippets as "rendered on 0 pages". That is a false positive: all eight vendor pages import them by
/snippets/...path, the same pattern other pages onmainuse.
- Snippet props: I fetched the Mintlify preview of
traces/new-relic.{vendor}renders as "…the values New Relic needs.", and the outro renders as its own paragraph after the vendor docs line. - I read
prometheus.mdxandagent-metrics.mdxat the head in full. The sample'sscrape_interval: 60sstays abovescrape_timeout: 30s, so the "30s or higher" advice is consistent with it. - The verification of the unchanged content from the last review still applies (monorepo
28c905592a, vendor docs). Its "could not verify" items are unchanged. - No end-to-end tests. These are copy and config changes with no runnable claims.
| | `state` | `queued`: scheduled but not yet picked up by an agent. `inflight`: being executed by an agent. | | ||
| | `stat` | `avg`, `p90`, `p99` over the 2-minute window. | | ||
|
|
||
| For metrics scraped from the agent container itself, including an opt-in per-check set with the `checkly_agent_` prefix, see [Agent metrics](/platform/private-locations/agent-metrics). |
Member
There was a problem hiding this comment.
Wording (optional): The always-on agent metrics carry the checkly_agent_ prefix too, so as written the sentence reads as if only the per-check set does.
Suggested change
| For metrics scraped from the agent container itself, including an opt-in per-check set with the `checkly_agent_` prefix, see [Agent metrics](/platform/private-locations/agent-metrics). | |
| To scrape metrics from the agent container itself, including opt-in per-check metrics, see [Agent metrics](/platform/private-locations/agent-metrics). |
| New Relic limits each request to 1 MB. | ||
|
|
||
| New Relic documents the endpoints in its [OTLP best practices](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/). | ||
| <OtlpVendorOutro /> |
Member
There was a problem hiding this comment.
Wording (optional), all eight vendor pages: Moving "To see failed runs, filter on checkly.check.status…" into the outro takes it out of step 4 ("Find the spans"). In the preview it now renders after the "New Relic documents the endpoints…" line, so the tip reads as an afterthought. One fix: put the filter sentence back at the end of step 4 and keep only the span-schema line in the outro snippet. This can be a follow-up.
hlenke
added a commit
that referenced
this pull request
Nov 4, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checkly exposes metrics on one Prometheus endpoint and sends check runs as OpenTelemetry spans to one OTLP endpoint. The pages for that are split between Integrations > Observability and Resolve > Traces > Exporting today, and Chronosphere is not in the nav. This PR puts them in one place.
What changes
/integrations/observability/prometheus-v2redirects to/integrations/observability/prometheusand the#heartbeat-metricsanchor survives.how-it-workskeeps its diagram and gets the four-type coverage wording. A vendor-neutral Mermaid version of the diagram sits on the page inside an MDX comment, ready to swap in.Moves and redirects
resolve/traces/export/grafana-tempointegrations/observability/traces/grafana-temporesolve/traces/export/honeycombintegrations/observability/traces/honeycombresolve/traces/export/new-relicintegrations/observability/traces/new-relicresolve/traces/export/dash0integrations/observability/traces/dash0resolve/traces/export/to-chronosphereintegrations/observability/traces/chronosphereintegrations/observability/prometheus-v2integrations/observability/prometheusAll six have
docs.jsonredirects, so existing links, including the ones in the web app, keep working.Verification
prometheusreceiver block and the legacy v1 config were run in Prometheus 3.0.0 and otelcol-contrib 0.160.0. Both Prometheus endpoints were scraped and compared with the documented metric families (35 of 35 v2, 6 of 6 v1).dash0-token.pngis masked.Deliberately dropped
dash0-checkly.pngandexport-traces-to-grafana-config.png(the old Export Traces form, replaced by the fresh capture) andprometheus_step2.png(the v1 activation card; v1 setup is now one sentence under Legacy v1 endpoint). The files stay on disk.enable_compressiontip (the endpoint gzips when the scraper asks, which Prometheus does by default)./integrations/observability/prometheus.agenticcheck type from thecheck_typeenumeration.Follow-up
checkly_otel_diagram.pngonhow-it-works: it hardcodestracestate: checkly=trueand lists Baselime among the vendors. The Mermaid version on the page is the starting point.Open PRs that touch the same files
prometheus-v2.mdx, which this PR moves or deletes, plusprometheus.mdx,coralogix.mdx,how-it-works.mdx,resolve/traces/overview.mdxandresolve/overview.mdx. The new pages here already use sentence case. Whichever lands second keeps this PR's moves and deletions and re-applies docs: sentence case for headings and titles #774's casing to the headings this PR did not write.agent-metrics.mdxlinks/integrations/observability/prometheus-v2/, which the new redirect covers; it should point at/integrations/observability/prometheus.