Skip to content

[Feature]: evidence.md convention for speckit.bug.assess (pre-digested inputs such as reduced logs) #4804

Open
Open
[Feature]: evidence.md convention for speckit.bug.assess (pre-digested inputs such as reduced logs)#4804
Labels
feature-assessRun the Spec Kit idea-assessment pipeline on this feature requestfeature-goFeature assessment verdict: go — ready to hand off to /speckit.specifytriage-can-waitVerdict: valid and in-scope but deprioritized; held behind the evidence gate

Description

@alexcpn
alexcpn
opened on Oct 1, 2026

Problem Statement

speckit.bug.assess ingests a bug report as pasted text or a URL. Real bug
reports often come with a log: a CI run, kubectl logs, a crash-looping
service. These run to thousands or millions of lines. Today the agent either
reads the raw log (blowing the context window and burying the one relevant
error under repeats) or skims it ad hoc. Whatever it looked at is not recorded
in .specify/bugs/<slug>/, so speckit.bug.fix and speckit.bug.test can't
rely on it, and a reviewer can't check what the agent actually saw.

Proposed Solution

In Execution → 1. Ingest the bug report of speckit.bug.assess, add:

If BUG_DIR/evidence.md exists (for example, written by a
before_bug_assess hook), read it as part of the report. Cite it under
Report and prefer it over re-reading any raw log it was derived from.

Also add evidence.md (optional) to the per-bug directory layout in the
README, alongside assessment.md, fix.md and test.md.

This names no tool. With no evidence.md, behavior is exactly as today. The
file can be written by hand or by any extension. Using the
before_bug_assess hook from #4799 is the natural route, but this convention
works without that hook.

Motivating producer: log intake

A community extension, speckit.logreduce.intake,
finds log files referenced in the report and runs
logreduce with a token budget.
logreduce is a single static binary that applies TF-IDF over masked templates
plus severity weighting. The extension writes BUG_DIR/evidence.md containing
the command it ran, the source path, the summary header and the reduced log.

  • On a 1M-line log, that is a ~99.9% token reduction.
  • On the LogDx CI-incident benchmark (35 cases), it kept 99% of the
    human-labelled critical lines at an 8k-token budget.

Assess, fix and test then all work from the same saved evidence file instead of
the raw log. I'll maintain that extension and submit it to the community
catalog. This issue only asks for the convention.

Alternatives Considered

  • Bake log reduction into speckit.bug.assess. That adds a tool-specific
    dependency to a bundled extension. A file convention keeps core neutral.
  • A standalone command the user runs first, with no convention. This works
    today, but assess doesn't know to read the result.

Component

Extensions: bundled bug extension (extensions/bug/)

AI Agent (if applicable)

All.

Use Cases

  1. A CI job fails with a 40k-line log. Intake writes about 7k tokens of reduced
    log to evidence.md, and the assessment cites the first error and the
    crash-loop pattern from it.
  2. speckit.bug.test fails and recommends re-running assess. The new failing
    log goes through the same intake, so evidence.md shows the before and
    after.

Acceptance Criteria

  • speckit.bug.assess reads BUG_DIR/evidence.md when present and cites it
    under Report.
  • evidence.md (optional) is listed in the per-bug directory layout in the
    bug extension README.
  • With no evidence.md, the command output is unchanged.

I'm happy to send the PR.

Additional Context

AI Disclosure

Drafted with Claude Code (Claude Opus 5.5) from my notes. I reviewed it.

Activity

  1. mnriem
    added
    triage-can-waitVerdict: valid and in-scope but deprioritized; held behind the evidence gate
    feature-assessRun the Spec Kit idea-assessment pipeline on this feature request
    on Oct 1, 2026

  2. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Contributor

    Feature assessment — evidence-convention · Stage 1/5: Intake

    Idea Intake: evidence.md convention for bug assessment

    Idea (as captured)

    speckit.bug.assess currently ingests pasted bug reports or URLs, but real reports can include CI, Kubernetes, or service logs that are thousands or millions of lines long. The proposal is to have speckit.bug.assess read .specify/bugs/<slug>/evidence.md when it exists, cite it under Report, and prefer it over re-reading the raw log it summarizes. The file would be optional, written by hand or by an extension, and the bug extension README would list it alongside assessment.md, fix.md, and test.md. With no evidence.md, behavior should remain unchanged.

    The issue describes a motivating community log-reduction extension that writes evidence.md, two use cases involving large CI logs and repeat assessments, alternatives of embedding reduction in the bundled extension or using a standalone command without a convention, acceptance criteria, and a possible future reader in a community bugfix extension.

    Restated

    Adopt an optional per-bug evidence.md convention so speckit.bug.assess can consume and cite pre-digested evidence, especially reduced logs, while retaining current behavior when the file is absent.

    Origin & Context

    First-Glance Unknowns

    • [NEEDS CLARIFICATION: What exact content and size limits, if any, should evidence.md have?]
    • [NEEDS CLARIFICATION: Should malformed, unreadable, or oversized evidence produce an explicit error, a warning, or fallback to the raw report?]
    • [NEEDS CLARIFICATION: How should the command distinguish evidence from untrusted instructions contained in a generated file?]
    • [NEEDS CLARIFICATION: What does “prefer it over re-reading” mean when both a raw log and evidence file are supplied?]
    • [NEEDS CLARIFICATION: Should the convention be documented or consumed by other bundled/community bug commands beyond speckit.bug.assess?]

    Generated by 💡 Assess a Feature Request by Installing and Running Spec Kit for #4804 · copilot · gpt52codex · 4.53 AIC · ⌖ 6.1 AIC · ⊞ 24K · ◷

  3. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Contributor

    Feature assessment — evidence-convention · Stage 2/5: Research

    Idea Research: evidence.md convention for bug assessment

    • Slug: evidence-convention
    • Created: 2026-10-01T17:41:18Z
    • Evidence confidence (overall): medium

    Users & Demand

    • The request is directly stated by the issue author, who describes CI, Kubernetes, and service logs ranging from tens of thousands to millions of lines and asks for a persisted reduced representation — source: GitHub issue #4804 (confidence: medium, cited).
    • The issue provides two concrete use cases: initial assessment of a 40k-line CI log and repeat assessment after a failed test, with both stages sharing saved evidence — source: GitHub issue #4804 (confidence: medium, cited).
    • No usage telemetry, support-ticket count, maintainer survey, or independent user reports are present in the issue or repository evidence reviewed — [NEEDS CLARIFICATION: quantify how often large-log intake occurs] (confidence: low, assumption).

    Prior Art

    Market & Context

    • The proposed convention keeps log reduction outside the bundled extension and allows a community extension or a human to produce the file, avoiding a bundled dependency on a specific reducer — source: GitHub issue #4804 (confidence: medium, cited).
    • The current alternative is for an agent to read raw logs or use an ad hoc summary; the issue claims this can exhaust context and makes the inspected evidence unavailable to downstream bug commands and reviewers — source: GitHub issue #4804 (confidence: medium, cited).
    • No repository evidence compares this convention with an existing artifact contract elsewhere in Spec Kit, and no external market data was gathered — [NEEDS CLARIFICATION: determine whether a shared artifact or provenance convention already exists] (confidence: low, assumption).

    Data & Constraints

    • The requested behavior is explicitly optional and backward-compatible when evidence.md is absent — source: GitHub issue #4804, acceptance criteria (confidence: high, cited).
    • The issue reports a community reducer's claimed result of approximately 99.9% token reduction on a 1M-line log and 99% retention of labelled critical lines on a 35-case benchmark; these claims are not independently verified in this repository — [UNVERIFIED: issue author’s benchmark and reduction figures] (confidence: low, cited claim).
    • speckit.bug.assess treats report content as untrusted input under its command policy, so any convention must preserve that handling for generated evidence — source: extensions/bug/commands/speckit.bug.assess.md:39-75 (confidence: high, cited).
    • The repository does not specify a maximum evidence size, encoding, provenance format, error behavior, or precedence semantics for evidence versus raw reports — [NEEDS CLARIFICATION: define these constraints during specification] (confidence: high, cited absence).

    Evidence Against the Idea

    • An optional file convention may add little value if large-log producers are rare or if downstream commands do not consume the artifact consistently — [NEEDS CLARIFICATION: validate adoption across bug workflows] (confidence: low, assumption).
    • Reading arbitrary generated Markdown can preserve misleading or stale summaries and may create ambiguity about provenance, freshness, and fallback behavior — [NEEDS CLARIFICATION: establish provenance and stale-evidence handling] (confidence: medium, assumption).
    • The claimed reduction and benchmark benefits come from one issue author and are not independently verified here — source: GitHub issue #4804 (confidence: high, cited).

    Gaps & Open Questions

    • [NEEDS CLARIFICATION: How frequent and costly are large-log bug assessments in actual Spec Kit usage?]
    • [NEEDS CLARIFICATION: What evidence metadata is required to establish source, generation time, and relationship to the raw report?]
    • [NEEDS CLARIFICATION: What are the size, encoding, malformed-file, and unreadable-file policies?]
    • [NEEDS CLARIFICATION: Which commands are in scope for consuming or citing evidence.md?]

    Sources

    Generated by 💡 Assess a Feature Request by Installing and Running Spec Kit for #4804 · copilot · gpt52codex · 4.53 AIC · ⌖ 6.1 AIC · ⊞ 24K · ◷

  4. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Contributor

    Feature assessment — evidence-convention · Stage 3/5: Problem

    Problem Definition: evidence.md convention for bug assessment

    • Slug: evidence-convention
    • Created: 2026-10-01T17:41:18Z
    • Inputs used: intake.md and research.md

    Problem Statement

    People assessing bugs with very large logs cannot reliably focus on relevant evidence or preserve what was inspected for later fix, test, and review stages. The current bundled bug workflow has no documented per-bug place for a pre-digested report, so repeated assessments may reprocess raw logs inconsistently and consume excessive context.

    Affected Users & Stakeholders

    Goals

    • Make the evidence used for large-log assessment persistently identifiable and reusable across assessment, fix, test, and review stages.
    • Reduce repeated handling of raw logs sufficiently to keep bug assessment practical for large inputs.
    • Preserve a tool-neutral workflow so evidence can be produced by a person or any extension.
    • Avoid changing behavior for bug reports that have no supplementary evidence.

    Non-Goals

    • Building or bundling a specific log-reduction tool.
    • Defining the implementation architecture, parser, command API, or storage model.
    • Guaranteeing that a reduced log retains every critical line; the issue's benchmark claims are unverified.
    • Expanding the convention to unrelated workflows without evidence that they need it.

    Success Metrics

    • When supplementary evidence is present, bug assessment can cite and reuse it without requiring the full raw log to be reprocessed (baseline: unsupported today; exact observable behavior [NEEDS CLARIFICATION]).
    • When supplementary evidence is absent, assessment output and workflow behavior remain unchanged (baseline: current documented behavior).
    • A reviewer can identify the evidence used for an assessment from the per-bug artifacts (baseline: no documented evidence artifact).
    • Qualitative: community or maintainer feedback confirms the convention is sufficiently tool-neutral and understandable (baseline: unknown).

    Cost of Inaction

    Users continue to choose between sending very large raw logs through assessment or making ad hoc summaries that are not part of the per-bug record. This can waste context, make repeated assessments inconsistent, and leave reviewers unable to determine what evidence informed the result. The frequency and magnitude of this cost are not measured in the repository.

    Open Questions

    • [NEEDS CLARIFICATION: What exact evidence size, encoding, provenance, and freshness rules are required?]
    • [NEEDS CLARIFICATION: What should happen when evidence is malformed, unreadable, stale, or conflicts with the raw report?]
    • [NEEDS CLARIFICATION: Which downstream commands must consume or cite the evidence artifact?]
    • [NEEDS CLARIFICATION: What measurable baseline and target should represent reduced raw-log handling?]

    Generated by 💡 Assess a Feature Request by Installing and Running Spec Kit for #4804 · copilot · gpt52codex · 4.53 AIC · ⌖ 6.1 AIC · ⊞ 24K · ◷

  5. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Contributor

    Feature assessment — evidence-convention · Stage 4/5: Concept

    Concept: evidence.md convention for bug assessment

    • Slug: evidence-convention
    • Created: 2026-10-01T17:41:18Z
    • Recommended option: Option A — Optional evidence handoff

    Options

    Option A — Optional evidence handoff

    +- Sketch: Treat an optional per-bug evidence document as the canonical supplementary input for bug assessment when present, and make the bundled workflow visibly reference it in the assessment record. Raw-report-only bugs continue through the existing path.
    +- Appetite: small
    +- Trade-offs: Wins a simple, tool-neutral contract and backward compatibility. Sacrifices richer guarantees about provenance, validation, and downstream consumption until those needs are better understood.
    +- Rabbit holes: Ambiguous precedence between raw and reduced material; unbounded or stale files; inconsistent adoption by community producers; unclear behavior for unreadable content.

    Option B — Evidence convention with explicit provenance expectations

    +- Sketch: Establish the optional evidence document as a reviewable handoff that includes enough context for users to understand where it came from and when it was produced, while keeping generation outside the bundled extension.
    +- Appetite: medium
    +- Trade-offs: Wins auditability and makes stale or misleading summaries easier to identify. Sacrifices the smallest possible change and may impose requirements that simple hand-authored evidence cannot meet.
    +- Rabbit holes: Designing a metadata standard prematurely; debating freshness and source identity; expanding documentation and tests across every producer.

    Option C — Do nothing in the bundled workflow

    +- Sketch: Leave the bug extension unchanged and let users or community extensions pass reduced content as ordinary pasted report text or manage their own files.
    +- Appetite: small
    +- Trade-offs: Wins zero implementation and documentation cost. Sacrifices persistent shared context, reviewability, and the issue's stated benefit for repeated assessments.
    +- Rabbit holes: Fragmented conventions, duplicated raw-log processing, and downstream tools that cannot discover the same evidence.

    Recommendation

    Recommend Option A because it addresses the stated problem with the smallest tool-neutral scope and preserves the current no-evidence path. It directly supports the issue's requested acceptance criteria while leaving provenance, size, and broader consumer requirements as explicit specification questions rather than inventing a standard without usage data.

    Out of Scope (for the recommended option)

    • Bundling or selecting a log reducer.
    • Defining a detailed evidence schema, parser, or transport.
    • Requiring every community producer to use a particular tool.
    • Automatically changing the behavior of speckit.bug.fix, speckit.bug.test, or unrelated extensions until their evidence needs are validated.
    • Proving the issue's benchmark and token-reduction claims.

    Assumptions to Validate

    • The per-bug directory is a discoverable and acceptable shared boundary for a producer and the bug-assessment workflow.
    • The minimum useful behavior can be defined without a mandatory metadata schema.
    • Missing evidence must remain a no-op for compatibility.
    • Explicit citation and safe handling of evidence are sufficient for initial reviewer trust.
    • Maintainers can agree on fallback behavior for unreadable or conflicting evidence during specification.

    Generated by 💡 Assess a Feature Request by Installing and Running Spec Kit for #4804 · copilot · gpt52codex · 4.53 AIC · ⌖ 6.1 AIC · ⊞ 24K · ◷

  6. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsBot
    on Oct 1, 2026 – with GitHub Actions
    Contributor

    Feature assessment — evidence-convention · Stage 5/5: Decision — verdict go

    Decision: evidence.md convention for bug assessment

    • Slug: evidence-convention
    • Decided: 2026-10-01T17:41:18Z
    • Verdict: go
    • Artifacts reviewed: intake.md, research.md, problem.md, concept.md

    Scorecard

    Criterion Rating Justification
    Problem validity adequate The issue gives a concrete large-log workflow problem, two use cases, and repository evidence that the bundled bug flow has no evidence artifact convention.
    Evidence strength adequate The request is specific and grounded in the issue plus current command, README, and workflow tests, although usage frequency and benchmark claims remain unverified.
    Value vs. inaction adequate A small tool-neutral convention could preserve shared context and avoid repeated raw-log handling; the scale of the benefit is not measured.
    Feasibility / appetite strong The recommended option is deliberately small, optional, and compatible with the current no-evidence path.
    Strategic fit adequate It fits the existing per-bug artifact workflow and keeps log-reduction tooling outside the bundled extension; broader extension alignment is not established.
    Risk posture weak Provenance, size, malformed-file, precedence, and untrusted-content handling remain open and must be resolved during specification.

    Verdict & Rationale

    Go. The problem is sufficiently concrete, the requested change has a bounded small option, and the repository already uses per-bug Markdown artifacts as a workflow boundary. The issue's adoption and performance claims are not independently verified, and several contract details remain open, but they are specification questions for a narrow backward-compatible convention rather than blockers to deciding whether the idea is worth specifying. The handoff must explicitly address evidence trust, failure behavior, precedence, and scope of downstream consumers.

    If needs-clarification

    • Blocking questions: Not applicable; open questions are carried into the specification handoff.
    • Revisit stage: Not applicable.

    If go — Handoff to __SPECKIT_COMMAND_SPECIFY__

    • Problem: Large-log bug assessments lack a persistent, reviewable, tool-neutral place for pre-digested evidence, causing repeated raw-log handling and inconsistent context across assessment stages.
    • Chosen approach: Option A — Optional evidence handoff: recognize and cite a per-bug evidence.md when present while leaving raw-report-only behavior unchanged.
    • In scope / out of scope: Define the optional convention, its citation/precedence behavior, safe handling, failure behavior, documentation, and regression coverage. Out of scope are log-reduction tooling, a bundled reducer, a detailed producer-specific schema, and automatic changes to unrelated commands.
    • Success metrics: Evidence is identifiable and reusable when present; the no-evidence path is unchanged; reviewers can determine what evidence informed the assessment. Establish measurable raw-log/context baselines during specification if possible.
    • Carried-forward open questions:
      • [NEEDS CLARIFICATION: What size, encoding, provenance, and freshness rules apply?]
      • [NEEDS CLARIFICATION: What happens for malformed, unreadable, stale, or conflicting evidence?]
      • [NEEDS CLARIFICATION: How does the workflow treat evidence as untrusted content?]
      • [NEEDS CLARIFICATION: Which downstream commands must consume or cite the file?]
      • [NEEDS CLARIFICATION: What measurable baseline and target represent reduced raw-log handling?]

    Generated by 💡 Assess a Feature Request by Installing and Running Spec Kit for #4804 · copilot · gpt52codex · 4.53 AIC · ⌖ 6.1 AIC · ⊞ 24K · ◷

  7. mnriem
    added
    feature-goFeature assessment verdict: go — ready to hand off to /speckit.specify
    on Oct 2, 2026

  8. mnriem commented on Oct 2, 2026

    @mnriem
    mnriem
    on Oct 2, 2026
    Collaborator

    @alexcpn I would recommend going for a community preset or presets as they can be stacked. And then one can say deliver it in a way that each user of the assess extension can opt into what they need for their specific environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature-assessRun the Spec Kit idea-assessment pipeline on this feature requestfeature-goFeature assessment verdict: go — ready to hand off to /speckit.specifytriage-can-waitVerdict: valid and in-scope but deprioritized; held behind the evidence gate

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions