Ask rtibblesbot to validate a contributor-opened issue #114

Overview

A contributor opens an issue and nothing checks the claim in it. A maintainer must read the
report and check it against the code before the issue can be labelled or worked.

When a contributor opens an issue, ask @rtibblesbot to validate the report, and label the
issue while the validation runs.

Complexity: Medium
Target branch: main

Context

Contributors opened 27 issues across kolibri, studio and kolibri-design-system in the last
six months, about five a month. Nine of the twelve still open on kolibri carry no label, so
the claim in each was never confirmed or rejected in the open.

The registry carries no automation for a contributor-opened issue. issue-open only swaps the
contributing header in the body.

learningequality/coding-agent#1229 adds the validation on the agent side. It must land first.
Until it does, the agent drops a mention on an issue it did not open, so this automation posts
comments that nothing answers.

The design rests on one assumption to check before anything is built: a mention posted with
the LE bot app's token reaches rtibblesbot as a notification. A mention is the only way to
reach the agent, because it polls notifications and holds no inbound endpoint. A label creates
no notification, so the label alone cannot start the work.

On a claim that holds, the agent also rewrites the issue title and body to correct what the
report states wrongly. manage-issue-header.js owns the block between <!---HEADER START-->
and <!---HEADER END-->, so the agent preserves that block and this side changes nothing. The
constraint is stated from the agent's side in learningequality/coding-agent#1229. A body edit
fires no event this repository listens to, because issue-label-header runs on labeled and
unlabeled only.

The change

A new automation. Add issue-validation to automation-registry.yml, on
issues: [opened], with the leaf issue-validation.yml and the script
scripts/issue-validation.js.

It runs when all of these hold:

• The repository is in COMMUNITY_REPOS.
• The opener is a contributor by isContributor, so a core team member and a bot are both
  excluded.
• The issue does not already carry community-issue-review.

It then posts one comment that mentions @rtibblesbot, and adds community-issue-review. The
comment tells the contributor that the report is under review and what happens next. The add
call creates the label in a repository that does not define it, as the
community-contribution-in-progress path already does.

A second automation to clear the label. Add issue-validation-done, on
issue_comment: [created]. It removes community-issue-review when the comment body carries
<!-- ISSUE_VALIDATION -->. The marker is the contract between the two repositories. A check
on the comment author alone is not enough, because rtibblesbot comments on issues for other
reasons. This cannot live in contributor-issue-comment, because rtibblesbot is in
BOT_USERNAMES and that automation drops a bot comment by design.

A sweep for the stuck case. If the agent never answers, because the service is stopped or
every token is capped, the label stays on and misleads a maintainer. Add a sweep to
unassign-inactive-issues that removes community-issue-review when its labeled event is
older than ISSUE_REVIEW_MAX_DAYS days. Three days is generous, because a healthy agent
answers in minutes.

Constants. Add the label name, the marker and the age limit to scripts/constants.js, and
add the comment body beside the other bot messages.

Out of Scope

• The validation itself, which lives in learningequality/coding-agent#1229.
• Any label beyond community-issue-review.
• Closing the issue. No automation here triggers a closure, and the agent decides none
  either. The verdict comment names the close it recommends. A maintainer asks for it in a
  reply, and the agent's own resolution path applies it, which accepts only a maintainer's
  ask. So a wrong verdict costs a reply rather than a closed report.
• Issues opened by core team members.
• Acting on the verdict, for example assigning the issue or adding help wanted.
• Pull requests. contributor-pr-reply already covers them.

Acceptance Criteria

• An issue opened by a contributor in a community repo gets one comment mentioning
  @rtibblesbot, and the community-issue-review label.
• An issue opened by a core team member gets neither.
• An issue opened by a bot gets neither.
• An issue opened in a repository outside COMMUNITY_REPOS gets neither.
• An issue that already carries the label gets no second comment.
• A comment carrying <!-- ISSUE_VALIDATION --> removes the label.
• A comment without the marker leaves the label on.
• The weekly sweep removes the label when it is older than ISSUE_REVIEW_MAX_DAYS days,
  and leaves a younger one on.
• A failure to add or remove the label does not fail the run, and it is logged.
• node scripts/generate-automation.js --check passes, so the generated files match the
  registry.
• Tests cover each guard: contributor, core member, bot, repository, label already
  present, marker present, marker absent.
• docs/community-automations.md describes both automations and the marker contract.

Testing

Test in test-actions. See
testing the automations.
The cases need an account outside the organization.

1. Post a comment mentioning @rtibblesbot with the LE bot app's token. Expect a notification
   on the bot account. If none arrives, the whole design needs a different trigger, so run
   this step first.
2. Open an issue from an account outside the organization. Expect one comment mentioning
   @rtibblesbot, and the community-issue-review label.
3. Open an issue as a core team member. Expect no comment and no label.
4. Post a comment carrying <!-- ISSUE_VALIDATION -->. Expect the label to come off.
5. Post a comment without the marker. Expect the label to stay on.
6. Reopen the issue from step 2 while the label is on. Expect no second comment.
7. Backdate a label, or lower ISSUE_REVIEW_MAX_DAYS, then run the sweep by
   workflow_dispatch. Expect the label to come off.

References

• learningequality/coding-agent#1229: adds the validation, posts the verdict and owns the
  marker. It must land first.
• [Proposal] Set up GitHub Discussions as the entry point for unplanned work #105 proposes GitHub Discussions as the entry point for unplanned work. That reduces how
  many reports reach this path. It does not replace the check on the ones that do.
• Update the contributing page to indicate that a bug report comes before the pull request #111 updates the contributing page to say that a bug report comes before a pull request.
• scripts/community-contribution-label.py: the existing add and remove pattern, and the
  proof that an add creates a label the repository does not define.

AI usage

I used Claude Code to measure how many issues contributors open and how many were never
labelled, and to trace why a mention is the only way to reach the agent. The split between
the two repositories and the label are my decision.