mirror of
https://github.com/pchuan98/codex.git
synced 2026-07-01 00:31:56 +08:00
Fix PR babysitter review comment monitoring (#16363)
## Summary - prioritize newly surfaced review comments ahead of CI and mergeability handling in the PR babysitter watcher - keep `--watch` running for open PRs even when they are currently merge-ready so later review feedback is not missed
This commit is contained in:
Eric Traut
committed by
GitHub
Unverified
GitHub
parent
e8de4ea953
commit
0fe873ad5f
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: babysit-pr
|
||||
description: Babysit a GitHub pull request after creation by continuously polling CI checks/workflow runs, new review comments, and mergeability state until the PR is ready to merge (or merged/closed). Diagnose failures, retry likely flaky failures up to 3 times, auto-fix/push branch-related issues when appropriate, and stop only when user help is required (for example CI infrastructure issues, exhausted flaky retries, or ambiguous/blocking situations). Use when the user asks Codex to monitor a PR, watch CI, handle review comments, or keep an eye on failures and feedback on an open PR.
|
||||
description: Babysit a GitHub pull request after creation by continuously polling review comments, CI checks/workflow runs, and mergeability state until the PR is merged/closed or user help is required. Diagnose failures, retry likely flaky failures up to 3 times, auto-fix/push branch-related issues when appropriate, and keep watching open PRs so fresh review feedback is surfaced promptly. Use when the user asks Codex to monitor a PR, watch CI, handle review comments, or keep an eye on failures and feedback on an open PR.
|
||||
---
|
||||
|
||||
# PR Babysitter
|
||||
@@ -9,8 +9,8 @@ description: Babysit a GitHub pull request after creation by continuously pollin
|
||||
Babysit a PR persistently until one of these terminal outcomes occurs:
|
||||
|
||||
- The PR is merged or closed.
|
||||
- CI is successful, there are no unaddressed review comments surfaced by the watcher, required review approval is not blocking merge, and there are no potential merge conflicts (PR is mergeable / not reporting conflict risk).
|
||||
- A situation requires user help (for example CI infrastructure issues, repeated flaky failures after retry budget is exhausted, permission problems, or ambiguity that cannot be resolved safely).
|
||||
- Optional handoff milestone: the PR is currently green + mergeable + review-clean. Treat this as a progress state, not a watcher stop, so late-arriving review comments are still surfaced promptly while the PR remains open.
|
||||
|
||||
Do not stop merely because a single snapshot returns `idle` while checks are still pending.
|
||||
|
||||
@@ -24,7 +24,7 @@ Accept any of the following:
|
||||
## Core Workflow
|
||||
|
||||
1. When the user asks to "monitor"/"watch"/"babysit" a PR, start with the watcher's continuous mode (`--watch`) unless you are intentionally doing a one-shot diagnostic snapshot.
|
||||
2. Run the watcher script to snapshot PR/CI/review state (or consume each streamed snapshot from `--watch`).
|
||||
2. Run the watcher script to snapshot PR/review/CI state (or consume each streamed snapshot from `--watch`).
|
||||
3. Inspect the `actions` list in the JSON response.
|
||||
4. If `diagnose_ci_failure` is present, inspect failed run logs and classify the failure.
|
||||
5. If the failure is likely caused by the current branch, patch code locally, commit, and push.
|
||||
@@ -33,10 +33,10 @@ Accept any of the following:
|
||||
8. If a review item from another author is non-actionable, already addressed, or not valid, post one reply on the comment/thread explaining that decision (for example answering the question or explaining why no change is needed). If the watcher later surfaces your own reply, treat that self-authored item as already handled and do not reply again.
|
||||
9. If the failure is likely flaky/unrelated and `retry_failed_checks` is present, rerun failed jobs with `--retry-failed-now`.
|
||||
10. If both actionable review feedback and `retry_failed_checks` are present, prioritize review feedback first; a new commit will retrigger CI, so avoid rerunning flaky checks on the old SHA unless you intentionally defer the review change.
|
||||
11. On every loop, verify mergeability / merge-conflict status (for example via `gh pr view`) in addition to CI and review state.
|
||||
11. On every loop, look for newly surfaced review feedback before acting on CI failures or mergeability state, then verify mergeability / merge-conflict status (for example via `gh pr view`) alongside CI.
|
||||
12. After any push or rerun action, immediately return to step 1 and continue polling on the updated SHA/state.
|
||||
13. If you had been using `--watch` before pausing to patch/commit/push, relaunch `--watch` yourself in the same turn immediately after the push (do not wait for the user to re-invoke the skill).
|
||||
14. Repeat polling until the PR is green + review-clean + mergeable, `stop_pr_closed` appears, or a user-help-required blocker is reached.
|
||||
14. Repeat polling until `stop_pr_closed` appears or a user-help-required blocker is reached. A green + review-clean + mergeable PR is a progress milestone, not a reason to stop the watcher while the PR is still open.
|
||||
15. Maintain terminal/session ownership: while babysitting is active, keep consuming watcher output in the same turn; do not leave a detached `--watch` process running and then end the turn as if monitoring were complete.
|
||||
|
||||
## Commands
|
||||
@@ -131,7 +131,7 @@ Use this loop in a live Codex session:
|
||||
8. Retry failed checks only when `retry_failed_checks` is present and you are not about to replace the current SHA with a review/CI fix commit.
|
||||
9. If you pushed a commit, resolved a review thread, replied to a review comment, or triggered a rerun, report the action briefly and continue polling (do not stop).
|
||||
10. After a review-fix push, proactively restart continuous monitoring (`--watch`) in the same turn unless a strict stop condition has already been reached.
|
||||
11. If everything is passing, mergeable, not blocked on required review approval, and there are no unaddressed review items, report success and stop.
|
||||
11. If everything is passing, mergeable, not blocked on required review approval, and there are no unaddressed review items, report that the PR is currently ready to merge but keep the watcher running so new review comments are surfaced quickly while the PR remains open.
|
||||
12. If blocked on a user-help-required issue (infra outage, exhausted flaky retries, unclear reviewer request, permissions), report the blocker and stop.
|
||||
13. Otherwise sleep according to the polling cadence below and repeat.
|
||||
|
||||
@@ -141,19 +141,18 @@ Do not hand control back to the user after a review-fix push just because a new
|
||||
If a `--watch` process is still running and no strict stop condition has been reached, the babysitting task is still in progress; keep streaming/consuming watcher output instead of ending the turn.
|
||||
|
||||
## Polling Cadence
|
||||
Use adaptive polling and continue monitoring even after CI turns green:
|
||||
Keep review polling aggressive and continue monitoring even after CI turns green:
|
||||
|
||||
- While CI is not green (pending/running/queued or failing): poll every 1 minute.
|
||||
- After CI turns green: start at every 1 minute, then back off exponentially when there is no change (for example 1m, 2m, 4m, 8m, 16m, 32m), capping at every 1 hour.
|
||||
- Reset the green-state polling interval back to 1 minute whenever anything changes (new commit/SHA, check status changes, new review comments, mergeability changes, review decision changes).
|
||||
- If CI stops being green again (new commit, rerun, or regression): return to 1-minute polling.
|
||||
- After CI turns green: keep polling at the base cadence while the PR remains open so newly posted review comments are surfaced promptly instead of waiting on a long green-state backoff.
|
||||
- Reset the cadence immediately whenever anything changes (new commit/SHA, check status changes, new review comments, mergeability changes, review decision changes).
|
||||
- If CI stops being green again (new commit, rerun, or regression): stay on the base polling cadence.
|
||||
- If any poll shows the PR is merged or otherwise closed: stop polling immediately and report the terminal state.
|
||||
|
||||
## Stop Conditions (Strict)
|
||||
Stop only when one of the following is true:
|
||||
|
||||
- PR merged or closed (stop as soon as a poll/snapshot confirms this).
|
||||
- PR is ready to merge: CI succeeded, no surfaced unaddressed review comments, not blocked on required review approval, and no merge conflict risk.
|
||||
- User intervention is required and Codex cannot safely proceed alone.
|
||||
|
||||
Keep polling when:
|
||||
@@ -162,14 +161,14 @@ Keep polling when:
|
||||
- CI is still running/queued.
|
||||
- Review state is quiet but CI is not terminal.
|
||||
- CI is green but mergeability is unknown/pending.
|
||||
- CI is green and mergeable, but the PR is still open and you are waiting for possible new review comments or merge-conflict changes per the green-state cadence.
|
||||
- The PR is green but blocked on review approval (`REVIEW_REQUIRED` / similar); continue polling on the green-state cadence and surface any new review comments without asking for confirmation to keep watching.
|
||||
- CI is green and mergeable, but the PR is still open and you are waiting for possible new review comments or merge-conflict changes.
|
||||
- The PR is green but blocked on review approval (`REVIEW_REQUIRED` / similar); continue polling at the base cadence and surface any new review comments without asking for confirmation to keep watching.
|
||||
|
||||
## Output Expectations
|
||||
Provide concise progress updates while monitoring and a final summary that includes:
|
||||
|
||||
- During long unchanged monitoring periods, avoid emitting a full update on every poll; summarize only status changes plus occasional heartbeat updates.
|
||||
- Treat push confirmations, intermediate CI snapshots, and review-action updates as progress updates only; do not emit the final summary or end the babysitting session unless a strict stop condition is met.
|
||||
- Treat push confirmations, intermediate CI snapshots, ready-to-merge snapshots, and review-action updates as progress updates only; do not emit the final summary or end the babysitting session unless a strict stop condition is met.
|
||||
- A user request to "monitor" is not satisfied by a couple of sample polls; remain in the loop until a strict stop condition or an explicit user interruption.
|
||||
- A review-fix commit + push is not a completion event; immediately resume live monitoring (`--watch`) in the same turn and continue reporting progress updates.
|
||||
- When CI first transitions to all green for the current SHA, emit a one-time celebratory progress update (do not repeat it on every green poll). Preferred style: `🚀 CI is all green! 33/33 passed. Still on watch for review approval.`
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
interface:
|
||||
display_name: "PR Babysitter"
|
||||
short_description: "Watch PR CI, reviews, and merge conflicts"
|
||||
default_prompt: "Babysit the current PR: monitor CI, reviewer comments, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); fix valid issues, push updates, and rerun flaky failures up to 3 times. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Continue polling autonomously after any push/rerun until a strict terminal stop condition is reached or the user interrupts."
|
||||
short_description: "Watch PR review comments, CI, and merge conflicts"
|
||||
default_prompt: "Babysit the current PR: monitor reviewer comments, CI, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); surface new review feedback before acting on CI or mergeability work, fix valid issues, push updates, and rerun flaky failures up to 3 times. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Do not treat a green + mergeable PR as a terminal stop while it is still open; continue polling autonomously after any push/rerun so newly posted review comments are surfaced until a strict terminal stop condition is reached or the user interrupts."
|
||||
|
||||
@@ -45,7 +45,6 @@ MERGE_CONFLICT_OR_BLOCKING_STATES = {
|
||||
"DRAFT",
|
||||
"UNKNOWN",
|
||||
}
|
||||
GREEN_STATE_MAX_POLL_SECONDS = 60 * 60
|
||||
|
||||
|
||||
class GhCommandError(RuntimeError):
|
||||
@@ -578,7 +577,7 @@ def recommend_actions(pr, checks_summary, failed_runs, new_review_items, retries
|
||||
return unique_actions(actions)
|
||||
|
||||
if is_pr_ready_to_merge(pr, checks_summary, new_review_items):
|
||||
actions.append("stop_ready_to_merge")
|
||||
actions.append("ready_to_merge")
|
||||
return unique_actions(actions)
|
||||
|
||||
if new_review_items:
|
||||
@@ -606,12 +605,6 @@ def collect_snapshot(args):
|
||||
if not state.get("started_at"):
|
||||
state["started_at"] = int(time.time())
|
||||
|
||||
# `gh pr checks -R <repo>` requires an explicit PR/branch/url argument.
|
||||
# After resolving `--pr auto`, reuse the concrete PR number.
|
||||
checks = get_pr_checks(str(pr["number"]), repo=pr["repo"])
|
||||
checks_summary = summarize_checks(checks)
|
||||
workflow_runs = get_workflow_runs_for_sha(pr["repo"], pr["head_sha"])
|
||||
failed_runs = failed_runs_from_workflow_runs(workflow_runs, pr["head_sha"])
|
||||
authenticated_login = get_authenticated_login()
|
||||
new_review_items = fetch_new_review_items(
|
||||
pr,
|
||||
@@ -619,6 +612,15 @@ def collect_snapshot(args):
|
||||
fresh_state=fresh_state,
|
||||
authenticated_login=authenticated_login,
|
||||
)
|
||||
# Surface review feedback before drilling into CI and mergeability details.
|
||||
# That keeps the babysitter responsive to new comments even when other
|
||||
# actions are also available.
|
||||
# `gh pr checks -R <repo>` requires an explicit PR/branch/url argument.
|
||||
# After resolving `--pr auto`, reuse the concrete PR number.
|
||||
checks = get_pr_checks(str(pr["number"]), repo=pr["repo"])
|
||||
checks_summary = summarize_checks(checks)
|
||||
workflow_runs = get_workflow_runs_for_sha(pr["repo"], pr["head_sha"])
|
||||
failed_runs = failed_runs_from_workflow_runs(workflow_runs, pr["head_sha"])
|
||||
|
||||
retries_used = current_retry_count(state, pr["head_sha"])
|
||||
actions = recommend_actions(
|
||||
@@ -761,7 +763,6 @@ def run_watch(args):
|
||||
if (
|
||||
"stop_pr_closed" in actions
|
||||
or "stop_exhausted_retries" in actions
|
||||
or "stop_ready_to_merge" in actions
|
||||
):
|
||||
print_event("stop", {"actions": snapshot.get("actions"), "pr": snapshot.get("pr")})
|
||||
return 0
|
||||
@@ -769,13 +770,13 @@ def run_watch(args):
|
||||
current_change_key = snapshot_change_key(snapshot)
|
||||
changed = current_change_key != last_change_key
|
||||
green = is_ci_green(snapshot)
|
||||
pr = snapshot.get("pr") or {}
|
||||
pr_open = not bool(pr.get("closed")) and not bool(pr.get("merged"))
|
||||
|
||||
if not green:
|
||||
if not green or pr_open:
|
||||
poll_seconds = args.poll_seconds
|
||||
elif changed or last_change_key is None:
|
||||
poll_seconds = args.poll_seconds
|
||||
else:
|
||||
poll_seconds = min(poll_seconds * 2, GREEN_STATE_MAX_POLL_SECONDS)
|
||||
|
||||
last_change_key = current_change_key
|
||||
time.sleep(poll_seconds)
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
import argparse
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).with_name("gh_pr_watch.py")
|
||||
MODULE_SPEC = importlib.util.spec_from_file_location("gh_pr_watch", MODULE_PATH)
|
||||
gh_pr_watch = importlib.util.module_from_spec(MODULE_SPEC)
|
||||
assert MODULE_SPEC.loader is not None
|
||||
MODULE_SPEC.loader.exec_module(gh_pr_watch)
|
||||
|
||||
|
||||
def sample_pr():
|
||||
return {
|
||||
"number": 123,
|
||||
"url": "https://github.com/openai/codex/pull/123",
|
||||
"repo": "openai/codex",
|
||||
"head_sha": "abc123",
|
||||
"head_branch": "feature",
|
||||
"state": "OPEN",
|
||||
"merged": False,
|
||||
"closed": False,
|
||||
"mergeable": "MERGEABLE",
|
||||
"merge_state_status": "CLEAN",
|
||||
"review_decision": "",
|
||||
}
|
||||
|
||||
|
||||
def sample_checks(**overrides):
|
||||
checks = {
|
||||
"pending_count": 0,
|
||||
"failed_count": 0,
|
||||
"passed_count": 12,
|
||||
"all_terminal": True,
|
||||
}
|
||||
checks.update(overrides)
|
||||
return checks
|
||||
|
||||
|
||||
def test_collect_snapshot_fetches_review_items_before_ci(monkeypatch, tmp_path):
|
||||
call_order = []
|
||||
pr = sample_pr()
|
||||
|
||||
monkeypatch.setattr(gh_pr_watch, "resolve_pr", lambda *args, **kwargs: pr)
|
||||
monkeypatch.setattr(gh_pr_watch, "load_state", lambda path: ({}, True))
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"get_authenticated_login",
|
||||
lambda: call_order.append("auth") or "octocat",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"fetch_new_review_items",
|
||||
lambda *args, **kwargs: call_order.append("review") or [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"get_pr_checks",
|
||||
lambda *args, **kwargs: call_order.append("checks") or [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"summarize_checks",
|
||||
lambda checks: call_order.append("summarize") or sample_checks(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"get_workflow_runs_for_sha",
|
||||
lambda *args, **kwargs: call_order.append("workflow") or [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"failed_runs_from_workflow_runs",
|
||||
lambda *args, **kwargs: call_order.append("failed_runs") or [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"recommend_actions",
|
||||
lambda *args, **kwargs: call_order.append("recommend") or ["idle"],
|
||||
)
|
||||
monkeypatch.setattr(gh_pr_watch, "save_state", lambda *args, **kwargs: None)
|
||||
|
||||
args = argparse.Namespace(
|
||||
pr="123",
|
||||
repo=None,
|
||||
state_file=str(tmp_path / "watcher-state.json"),
|
||||
max_flaky_retries=3,
|
||||
)
|
||||
|
||||
gh_pr_watch.collect_snapshot(args)
|
||||
|
||||
assert call_order.index("review") < call_order.index("checks")
|
||||
assert call_order.index("review") < call_order.index("workflow")
|
||||
|
||||
|
||||
def test_recommend_actions_prioritizes_review_comments():
|
||||
actions = gh_pr_watch.recommend_actions(
|
||||
sample_pr(),
|
||||
sample_checks(failed_count=1),
|
||||
[{"run_id": 99}],
|
||||
[{"kind": "review_comment", "id": "1"}],
|
||||
0,
|
||||
3,
|
||||
)
|
||||
|
||||
assert actions == [
|
||||
"process_review_comment",
|
||||
"diagnose_ci_failure",
|
||||
"retry_failed_checks",
|
||||
]
|
||||
|
||||
|
||||
def test_run_watch_keeps_polling_open_ready_to_merge_pr(monkeypatch):
|
||||
sleeps = []
|
||||
events = []
|
||||
snapshot = {
|
||||
"pr": sample_pr(),
|
||||
"checks": sample_checks(),
|
||||
"failed_runs": [],
|
||||
"new_review_items": [],
|
||||
"actions": ["ready_to_merge"],
|
||||
"retry_state": {
|
||||
"current_sha_retries_used": 0,
|
||||
"max_flaky_retries": 3,
|
||||
},
|
||||
}
|
||||
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"collect_snapshot",
|
||||
lambda args: (snapshot, Path("/tmp/codex-babysit-pr-state.json")),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
gh_pr_watch,
|
||||
"print_event",
|
||||
lambda event, payload: events.append((event, payload)),
|
||||
)
|
||||
|
||||
class StopWatch(Exception):
|
||||
pass
|
||||
|
||||
def fake_sleep(seconds):
|
||||
sleeps.append(seconds)
|
||||
if len(sleeps) >= 2:
|
||||
raise StopWatch
|
||||
|
||||
monkeypatch.setattr(gh_pr_watch.time, "sleep", fake_sleep)
|
||||
|
||||
with pytest.raises(StopWatch):
|
||||
gh_pr_watch.run_watch(argparse.Namespace(poll_seconds=30))
|
||||
|
||||
assert sleeps == [30, 30]
|
||||
assert [event for event, _ in events] == ["snapshot", "snapshot"]
|
||||
Reference in New Issue
Block a user