[codex] Ignore pending PR review comments (#27080)

## Why The PR babysitter could surface inline comments from a GitHub review that was still in the `PENDING` state. That allowed Codex to start acting on feedback before the reviewer submitted it. ## What changed - Correlate inline comments with their parent review and ignore pending reviews and their comments. - Remove pending review IDs from saved watcher state so the feedback surfaces normally after publication. - Update the skill instructions and add regression coverage for the draft-to-published transition. ## Validation - `python3 -m pytest .codex/skills/babysit-pr/scripts/test_gh_pr_watch.py` - Skill package validation with `quick_validate.py` - Live verification on #26835: the draft comment stayed hidden and surfaced after the review was submitted.
2026-07-01 00:31:56 +08:00 · 2026-06-09 08:29:32 -07:00
parent 1547785657
commit a770e5b847
5 changed files with 103 additions and 6 deletions
@@ -28,7 +28,7 @@ Accept any of the following:
 3. Inspect the `actions` list in the JSON response.
 4. If `diagnose_ci_failure` is present, inspect failed run logs and classify the failure.
 5. If the failure is likely caused by the current branch, patch code locally, commit, and push. Do not patch random flaky tests, CI infrastructure, dependency outages, runner issues, or other failures that are unrelated to the branch.
-6. If `process_review_comment` is present, inspect surfaced review items and decide whether to address them.
+6. If `process_review_comment` is present, inspect surfaced published review items and decide whether to address them.
 7. If a review item is actionable and correct, patch code locally, commit, push, and then resolve the associated review thread only when allowed by the GitHub state mutation policy below.
 8. Do not post replies to human-authored review comments/threads unless the user explicitly confirms the exact response. If a human review item is non-actionable, already addressed, or not valid, surface the item and recommended response to the user instead of replying on GitHub.
 9. If the failure is likely flaky/unrelated and `retry_failed_checks` is present, rerun failed jobs with `--retry-failed-now`.
@@ -92,9 +92,13 @@ The watcher surfaces review items from:
 - Inline review comments
 - Review submissions (COMMENT / APPROVED / CHANGES_REQUESTED)

+Only act on published feedback. Ignore review submissions in GitHub's `PENDING` state and inline
+comments attached to those pending reviews. Do not mark pending review feedback as seen; it should
+be eligible to surface after the reviewer submits the review.
+
 It intentionally surfaces Codex reviewer bot feedback (for example comments/reviews from `chatgpt-codex-connector[bot]`) in addition to human reviewer feedback. Most unrelated bot noise should still be ignored.
 For safety, the watcher only auto-surfaces trusted human review authors (for example repo OWNER/MEMBER/COLLABORATOR, plus the authenticated operator) and approved review bots such as Codex.
-On a fresh watcher state file, existing pending review feedback may be surfaced immediately (not only comments that arrive after monitoring starts). This is intentional so already-open review comments are not missed.
+On a fresh watcher state file, existing unaddressed published review feedback may be surfaced immediately (not only comments that arrive after monitoring starts). This is intentional so already-open review comments are not missed.

 When you agree with a comment and it is actionable:

@@ -1,4 +1,4 @@
 interface:
  display_name: "PR Babysitter"
  short_description: "Watch PR review comments, CI, and merge conflicts"
-  default_prompt: "Babysit the current PR: monitor reviewer comments, CI, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); surface new review feedback before acting on CI or mergeability work, fix valid issues, push updates, and rerun flaky failures up to 3 times. Do not post replies to human-authored review comments unless the user explicitly confirms the exact response. Do not patch unrelated flaky tests, CI infrastructure, dependency outages, runner issues, or other failures that are not caused by the branch. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Do not treat a green + mergeable PR as a terminal stop while it is still open; continue polling autonomously after any push/rerun so newly posted review comments are surfaced until a strict terminal stop condition is reached or the user interrupts."
+  default_prompt: "Babysit the current PR: monitor published reviewer comments, CI, and merge-conflict status (prefer the watcher’s --watch mode for live monitoring); ignore unpublished comments in pending GitHub reviews; surface new published review feedback before acting on CI or mergeability work, fix valid issues, push updates, and rerun flaky failures up to 3 times. Do not post replies to human-authored review comments unless the user explicitly confirms the exact response. Do not patch unrelated flaky tests, CI infrastructure, dependency outages, runner issues, or other failures that are not caused by the branch. Keep exactly one watcher session active for the PR (do not leave duplicate --watch terminals running). If you pause monitoring to patch review/CI feedback, restart --watch yourself immediately after the push in the same turn. If a watcher is still running and no strict stop condition has been reached, the task is still in progress: keep consuming watcher output and sending progress updates instead of ending the turn. Do not treat a green + mergeable PR as a terminal stop while it is still open; continue polling autonomously after any push/rerun so newly posted review comments are surfaced until a strict terminal stop condition is reached or the user interrupts."
@@ -44,6 +44,9 @@ Reruns only failed jobs (and dependencies) for a workflow run.
 - Review submissions:
  - `gh api repos/{owner}/{repo}/pulls/<pr_number>/reviews?per_page=100`

+Use each inline comment's `pull_request_review_id` to find its parent review. Ignore parent reviews
+whose `state` is `PENDING`, along with their inline comments, until the review is submitted.
+
 ## JSON fields consumed by the watcher

 ### `gh pr view`
@@ -452,11 +452,14 @@ def normalize_issue_comments(items):
    return out


-def normalize_review_comments(items):
+def normalize_review_comments(items, review_states):
    out = []
    for item in items:
        if not isinstance(item, dict):
            continue
+        review_id = str(item.get("pull_request_review_id") or "")
+        if review_states.get(review_id) == "PENDING":
+            continue
        line = item.get("line")
        if line is None:
            line = item.get("original_line")
@@ -481,6 +484,8 @@ def normalize_reviews(items):
    for item in items:
        if not isinstance(item, dict):
            continue
+        if str(item.get("state") or "").upper() == "PENDING":
+            continue
        out.append(
            {
                "kind": "review",
@@ -534,16 +539,33 @@ def fetch_new_review_items(pr, state, fresh_state, authenticated_login=None):
    review_payload = gh_api_list_paginated(endpoints["review"], repo=repo)

    issue_items = normalize_issue_comments(issue_payload)
-    review_comment_items = normalize_review_comments(review_comment_payload)
+    review_states = {
+        str(item.get("id")): str(item.get("state") or "").upper()
+        for item in review_payload
+        if isinstance(item, dict) and item.get("id") not in (None, "")
+    }
+    pending_review_ids = {
+        review_id for review_id, review_state in review_states.items() if review_state == "PENDING"
+    }
+    pending_review_comment_ids = {
+        str(item.get("id"))
+        for item in review_comment_payload
+        if isinstance(item, dict)
+        and item.get("id") not in (None, "")
+        and str(item.get("pull_request_review_id") or "") in pending_review_ids
+    }
+    review_comment_items = normalize_review_comments(review_comment_payload, review_states)
    review_items = normalize_reviews(review_payload)
    all_items = issue_items + review_comment_items + review_items

    seen_issue = {str(x) for x in state.get("seen_issue_comment_ids") or []}
    seen_review_comment = {str(x) for x in state.get("seen_review_comment_ids") or []}
    seen_review = {str(x) for x in state.get("seen_review_ids") or []}
+    seen_review_comment.difference_update(pending_review_comment_ids)
+    seen_review.difference_update(pending_review_ids)

    # On a brand-new state file, surface existing review activity instead of
-    # silently treating it as seen. This avoids missing already-pending review
+    # silently treating it as seen. This avoids missing already-published review
    # feedback when monitoring starts after comments were posted.

    new_items = []
@@ -118,6 +118,74 @@ def test_recommend_actions_prioritizes_review_comments():
    ]


+def test_pending_review_feedback_surfaces_only_after_publication(monkeypatch):
+    state = {
+        "seen_review_comment_ids": ["20"],
+        "seen_review_ids": ["10"],
+    }
+    review = {
+        "id": 10,
+        "user": {"login": "octocat"},
+        "author_association": "MEMBER",
+        "state": "PENDING",
+        "body": "Please rename this.",
+        "created_at": "2026-06-08T10:00:00Z",
+        "submitted_at": None,
+        "html_url": "https://github.com/openai/codex/pull/123#pullrequestreview-10",
+    }
+    review_comment = {
+        "id": 20,
+        "pull_request_review_id": 10,
+        "user": {"login": "octocat"},
+        "author_association": "MEMBER",
+        "body": "Please rename this.",
+        "created_at": "2026-06-08T10:00:00Z",
+        "path": "src/example.rs",
+        "line": 7,
+        "html_url": "https://github.com/openai/codex/pull/123#discussion_r20",
+    }
+
+    def fake_list(endpoint, **kwargs):
+        if endpoint.endswith("/issues/123/comments"):
+            return []
+        if endpoint.endswith("/pulls/123/comments"):
+            return [review_comment]
+        if endpoint.endswith("/pulls/123/reviews"):
+            return [review]
+        raise AssertionError(f"unexpected endpoint: {endpoint}")
+
+    monkeypatch.setattr(gh_pr_watch, "gh_api_list_paginated", fake_list)
+
+    assert (
+        gh_pr_watch.fetch_new_review_items(
+            sample_pr(),
+            state,
+            fresh_state=True,
+            authenticated_login="octocat",
+        )
+        == []
+    )
+    assert state["seen_review_comment_ids"] == []
+    assert state["seen_review_ids"] == []
+
+    review["state"] = "COMMENTED"
+    review["submitted_at"] = "2026-06-08T10:05:00Z"
+
+    published_items = gh_pr_watch.fetch_new_review_items(
+        sample_pr(),
+        state,
+        fresh_state=False,
+        authenticated_login="octocat",
+    )
+
+    assert {(item["kind"], item["id"]) for item in published_items} == {
+        ("review", "10"),
+        ("review_comment", "20"),
+    }
+    assert state["seen_review_comment_ids"] == ["20"]
+    assert state["seen_review_ids"] == ["10"]
+
+
 def test_run_watch_keeps_polling_open_ready_to_merge_pr(monkeypatch):
    sleeps = []
    events = []