Python: Fix _deduplicate_messages catch-all branch dropping valid repeated messages (#4716)

* Fix _deduplicate_messages catch-all branch dropping valid repeated messages (#4682)

Remove the catch-all dedup branch that used (role, hash(content_str)) as a
dedup key. This incorrectly treated any two messages with the same role and
identical content as duplicates, dropping valid repeated messages (e.g., a
user saying 'yes' to confirm two separate things).

The tool-specific dedup branches (tool results by call_id, assistant tool
calls by call_id tuple) remain unchanged as they correctly identify true
protocol-level duplicates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: consecutive-duplicate detection for non-tool messages (#4682)

- Replace blanket dedup removal with consecutive-duplicate detection:
  only skip a message if the immediately preceding message has the same
  role and content, preserving protection against upstream replays while
  allowing identical messages at different conversation points.
- Strengthen test assertions to verify message identity and order, not
  just list length.
- Add tests for consecutive duplicate skipping, non-consecutive
  preservation, and messages with contents=None.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply pre-commit auto-fixes

* Use message_id for deduplication instead of content hashing

Deduplicate general messages by message_id when available, replacing
the consecutive-duplicate content check. Two messages with the same id
are definitively the same message (upstream replay), while identical
content with distinct ids (e.g. repeated "yes" confirmations) is
preserved. Messages without a message_id are always kept.

* Fix message_id dedup: truthy check, content-hash fallback, log safety

- Use truthy check (`if msg.message_id`) instead of `is not None` so
  empty-string IDs fall through to content-hash dedup rather than
  collapsing unrelated messages.
- Add content-hash fallback for messages without message_id, preventing
  false negatives from integrations that don't set IDs.
- Remove raw message_id from log format string (addresses log-injection
  surface with control characters).
- Add tests for empty-string message_id edge cases.
- Update existing tests to reflect content-hash dedup behavior.

Fixes #4682

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py

@@ -242,8 +242,16 @@ def _deduplicate_messages(messages: list[Message]) -> list[Message]:
             unique_messages.append(msg)
 
         else:
-            content_str = str([str(c) for c in msg.contents]) if msg.contents else ""
-            key = (role_value, hash(content_str))
+            # Use message_id for deduplication when available — two messages with the
+            # same id are definitively the same message (e.g. upstream replays), while
+            # different messages that happen to share identical content (e.g. repeated
+            # "yes" confirmations) will have distinct ids and be preserved.
+            # Fall back to content-hash when message_id is absent or empty.
+            if msg.message_id:
+                key = ("id", msg.message_id)
+            else:
+                content_str = str([str(c) for c in msg.contents]) if msg.contents else ""
+                key = ("content", role_value, hash(content_str))
 
             if key in seen_keys:
                 logger.info(f"Skipping duplicate message at index {idx}: role={role_value}")