Python: Enhance Azure AI Search Citations with Document URLs in Foundry V2 (#4028)

* Python: Enhance Azure AI Search citations with document URLs in Foundry V2 (Responses API)

Override _parse_response_from_openai and _parse_chunk_from_openai in
RawAzureAIClient to extract get_urls from azure_ai_search_call_output
items and enrich url_citation annotations with document-specific URLs.

- Non-streaming: first pass collects get_urls, post-processes annotations
- Streaming: captures search output state, enriches url_citation events
  (also handles url_citation annotation type not handled by base class)
- Updated V2 sample to demonstrate citation URL extraction
- Added 14 unit tests covering extraction, enrichment, and edge cases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: rework search citation enrichment to override _inner_get_response

- Remove all direct openai/pydantic imports from _client.py
- Override _inner_get_response instead of _parse_response_from_openai/_parse_chunk_from_openai
- Use closure-local state for streaming instead of instance-level _streaming_search_get_urls
- Add _build_url_citation_content helper for streaming url_citation handling
- Fix mypy errors by using str(value or '') for Annotation TypedDict fields
- Fix docstring to say 'citation' instead of 'url_citation'
- Update tests to match new approach

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: handle streaming search citations from output_item.done events

The azure_ai_search_call_output item only has populated output data
(including get_urls) in the response.output_item.done event, not in
the response.output_item.added event. Also removed the search_get_urls
guard on url_citation handling so annotations are always produced even
if get_urls haven't been captured yet.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* addressed comments

* refactor: address PR review - eliminate type: ignore[assignment] pattern

Call super()._inner_get_response() independently in each branch instead
of once at the top with union type reassignment. Non-streaming uses
two-arg super() in the closure; streaming uses cast() for type narrowing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: remove defensive patterns per PR review

- Replace all getattr() with direct attribute access
- Remove cast() for streaming branch, use type: ignore[assignment]
- Simplify _build_url_citation_content to use dict access directly
- Simplify _extract_azure_search_urls to use item.type/item.output
- Handle empty list output from streaming 'added' events
- Update tests to match actual runtime types (objects, not dicts)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* mypy fix

* small fixes

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

python/samples/02-agents/providers/azure_ai/azure_ai_with_azure_ai_search.py

@@ -2,6 +2,7 @@
 import asyncio
 import os
 
+from agent_framework import Annotation
 from agent_framework.azure import AzureAIProjectAgentProvider
 from azure.identity.aio import AzureCliCredential
 from dotenv import load_dotenv
@@ -15,6 +16,9 @@ Azure AI Agent with Azure AI Search Example
 This sample demonstrates usage of AzureAIProjectAgentProvider with Azure AI Search
 to search through indexed data and answer user questions about it.
 
+Citations from Azure AI Search are automatically enriched with document-specific
+URLs (get_url) that can be used to retrieve the original documents.
+
 Prerequisites:
 1. Set AZURE_AI_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME environment variables.
 2. Ensure you have an Azure AI Search connection configured in your Azure AI project
@@ -29,8 +33,10 @@ async def main() -> None:
     ):
         agent = await provider.create_agent(
             name="MySearchAgent",
-            instructions="""You are a helpful assistant. You must always provide citations for
-            answers using the tool and render them as: `[message_idx:search_idx†source]`.""",
+            instructions=(
+                "You are a helpful agent that searches hotel information using Azure AI Search. "
+                "Always use the search tool and index to find hotel data and provide accurate information."
+            ),
             tools={
                 "type": "azure_ai_search",
                 "azure_ai_search": {
@@ -46,11 +52,59 @@ async def main() -> None:
             },
         )
 
-        query = "Tell me about insurance options"
+        query = (
+            "Use Azure AI search knowledge tool to find detailed information about a winter hotel."
+            " Use the search tool and index."  # You can modify prompt to force tool usage
+        )
         print(f"User: {query}")
+
+        # Non-streaming: get response with enriched citations
         result = await agent.run(query)
         print(f"Result: {result}\n")
 
+        # Display citations with document-specific URLs
+        if result.messages:
+            citations: list[Annotation] = []
+            for msg in result.messages:
+                for content in msg.contents:
+                    if hasattr(content, "annotations") and content.annotations:
+                        citations.extend(content.annotations)
+
+            if citations:
+                print("Citations:")
+                for i, citation in enumerate(citations, 1):
+                    url = citation.get("url", "N/A")
+                    # get_url contains the document-specific REST API URL from Azure AI Search
+                    get_url = (citation.get("additional_properties") or {}).get("get_url")
+                    print(f"  [{i}] {citation.get('title', 'N/A')}")
+                    print(f"      URL: {url}")
+                    if get_url:
+                        print(f"      Document URL: {get_url}")
+
+        # Streaming: collect citations from streamed response
+        print("\n--- Streaming ---")
+        print(f"User: {query}")
+        print("Agent: ", end="", flush=True)
+        streaming_citations: list[Annotation] = []
+        async for chunk in agent.run(query, stream=True):
+            if chunk.text:
+                print(chunk.text, end="", flush=True)
+            for content in getattr(chunk, "contents", []):
+                annotations = getattr(content, "annotations", [])
+                if annotations:
+                    streaming_citations.extend(annotations)
+
+        print()
+        if streaming_citations:
+            print("\nStreaming Citations:")
+            for i, citation in enumerate(streaming_citations, 1):
+                url = citation.get("url", "N/A")
+                get_url = (citation.get("additional_properties") or {}).get("get_url")
+                print(f"  [{i}] {citation.get('title', 'N/A')}")
+                print(f"      URL: {url}")
+                if get_url:
+                    print(f"      Document URL: {get_url}")
+
 
 if __name__ == "__main__":
     asyncio.run(main())