[Python] Add agent-framework-azure-ai-contentunderstanding package (#4829)

* feat: add agent-framework-azure-contentunderstanding package Add Azure Content Understanding integration as a context provider for the Agent Framework. The package automatically analyzes file attachments (documents, images, audio, video) using Azure CU and injects structured results (markdown, fields) into the LLM context. Key features: - Multi-document session state with status tracking (pending/ready/failed) - Configurable timeout with async background fallback for large files - Output filtering via AnalysisSection enum - Auto-registered list_documents() and get_analyzed_document() tools - Supports all CU modalities: documents, images, audio, video - Content limits enforcement (pages, file size, duration) - Binary stripping of supported files from input messages Public API: - ContentUnderstandingContextProvider (main class) - AnalysisSection (output section selector enum) - ContentLimits (configurable limits dataclass) Tests: 46 unit tests, 91% coverage, all linting and type checks pass. * fix: update CU fixtures with real API data, fix test assertions - Replace synthetic fixtures with real CU API responses (sanitized) - Update test assertions to match real data (Contoso vs CONTOSO, TotalAmount vs InvoiceTotal, field values from real analysis) - Add --pre install note in README (preview package) - Document unenforced ContentLimits fields (max_pages, duration) * chore: add connector .gitignore, update uv.lock * refactor: rename to azure-ai-contentunderstanding, fix CI issues Align naming with Azure SDK convention and AF pattern: - Directory: azure-contentunderstanding -> azure-ai-contentunderstanding - PyPI: agent-framework-azure-contentunderstanding -> agent-framework-azure-ai-contentunderstanding - Module: agent_framework_azure_contentunderstanding -> agent_framework_azure_ai_contentunderstanding CI fixes: - Inline conftest helpers to avoid cross-package import collision in xdist - Remove PyPI badge and dead API reference link from README (package not published yet) * feat: add samples (document_qa, invoice_processing, multimodal_chat) - document_qa.py: Single PDF upload, CU context provider, follow-up Q&A - invoice_processing.py: Structured field extraction with prebuilt-invoice - multimodal_chat.py: Multi-file session with status tracking - Add ruff per-file-ignores for samples/ directory - Update README with samples section, env vars, and run instructions * feat: add remaining samples (devui_multimodal_agent, large_doc_file_search) - S3: devui_multimodal_agent/ — DevUI web UI with CU-powered file analysis - S4: large_doc_file_search.py — CU extraction + OpenAI vector store RAG - Update README and samples/README.md with all 5 samples * feat: add file_search integration for large document RAG Add FileSearchConfig — when provided, CU-extracted markdown is automatically uploaded to an OpenAI vector store and a file_search tool is registered on the context. This enables token-efficient RAG retrieval for large documents without users needing to manage vector stores manually. - FileSearchConfig dataclass (openai_client, vector_store_name) - Auto-create vector store, upload markdown, register file_search tool - Auto-cleanup on close() - When file_search is enabled, skip full content injection (use RAG instead) - Update large_doc_file_search sample to use the integration - 4 new tests (50 total, 90% coverage) * fix: add key-based auth support to all samples Follow established AF pattern: check for API key env var first, fall back to AzureCliCredential. Supports AZURE_OPENAI_API_KEY and AZURE_CONTENTUNDERSTANDING_API_KEY environment variables. * FEATURE(python): add analyzer auto-detection, file_search RAG, and lazy init _context_provider.py: - Make analyzer_id optional (default None) with auto-detection by media type prefix: audio->audioSearch, video->videoSearch, else documentSearch - Add _ensure_initialized() for lazy client creation in before_run() - Add FileSearchConfig-based vector store upload - Fix: background-completed docs in file_search mode now upload to vector store instead of injecting full markdown into context messages - Add _pending_uploads queue for deferred vector store uploads devui_file_search_agent/ (new sample): - DevUI agent combining CU extraction + OpenAI file_search RAG azure_responses_agent (existing sample fix): - Add AzureCliCredential support and AZURE_AI_PROJECT_ENDPOINT fallback Tests (19 new), Docs updated (AGENTS.md, README.md) * feat(cu): MIME sniffing, media-aware formatting, unified timeout, vector store expiration - Add three-layer MIME detection (fast path → filetype binary sniff → filename fallback) to handle unreliable upstream MIME types (e.g. mp4 sent as application/octet-stream). Adds filetype>=1.2,<2 dependency. - Media-aware output formatting: video shows duration/resolution + all fields as JSON; audio promotes Summary as prose; document unchanged. - Unified timeout for all media types (removed file_search special-case that waited indefinitely for video/audio). All files use max_wait with background polling fallback. - Vector store created with expires_after=1 day as crash safety net. - Add 8 MIME sniffing tests (TestMimeSniffing class). * fix: merge all CU content segments for video/audio analysis CU's prebuilt-videoSearch and prebuilt-audioSearch analyzers split long media files into multiple `contents[]` segments. Previously, `_extract_sections()` only read `contents[0]`, causing truncated duration, missing transcript, and incomplete fields for any video/audio longer than a single scene. Now iterates all segments and merges: - duration: global min(startTimeMs) → max(endTimeMs) - markdown: concatenated with `---` separators - fields: same-named fields collected into per-segment list - metadata (kind, resolution): taken from first segment Single-segment results (documents, short audio) are unaffected. Update test fixture to realistic 3-segment video structure and expand assertions to verify multi-segment merging. Add documentation for multi-segment processing and speaker diarization limitation. * refactor: improve CU context provider docs and remove ContentLimits - Improve class docstring: clarify endpoint (Azure AI Foundry URL with example), credential (AzureKeyCredential vs Entra ID), and analyzer_id (prebuilt/custom with auto-selection behavior and reference links) - Add SUPPORTED_MEDIA_TYPES comments explaining MIME-based matching behavior and add missing file types per CU service docs - Use namespaced logger to align with other packages - Remove ContentLimits and related code/tests - Rename DEFAULT_MAX_WAIT to DEFAULT_MAX_WAIT_SECONDS for clarity * feat: support user-provided vector store in FileSearchConfig - Add vector_store_id field to FileSearchConfig (None = auto-create) - Track _owns_vector_store to only delete auto-created stores on close() - Remove vector_store_name; use internal _DEFAULT_VECTOR_STORE_NAME - Add inline comments for private state fields - Document output_sections default in docstring - Update AGENTS.md, samples, and tests * fix: remove ContentLimits from README code block * refactor: create CU client in __init__ instead of __aenter__ Follow Azure AI Search provider pattern: create the client eagerly in __init__, make __aenter__ a no-op. This ensures __aexit__/close() is always safe to call and eliminates the _ensure_initialized() workaround. * docs: add file_search param to class docstring * feat: introduce FileSearchBackend abstraction for cross-client support Replace direct OpenAI client usage with FileSearchBackend ABC: - OpenAIFileSearchBackend: for OpenAIChatClient (Responses API) - FoundryFileSearchBackend: for FoundryChatClient (Azure Foundry) - Shared base _OpenAICompatBackend for common vector store CRUD FileSearchConfig now takes a backend instead of openai_client. Factory methods from_openai() and from_foundry() for convenience. BREAKING: FileSearchConfig(openai_client=...) -> FileSearchConfig.from_openai(...) * refactor: FileSearchBackend abstraction + caller-owned vector store * fix: file_search reliability and sample improvements - Poll vector store indexing (create_and_poll) to ensure file_search returns results immediately after upload - Set status to failed when vector store upload fails - Skip get_analyzed_document tool in file_search mode to prevent LLM from bypassing RAG - Simplify sample auth: single credential, direct parameters - Use from_foundry backend for Foundry project endpoints * perf: set max_num_results=10 for file_search to reduce token usage * fix: move import to top of file (E402 lint) * chore: remove unused imports * fix: align azure-ai-contentunderstanding with MAF coding conventions - Add module-level docstrings to __init__.py and _context_provider.py - Use Self return type for __aenter__ (with typing_extensions fallback) - Use explicit typed params for __aexit__ signature - Add sync TokenCredential to AzureCredentialTypes union - Pass AGENT_FRAMEWORK_USER_AGENT to ContentUnderstandingClient - Remove unused ContentLimits from public API and tests - Fix FileSearchConfig tests to match refactored backend API - Fix lifecycle tests to match eager client initialization * refactor: improve CU context provider API surface and fix CI - Refactor _analyze_file to return DocumentEntry instead of mutating dict - Remove TokenCredential from AzureCredentialTypes (fixes mypy/pyright CI) - Remove OpenAIFileSearchBackend/FoundryFileSearchBackend from public API (internal to FileSearchConfig factory methods) - Remove DocumentStatus from public exports (implementation detail) - Update file_search comments to reflect backend-agnostic design - Add DocumentStatus enum, analysis/upload duration tracking - Add combined timeout for CU analysis + vector store upload * fix: improve file_search samples and move tool guidelines to context provider - Delete redundant devui_file_search_agent sample (duplicate of azure_openai variant) - Move tool usage guidelines from sample agent instructions into context provider (extend_instructions in step 6, applied automatically for all file_search users) - Fix file_search purpose: use from_foundry() for Azure OpenAI (purpose="assistants") - Add filename hint in upload instructions for targeted file_search queries - Reduce max_num_results from 10 to 3 in both devui samples - Simplify agent instructions in both samples (remove tool-specific guidance) * feat: improve source_id, integration tests, and content assertions - Rename DEFAULT_SOURCE_ID to "azure_ai_contentunderstanding" (matches azure_ai_search convention) - Improve source_id docstring to describe default value - Clarify _detect_and_strip_files docstring (CU-supported files) - Add invoice.pdf test fixture from Azure CU samples repo - Refactor integration tests to use invoice.pdf directly (assert instead of skip when fixture missing) - Add URI content test (Content.from_uri with external URL) - Add "CONTOSO LTD." content assertion to all integration tests - Use max_wait=None in integration tests (wait until complete) * feat: reject duplicate filenames, add integration tests and sample comments - Reject duplicate document keys in before_run (skip + warn LLM to rename) - Update _derive_doc_key docstring to document uniqueness constraint - Add unit tests for duplicate filename rejection (cross-turn and same-turn) - Add integration test for data URI content (from_uri with base64) - Add integration test for background analysis (max_wait timeout + resolve) - Add filename recommendation comments to all samples' Content.from_data() * chore: improve doc key derivation, comments, and README - Replace hash-based doc key with uuid4 for anonymous uploads (O(1), no payload traversal) - Remove hashlib import (no longer needed) - Add File Naming section to README (filename importance, duplicate rejection) - Improve inline comments (_derive_doc_key, _extract_binary, URL parsing) * test: strengthen _format_result assertions with exact expected strings - Replace loose 'in' checks with exact 'assert formatted == expected' for both multi-segment and single-segment format tests - Add object-type fields (ShippingAddress, Speakers) to test data to cover nested dict/list serialization - Add position-based ordering assertions to verify structural correctness (header -> markdown -> fields across segments) * refactor: move invoice.pdf to shared sample_assets directory - Move invoice.pdf from tests/cu/test_data/ to python/samples/shared/sample_assets/ as single source of truth - Add INVOICE_PDF_PATH constant in test_integration.py pointing to the shared location - Update document_qa.py, invoice_processing.py, large_doc_file_search.py to use invoice.pdf instead of sample.pdf * refactor: reorganize samples into numbered dirs and simplify auth - Move script samples into 01-get-started/ with numbered prefixes (01_document_qa, 02_multimodal_chat, 03_invoice_processing, 04_large_doc_file_search) - Move devui samples into 02-devui/ with 01-multimodal_agent and 02-file_search_agent/{azure_openai_backend,foundry_backend} - Move invoice.pdf to CU package-local samples/shared/sample_assets/ - Replace kwargs dicts with direct constructor calls; support both API key (AZURE_OPENAI_API_KEY) and AzureCliCredential - Update README sample table with new paths * fix: resolve CI lint errors (D205, RUF001, E501) - Fix D205: single-line docstring summary for _detect_and_strip_files - Fix RUF001: replace EN DASH with HYPHEN-MINUS in segment headers - Fix E501: wrap long assertion lines in tests - Also includes samples reorg and auth simplification * refactor: overhaul samples — FoundryChatClient, sessions, remove get_analyzed_document Samples: - Switch all samples from deprecated AzureOpenAIResponsesClient to FoundryChatClient - Add 02_multi_turn_session.py showing AgentSession persistence across turns - Rewrite 03_multimodal_chat.py with real PDF + audio + video (parallel analysis), per-modality follow-ups, cross-document question, elapsed time, user prompts, and input token counts - Renumber: 02->03 multimodal, 03->04 invoice, 04->05 file_search Context provider: - Remove get_analyzed_document tool -- full content is in conversation history via InMemoryHistoryProvider, no retrieval tool needed - Remove follow-up turn instructions about tools - Only list_documents tool remains (for status queries) - Update README to reflect tool removal * feat: add 05_background_analysis sample and fix 04 session/max_wait - Add 05_background_analysis.py demonstrating non-blocking CU analysis with max_wait=1s, status tracking via list_documents(), and automatic background task resolution on subsequent turns - Fix 04_invoice_processing.py: add max_wait=None and AgentSession - Rename 05→06 large_doc_file_search - Update README sample table * docs: update README and fix sample 06 README: - Switch Quick Start from AzureOpenAIResponsesClient to FoundryChatClient - Add AgentSession to Quick Start example - Fix status values: pending -> analyzing/uploading/ready/failed - Fix env var: AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME -> AZURE_OPENAI_DEPLOYMENT_NAME - Update samples section with new paths, link to samples/README.md - Update multi-segment description to reflect per-segment fields Sample 06: - Fix from_openai -> from_foundry for Azure endpoints - Add AgentSession and max_wait=None * docs: rewrite README — concise format, prerequisites, CU link * fix: resolve pyright errors in _format_result segment cast * docs: add numbered section comments and fresh sample output to all samples - Add numbered section comments (# 1. ..., # 2. ...) per SAMPLE_GUIDELINES - Re-run all 6 samples and update expected output with real results - Fix duplicate sample output blocks in 04 and 05 - Update README code example to use public invoice URL * feat: add load_settings support for env var configuration - Make endpoint optional in constructor — auto-loads from AZURE_CONTENTUNDERSTANDING_ENDPOINT env var via load_settings() - Add ContentUnderstandingSettings TypedDict - Add env_file_path/env_file_encoding params for .env file support - Add 4 unit tests: env var loading, explicit override, missing endpoint error, missing credential error - Update README with env var auto-resolution docs - Follows framework convention used by all other packages * docs: polish README — fix duplicate env var, add Next steps, service limits link * chore: trim invoice fixture from 199K to 33 lines Keep only VendorName, InvoiceTotal, DueDate, InvoiceDate, InvoiceId fields and first 500 chars of markdown. Strip spans/source/coordinates. Reduces fixture from 6.6MB to 1.2KB. * feat: per-file analyzer_id override via additional_properties - Read analyzer_id from Content.additional_properties for per-file override - Resolution order: per-file > provider-level > auto-detect by media type - Update class docstring documenting filename and analyzer_id properties - Update sample 04 to demonstrate per-file override (prebuilt-invoice) - Add unit test for per-file analyzer override * Trim PDF test fixture and clarify unique filename requirement - Trim analyze_pdf_result.json from 4427 to 23 lines by removing pages, words, lines, paragraphs, sections, spans, and source fields that are not used by any unit test. - Add docstring note that filename must be unique within a session; duplicate filenames are rejected and the file will not be analyzed. * Update python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_context_provider.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_context_provider.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/azure_openai_backend/agent.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update python/packages/azure-ai-contentunderstanding/samples/02-devui/01-multimodal_agent/agent.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update python/packages/azure-ai-contentunderstanding/samples/01-get-started/06_large_doc_file_search.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix AGENTS.md to match implementation; remove unused variable in test helper AGENTS.md: - Remove _ensure_initialized() reference (client is created in __init__) - Fix multi-segment docs: segments kept as list, not merged into fields - Remove get_analyzed_document() reference (only list_documents registered) - Update sample names to match current directory structure test_context_provider.py: - Simplify _make_data_uri() — remove unused 'encoded' variable * Fix premature file_search instruction for background-completed docs - Change _resolve_pending_tasks() instruction from 'Use file_search' to 'being indexed' since the upload hasn't completed yet at that point. - Add LLM instruction on upload failure in step 1b so the agent can inform the user the document isn't searchable. * fix: wrap long line in devui agent instructions (E501) * Fix Copilot review: unused logger, stray code in README, await cancelled tasks - _file_search.py: Remove unused logger and logging import - 01-multimodal_agent/README.md: Remove accidentally pasted Python script - _context_provider.py close(): Await cancelled tasks before closing client to prevent 'Task destroyed but pending' warnings * Sanitize doc keys and fix duplicate filename re-injection - Add _sanitize_doc_key() to strip control characters, collapse whitespace, and cap length at 255 chars — prevents prompt injection via crafted filenames in extend_instructions() calls. - Track accepted doc_keys in step 3 so step 5 only injects content for files actually analyzed this turn, not pre-existing duplicates. - Soften duplicate upload instruction wording (remove IMPORTANT/caps). * fix: add type annotation to tasks_to_cancel for pyright * Move per-session mutable state to state dict for session isolation Previously _pending_tasks, _pending_uploads, and _uploaded_file_ids were stored on self, shared across all sessions. This caused cross-session leakage: Session A's background task results could be injected into Session B's context. Now these are stored in the per-session state dict. Global copies (_all_pending_tasks, _all_uploaded_file_ids) are kept on self only for best-effort cleanup in close(). Add 2 new TestSessionIsolation tests verifying that background tasks and resolved content stay within their originating session. * Remove unused AnalysisSection enum values Only MARKDOWN and FIELDS are handled by _extract_sections(). Remove FIELD_GROUNDING, TABLES, PARAGRAPHS, SECTIONS to avoid exposing dead options to users. * Recursively flatten object/array field values for cleaner LLM output - Use SDK .value property with recursive extraction for object/array fields - Object: AmountDue -> {Amount: 610, CurrencyCode: USD} (was raw SDK dict) - Array: LineItems -> list of flattened items (was raw SDK list) - Update invoice fixture with object/array fields from prebuilt-invoice - Add 3 unit tests for object, array, and nested object field extraction * Preserve sub-field confidence; compare full expected JSON in tests * Remove incorrect MIME aliases (audio/mp4, video/x-matroska) * feat: add AnalysisInput, content_range, warnings, and category support - Use SDK AnalysisInput model instead of raw body dict for begin_analyze - Forward content_range from additional_properties to CU (page/time ranges) - Extract CU warnings with code/message/target (ODataV4Format) into output - Include content-level category from classifier analyzers - Add 5 new tests: warnings, category, content_range forwarding - Fix pyright with explicit casts; fix en-dash lint (RUF002) * fix: falsy-0 bug in duration calc; improve test coverage - Fix start_time_ms=0 treated as falsy by 'or' short-circuit, use 'is None' checks instead for duration and segment time extraction - Update warnings test to use RAI ContentFiltered codes - Enrich warnings extraction to include code/message/target (ODataV4Format) - Add multi-segment video category test with per-segment assertions * refactor: split _context_provider.py into focused modules - Extract _constants.py: SUPPORTED_MEDIA_TYPES, MIME_ALIASES, analyzer maps - Extract _detection.py: file detection, MIME sniffing, doc key derivation - Extract _extraction.py: result extraction, field flattening, LLM formatting - _context_provider.py delegates via thin wrappers (793 lines, was 1255) - Update test imports to use _constants.py for SUPPORTED_MEDIA_TYPES * docs: update AGENTS.md with DocumentStatus, FileSearchBackend, and _file_search.py * refactor: replace AnalysisSection enum with Literal type for simpler DX - Remove AnalysisSection(str, Enum) class, replace with Literal["markdown", "fields"] type alias - Users can now pass plain strings: output_sections=["markdown"] — no extra import needed - AnalysisSection type alias still exported for type annotation use - Update all samples, tests, and internal code to use string literals - Address PR review feedback (eavanvalkenburg) * refactor: replace asyncio.Task with continuation tokens for serializable state - Replace state["_pending_tasks"] (asyncio.Task — not serializable) with state["_pending_tokens"] (dict of continuation token strings) so the framework can persist session state to disk/storage - Resume pending analyses via Azure SDK continuation_token mechanism - Fix: resumed pollers have stale cached status (done() always False), use asyncio.wait_for(poller.result()) with 10s min timeout instead - Remove _background_poll(), _all_pending_tasks, and task cancellation - Address PR review feedback (eavanvalkenburg): state must be serializable * fix: resolve CI lint (RUF052) and mypy (call-overload) errors * feat: add structured output (Pydantic model) to invoice processing sample - Use response_format=InvoiceResult for schema-constrained LLM output - Use output_sections=["fields"] only (no markdown needed for structured output) - Add LowConfidenceField model with confidence values - Add comments about prebuilt-invoice extensive schema vs simplified model - Address PR review feedback (eavanvalkenburg): use structured response * fix: use FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL env vars in all samples Replace AZURE_AI_PROJECT_ENDPOINT → FOUNDRY_PROJECT_ENDPOINT and AZURE_OPENAI_DEPLOYMENT_NAME → FOUNDRY_MODEL across all sample .py and README.md files. Address PR review feedback (eavanvalkenburg). * refactor: remove background_analysis sample, use FoundryChatClient in DevUI - Remove 05_background_analysis.py (per reviewer feedback — discuss max_wait design separately from samples) - Renumber 06_large_doc_file_search.py → 05_large_doc_file_search.py - Replace AzureOpenAIResponsesClient with FoundryChatClient in all DevUI samples - Replace client.as_agent() with Agent(client=client, ...) everywhere - Add max_wait comments explaining interactive vs batch usage - Update README.md and AGENTS.md - Address PR review feedback (eavanvalkenburg) * fix: vector_stores API moved from beta namespace in OpenAI SDK * docs: add comments about multi-file support and CU service limits in file_search sample * fix: broken markdown links after sample removal and renumbering * fix: migrate BaseContextProvider to ContextProvider (non-deprecated) * fix: Message(text=) -> Message(contents=[]) for API compatibility * Inline _constants.py into consuming modules Remove _constants.py and move constants to where they are used: - SUPPORTED_MEDIA_TYPES, MIME_ALIASES → _detection.py - MEDIA_TYPE_ANALYZER_MAP, DEFAULT_ANALYZER → _context_provider.py Addresses review feedback to reduce file count. * Mark package as alpha per package management skill - Version: 1.0.0b260401 → 1.0.0a260401 - Classifier: Development Status 4 - Beta → 3 - Alpha - Add to PACKAGE_STATUS.md as alpha Follows the alpha package checklist from python-package-management skill. * Replace extend_instructions with extend_messages for status notifications Status/error/result notifications now use extend_messages (conversation context) instead of extend_instructions (system prompt). This avoids system prompt bloat and keeps behavioral directives separate from event notifications. - 11 extend_instructions calls → extend_messages (role='user') - 1 extend_instructions retained: tool usage guidelines (behavioral) - 6 test assertions updated to check context_messages All 84 unit tests + 5 live integration tests pass. * Fix lint: E402 import order, ISC004 implicit string concatenation - Move constants after all imports to fix E402 - Wrap multi-line strings in parentheses inside contents=[] to fix ISC004 * Fix lint: remove unused json import in invoice sample * Fix CI: apply ruff format + fix E501 line length after reformatting ruff format expands Message() calls to multi-line, pushing string indentation deeper. Break long strings to fit within 120 char limit after formatting. Also removes unused json import in sample. * Address review feedback: keyword-only args, accept pre-built client, remove wrappers - All __init__ args now keyword-only (matches FoundryChatClient pattern) - New 'client' param accepts pre-built ContentUnderstandingClient - core dep bound: >=1.0.0rc5 → >=1.0.0,<2 - Self import moved after local imports - Removed 9 static method wrappers; callsites use module functions directly - Tests updated to import derive_doc_key and format_result directly * fix: remove duplicate ContentUnderstandingClient instantiation The client was being created twice — once inside the if/else block and again unconditionally after it. The second instantiation overwrote the pre-built client path and failed type checking when credential was None. * rename: azure-ai-contentunderstanding → azure-contentunderstanding Package: agent-framework-azure-ai-contentunderstanding → agent-framework-azure-contentunderstanding Module: agent_framework_azure_ai_contentunderstanding → agent_framework_azure_contentunderstanding Directory: packages/azure-ai-contentunderstanding → packages/azure-contentunderstanding Per agreement with PM and MAF team to drop 'AI' from the package name. * feat: add ContentUnderstanding re-export to agent_framework.foundry namespace Enables: from agent_framework.foundry import ContentUnderstandingContextProvider Exports: ContentUnderstandingContextProvider, FileSearchConfig, FileSearchBackend, AnalysisSection, DocumentStatus Updates all samples and README to use the foundry namespace import. * fix: add missing copyright headers to standalone sample scripts * chore: remove .vscode/settings.json and add to .gitignore * refactor: reuse FoundryChatClient.client for vector store ops in file_search sample Address review feedback from TaoChenOSU: - 05_large_doc_file_search.py: use client.client instead of manually constructing AsyncAzureOpenAI; remove openai dependency - azure_openai_backend/agent.py: import reorder only (AIProjectClient kept — required for sync vector store creation in DevUI) * fix: skip closing client when caller passes pre-built client When a ContentUnderstandingClient is passed via client=, the caller owns its lifecycle. Added _owns_client flag so close() only closes the client when we created it internally. --------- Co-authored-by: yungshinlin <yungshin@msn.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-06-16 21:04:09 +08:00 · 2026-04-28 20:55:59 +00:00
parent 3a463b8bf6
commit 1e1eda65ce
42 changed files with 6878 additions and 0 deletions
@@ -0,0 +1,119 @@
+# Copyright (c) Microsoft. All rights reserved.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "agent-framework-azure-contentunderstanding",
+#     "agent-framework-foundry",
+#     "azure-identity",
+# ]
+# ///
+# Run with: uv run packages/azure-contentunderstanding/samples/01-get-started/01_document_qa.py
+
+
+import asyncio
+import os
+from pathlib import Path
+
+from agent_framework import Agent, Content, Message
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+from agent_framework.foundry import ContentUnderstandingContextProvider
+
+load_dotenv()
+
+"""
+Document Q&A — PDF upload with CU-powered extraction
+
+This sample demonstrates the simplest CU integration: upload a PDF and
+ask questions about it. Azure Content Understanding extracts structured
+markdown with table preservation — superior to LLM-only vision for
+scanned PDFs, handwritten content, and complex layouts.
+
+Environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL             — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+"""
+
+# Path to a sample PDF — uses the shared sample asset if available,
+# otherwise falls back to a public URL
+SAMPLE_PDF_PATH = Path(__file__).resolve().parents[1] / "shared" / "sample_assets" / "invoice.pdf"
+
+
+async def main() -> None:
+    credential = AzureCliCredential()
+
+    # Set up Azure Content Understanding context provider
+    cu = ContentUnderstandingContextProvider(
+        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+        credential=credential,
+        analyzer_id="prebuilt-documentSearch",  # RAG-optimized document analyzer
+        max_wait=None,  # wait until CU analysis finishes (no background deferral)
+    )
+
+    # Set up the LLM client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+
+    # Create agent with CU context provider.
+    # The provider extracts document content via CU and injects it into the
+    # LLM context so the agent can answer questions about the document.
+    async with cu:
+        agent = Agent(
+            client=client,
+            name="DocumentQA",
+            instructions=(
+                "You are a helpful document analyst. Use the analyzed document "
+                "content and extracted fields to answer questions precisely."
+            ),
+            context_providers=[cu],
+        )
+
+        # --- Turn 1: Upload PDF and ask a question ---
+        # 4. Upload PDF and ask questions
+        # The CU provider extracts markdown + fields from the PDF and injects
+        # the full content into context so the agent can answer precisely.
+        print("--- Upload PDF and ask questions ---")
+
+        pdf_bytes = SAMPLE_PDF_PATH.read_bytes()
+
+        response = await agent.run(
+            Message(
+                role="user",
+                contents=[
+                    Content.from_text(
+                        "What is this document about? Who is the vendor, and what is the total amount due?"
+                    ),
+                    Content.from_data(
+                        pdf_bytes,
+                        "application/pdf",
+                        # Always provide filename — used as the document key
+                        additional_properties={"filename": SAMPLE_PDF_PATH.name},
+                    ),
+                ],
+            )
+        )
+        usage = response.usage_details or {}
+        print(f"Agent: {response}")
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]\n")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+"""
+Sample output:
+
+--- Upload PDF and ask questions ---
+Agent: This document is an **invoice** for services and fees billed to
+  **MICROSOFT CORPORATION** (Invoice **INV-100**), including line items
+  (e.g., Consulting Services, Document Fee, Printing Fee) and a billing summary.
+  - **Vendor:** **CONTOSO LTD.**
+  - **Total amount due:** **$610.00**
+  [Input tokens: 988]
+"""
@@ -0,0 +1,145 @@
+# Copyright (c) Microsoft. All rights reserved.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "agent-framework-azure-contentunderstanding",
+#     "agent-framework-foundry",
+#     "azure-identity",
+# ]
+# ///
+# Run with: uv run packages/azure-contentunderstanding/samples/01-get-started/02_multi_turn_session.py
+
+
+import asyncio
+import os
+from pathlib import Path
+
+from agent_framework import Agent, AgentSession, Content, Message
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+from agent_framework.foundry import ContentUnderstandingContextProvider
+
+load_dotenv()
+
+"""
+Multi-Turn Session — Cached results across turns
+
+This sample demonstrates multi-turn document Q&A using an AgentSession.
+The session persists CU analysis results and conversation history across
+turns so the agent can answer follow-up questions about previously
+uploaded documents without re-analyzing them.
+
+Key concepts:
+  - AgentSession keeps CU state and conversation history across agent.run() calls
+  - Turn 1: CU analyzes the PDF and injects full content into context
+  - Turn 2: Unrelated question — agent answers from general knowledge
+  - Turn 3: Detailed question — agent uses document content from conversation
+    history (injected in Turn 1) to answer precisely
+
+Environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL             — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+"""
+
+SAMPLE_PDF_PATH = Path(__file__).resolve().parents[1] / "shared" / "sample_assets" / "invoice.pdf"
+
+
+async def main() -> None:
+    # 1. Set up credentials and CU context provider
+    credential = AzureCliCredential()
+
+    cu = ContentUnderstandingContextProvider(
+        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+        credential=credential,
+        analyzer_id="prebuilt-documentSearch",
+        max_wait=None,  # wait until CU analysis finishes (no background deferral)
+    )
+
+    # 2. Set up the LLM client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+
+    # 3. Create agent and persistent session
+    async with cu:
+        agent = Agent(
+            client=client,
+            name="DocumentQA",
+            instructions=(
+                "You are a helpful document analyst. Use the analyzed document "
+                "content and extracted fields to answer questions precisely."
+            ),
+            context_providers=[cu],
+        )
+
+        # Create a persistent session — this keeps CU state across turns
+        session = AgentSession()
+
+        # 4. Turn 1: Upload PDF
+        # CU analyzes the PDF and injects full content into context.
+        print("--- Turn 1: Upload PDF ---")
+        pdf_bytes = SAMPLE_PDF_PATH.read_bytes()
+        response = await agent.run(
+            Message(
+                role="user",
+                contents=[
+                    Content.from_text("What is this document about?"),
+                    Content.from_data(
+                        pdf_bytes,
+                        "application/pdf",
+                        additional_properties={"filename": SAMPLE_PDF_PATH.name},
+                    ),
+                ],
+            ),
+            session=session,  # <-- persist state across turns
+        )
+        usage = response.usage_details or {}
+        print(f"Agent: {response}")
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]\n")
+
+        # 5. Turn 2: Unrelated question
+        # No document needed — agent answers from general knowledge.
+        print("--- Turn 2: Unrelated question ---")
+        response = await agent.run("What is the capital of France?", session=session)
+        usage = response.usage_details or {}
+        print(f"Agent: {response}")
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]\n")
+
+        # 6. Turn 3: Detailed follow-up
+        # The agent answers from the full document content that was injected
+        # into conversation history in Turn 1. No re-analysis or tool call needed.
+        print("--- Turn 3: Detailed follow-up ---")
+        response = await agent.run(
+            "What is the shipping address on the invoice?",
+            session=session,
+        )
+        usage = response.usage_details or {}
+        print(f"Agent: {response}")
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]\n")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+"""
+Sample output:
+
+--- Turn 1: Upload PDF ---
+Agent: This document is an **invoice** from **CONTOSO LTD.** to **MICROSOFT
+  CORPORATION**. Amount Due: $610.00. Invoice INV-100, dated 11/15/2019.
+  [Input tokens: 975]
+
+--- Turn 2: Unrelated question ---
+Agent: Paris.
+  [Input tokens: 1134]
+
+--- Turn 3: Detailed follow-up ---
+Agent: Shipping address (SHIP TO): Microsoft Delivery, 123 Ship St,
+  Redmond WA, 98052.
+  [Input tokens: 1155]
+"""
@@ -0,0 +1,188 @@
+# Copyright (c) Microsoft. All rights reserved.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "agent-framework-azure-contentunderstanding",
+#     "agent-framework-foundry",
+#     "azure-identity",
+# ]
+# ///
+# Run with: uv run packages/azure-contentunderstanding/samples/01-get-started/03_multimodal_chat.py
+
+
+import asyncio
+import os
+import time
+from pathlib import Path
+
+from agent_framework import Agent, AgentSession, Content, Message
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+from agent_framework.foundry import ContentUnderstandingContextProvider
+
+load_dotenv()
+
+"""
+Multi-Modal Chat — PDF, audio, and video in a single turn
+
+This sample demonstrates CU's multi-modal capability: upload a PDF invoice,
+an audio call recording, and a video file all at once. The provider analyzes
+all three in parallel using the right CU analyzer for each media type.
+
+The provider auto-detects the media type and selects the right CU analyzer:
+  - PDF/images  → prebuilt-documentSearch
+  - Audio       → prebuilt-audioSearch
+  - Video       → prebuilt-videoSearch
+
+Environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL             — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+"""
+
+# Local PDF from package assets
+SAMPLE_PDF = Path(__file__).resolve().parents[1] / "shared" / "sample_assets" / "invoice.pdf"
+
+# Public audio/video from Azure CU samples repo (raw GitHub URLs)
+_CU_ASSETS = "https://raw.githubusercontent.com/Azure-Samples/azure-ai-content-understanding-assets/main"
+AUDIO_URL = f"{_CU_ASSETS}/audio/callCenterRecording.mp3"
+VIDEO_URL = f"{_CU_ASSETS}/videos/sdk_samples/FlightSimulator.mp4"
+
+
+async def main() -> None:
+    # 1. Set up credentials and CU context provider
+    credential = AzureCliCredential()
+
+    # No analyzer_id specified — the provider auto-detects from media type:
+    #   PDF/images → prebuilt-documentSearch
+    #   Audio      → prebuilt-audioSearch
+    #   Video      → prebuilt-videoSearch
+    cu = ContentUnderstandingContextProvider(
+        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+        credential=credential,
+        max_wait=None,  # wait until each analysis finishes
+    )
+
+    # 2. Set up the LLM client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+
+    # 3. Create agent and session
+    async with cu:
+        agent = Agent(
+            client=client,
+            name="MultiModalAgent",
+            instructions=(
+                "You are a helpful assistant that can analyze documents, audio, "
+                "and video files. Answer questions using the extracted content."
+            ),
+            context_providers=[cu],
+        )
+
+        session = AgentSession()
+
+        # --- Turn 1: Upload all 3 modalities at once ---
+        # The provider analyzes all files in parallel using the appropriate
+        # CU analyzer for each media type. All results are injected into
+        # the same context so the agent can answer about all of them.
+        turn1_prompt = (
+            "I'm uploading three files: an invoice PDF, a call center "
+            "audio recording, and a flight simulator video. "
+            "Give a brief summary of each file."
+        )
+        print("--- Turn 1: Upload PDF + audio + video (parallel analysis) ---")
+        print("  (CU analysis may take a few minutes for these audio/video files...)")
+        print(f"User: {turn1_prompt}")
+        t0 = time.perf_counter()
+        response = await agent.run(
+            Message(
+                role="user",
+                contents=[
+                    Content.from_text(turn1_prompt),
+                    Content.from_data(
+                        SAMPLE_PDF.read_bytes(),
+                        "application/pdf",
+                        additional_properties={"filename": "invoice.pdf"},
+                    ),
+                    Content.from_uri(
+                        AUDIO_URL,
+                        media_type="audio/mp3",
+                        additional_properties={"filename": "callCenterRecording.mp3"},
+                    ),
+                    Content.from_uri(
+                        VIDEO_URL,
+                        media_type="video/mp4",
+                        additional_properties={"filename": "FlightSimulator.mp4"},
+                    ),
+                ],
+            ),
+            session=session,
+        )
+        elapsed = time.perf_counter() - t0
+        usage = response.usage_details or {}
+        print(f"  [Analyzed in {elapsed:.1f}s | Input tokens: {usage.get('input_token_count', 'N/A')}]")
+        print(f"Agent: {response}\n")
+
+        # --- Turn 2: Detail question about the PDF ---
+        turn2_prompt = "What are the line items and their amounts on the invoice?"
+        print("--- Turn 2: PDF detail ---")
+        print(f"User: {turn2_prompt}")
+        response = await agent.run(turn2_prompt, session=session)
+        usage = response.usage_details or {}
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]")
+        print(f"Agent: {response}\n")
+
+        # --- Turn 3: Detail question about the audio ---
+        turn3_prompt = "What was the customer's issue in the call recording?"
+        print("--- Turn 3: Audio detail ---")
+        print(f"User: {turn3_prompt}")
+        response = await agent.run(turn3_prompt, session=session)
+        usage = response.usage_details or {}
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]")
+        print(f"Agent: {response}\n")
+
+        # --- Turn 4: Detail question about the video ---
+        turn4_prompt = "What key scenes or actions are shown in the flight simulator video?"
+        print("--- Turn 4: Video detail ---")
+        print(f"User: {turn4_prompt}")
+        response = await agent.run(turn4_prompt, session=session)
+        usage = response.usage_details or {}
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]")
+        print(f"Agent: {response}\n")
+
+        # --- Turn 5: Cross-document question ---
+        turn5_prompt = (
+            "Across all three files, which one contains financial data, "
+            "which one involves a customer interaction, and which one is "
+            "a visual demonstration?"
+        )
+        print("--- Turn 5: Cross-document question ---")
+        print(f"User: {turn5_prompt}")
+        response = await agent.run(turn5_prompt, session=session)
+        usage = response.usage_details or {}
+        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]")
+        print(f"Agent: {response}\n")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+"""
+Sample output:
+
+--- Turn 1: Upload PDF + audio + video (parallel analysis) ---
+User: I'm uploading three files...
+  (CU analysis may take 1-2 minutes for audio/video files...)
+  [Analyzed in ~94s | Input tokens: ~2939]
+Agent: ### invoice.pdf: An invoice from CONTOSO LTD. to MICROSOFT CORPORATION...
+       ### callCenterRecording.mp3: A customer service call about point balance...
+       ### FlightSimulator.mp4: A clip discussing neural text-to-speech...
+
+--- Turn 2-5: Detail and cross-document questions ---
+(Agent answers from conversation history without re-analysis)
+"""
@@ -0,0 +1,195 @@
+# Copyright (c) Microsoft. All rights reserved.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "agent-framework-azure-contentunderstanding",
+#     "agent-framework-foundry",
+#     "azure-identity",
+#     "pydantic",
+# ]
+# ///
+# Run with: uv run packages/azure-contentunderstanding/samples/01-get-started/04_invoice_processing.py
+
+
+import asyncio
+import os
+from pathlib import Path
+
+from agent_framework import Agent, AgentSession, Content, Message
+from agent_framework.foundry import FoundryChatClient
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+from pydantic import BaseModel, Field
+
+from agent_framework.foundry import ContentUnderstandingContextProvider
+
+load_dotenv()
+
+"""
+Invoice Processing — Structured output with prebuilt-invoice analyzer
+
+This sample demonstrates CU's structured field extraction combined with
+LLM structured output (Pydantic model). The prebuilt-invoice analyzer extracts
+typed fields (VendorName, InvoiceTotal, DueDate, LineItems, etc.) with
+confidence scores. We use output_sections=["fields"] only (no markdown needed)
+since we want the LLM to produce a structured JSON response from the extracted
+fields, not summarize document text.
+
+Environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL             — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+"""
+
+SAMPLE_PDF_PATH = Path(__file__).resolve().parents[1] / "shared" / "sample_assets" / "invoice.pdf"
+
+
+# Structured output model — the LLM will return JSON matching this schema
+# Structured output models — the LLM returns JSON matching this schema.
+#
+# Note: the prebuilt-invoice analyzer extracts an extensive set of fields
+# (VendorName, BillingAddress, ShippingAddress, TaxDetails, PONumber, etc.).
+# This sample defines a simplified schema to extract only the fields of
+# interest to the caller. The LLM maps the full CU field output to this
+# subset automatically.
+# Learn more about prebuilt analyzers: https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/prebuilt-analyzers
+
+
+class LineItem(BaseModel):
+    description: str
+    quantity: float | None = None
+    unit_price: float | None = None
+    amount: float | None = None
+
+
+class LowConfidenceField(BaseModel):
+    field_name: str
+    confidence: float
+
+
+class InvoiceResult(BaseModel):
+    vendor_name: str
+    total_amount: float | None = None
+    currency: str = "USD"
+    due_date: str | None = None
+    line_items: list[LineItem] = Field(default_factory=list)
+    low_confidence_fields: list[LowConfidenceField] = Field(
+        default_factory=list,
+        description="Fields with confidence < 0.8, including their confidence score",
+    )
+
+
+async def main() -> None:
+    # 1. Set up credentials and CU context provider
+    credential = AzureCliCredential()
+
+    # Default analyzer is prebuilt-documentSearch (RAG-optimized).
+    # Per-file override via additional_properties["analyzer_id"] lets us
+    # use prebuilt-invoice for structured field extraction on specific files.
+    #
+    # Only request "fields" (not "markdown") — we want the extracted typed
+    # fields for structured output, not the raw document text.
+    cu = ContentUnderstandingContextProvider(
+        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+        credential=credential,
+        analyzer_id="prebuilt-documentSearch",  # default for all files
+        max_wait=None,  # wait until CU analysis finishes
+        output_sections=["fields"],  # fields only — structured output doesn't need markdown
+    )
+
+    # 2. Set up the LLM client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+
+    # 3. Create agent and session
+    async with cu:
+        agent = Agent(
+            client=client,
+            name="InvoiceProcessor",
+            instructions=(
+                "You are an invoice processing assistant. Extract invoice data from "
+                "the provided CU fields (JSON with confidence scores). Return structured "
+                "output matching the requested schema. Flag fields with confidence < 0.8 "
+                "in the low_confidence_fields list."
+            ),
+            context_providers=[cu],
+        )
+
+        session = AgentSession()
+
+        # 4. Upload an invoice PDF — uses structured output (Pydantic model)
+        print("--- Upload Invoice (Structured Output) ---")
+
+        pdf_bytes = SAMPLE_PDF_PATH.read_bytes()
+
+        response = await agent.run(
+            Message(
+                role="user",
+                contents=[
+                    Content.from_text(
+                        "Process this invoice. Extract the vendor name, total amount, due date, and all line items."
+                    ),
+                    Content.from_data(
+                        pdf_bytes,
+                        "application/pdf",
+                        # Per-file analyzer override: use prebuilt-invoice for
+                        # structured field extraction (VendorName, InvoiceTotal, etc.)
+                        # instead of the provider default (prebuilt-documentSearch).
+                        additional_properties={
+                            "filename": SAMPLE_PDF_PATH.name,
+                            "analyzer_id": "prebuilt-invoice",
+                        },
+                    ),
+                ],
+            ),
+            session=session,
+            options={"response_format": InvoiceResult},
+        )
+
+        # Parse the structured output from JSON text
+        try:
+            invoice = InvoiceResult.model_validate_json(response.text)
+            print(f"Vendor: {invoice.vendor_name}")
+            print(f"Total: {invoice.currency} {invoice.total_amount}")
+            print(f"Due date: {invoice.due_date}")
+            print(f"Line items ({len(invoice.line_items)}):")
+            for item in invoice.line_items:
+                print(f"  - {item.description}: {item.amount}")
+            if invoice.low_confidence_fields:
+                print("⚠ Low confidence fields:")
+                for f in invoice.low_confidence_fields:
+                    print(f"  - {f.field_name}: {f.confidence:.3f}")
+        except Exception:
+            print(f"Agent (raw): {response.text}\n")
+
+        # 5. Follow-up: free-text question about the invoice
+        print("\n--- Follow-up (Free Text) ---")
+        response = await agent.run(
+            "What is the payment term? Are there any fields with low confidence?",
+            session=session,
+        )
+        print(f"Agent: {response}\n")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+"""
+Sample output:
+
+--- Upload Invoice (Structured Output) ---
+Vendor: CONTOSO LTD.
+Total: USD 110.0
+Due date: 2019-12-15
+Line items (3):
+  - Consulting Services: 60.0
+  - Document Fee: 30.0
+  - Printing Fee: 10.0
+⚠ Low confidence: VendorName, CustomerName
+
+--- Follow-up (Free Text) ---
+Agent: The payment terms are not explicitly stated on the invoice...
+"""
@@ -0,0 +1,166 @@
+# Copyright (c) Microsoft. All rights reserved.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "agent-framework-azure-contentunderstanding",
+#     "agent-framework-foundry",
+#     "azure-identity",
+# ]
+# ///
+# Run with: uv run packages/azure-contentunderstanding/samples/01-get-started/05_large_doc_file_search.py
+
+
+import asyncio
+import os
+from pathlib import Path
+
+from agent_framework import Agent, AgentSession, Content, Message
+from agent_framework.foundry import (
+    ContentUnderstandingContextProvider,
+    FileSearchConfig,
+    FoundryChatClient,
+)
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+"""
+Large Document + file_search RAG — CU extraction + OpenAI vector store
+
+For large documents (100+ pages) or long audio/video, injecting the full
+CU-extracted content into the LLM context is impractical. This sample shows
+how to use the built-in file_search integration: CU extracts markdown and
+automatically uploads it to an OpenAI vector store for token-efficient RAG.
+
+When ``FileSearchConfig`` is provided, the provider:
+  1. Extracts markdown via CU (handles scanned PDFs, audio, video)
+  2. Uploads the extracted markdown to a vector store
+  3. Registers a ``file_search`` tool on the agent context
+  4. Cleans up the vector store on close
+
+Architecture:
+  Large PDF -> CU extracts markdown -> auto-upload to vector store -> file_search
+  Follow-up -> file_search retrieves top-k chunks -> LLM answers
+
+NOTE: Requires an async OpenAI client for vector store operations.
+
+This sample uses a single small invoice PDF for simplicity. In practice,
+you can upload multiple files in the same session (each is indexed
+separately in the vector store), and this pattern is most valuable for
+large documents (up to 300 pages), long audio recordings, or video files
+where full-context injection would exceed the LLM's context window.
+CU supports PDFs up to 300 pages / 200 MB, and audio files up to 300 MB
+— see the full service limits:
+https://learn.microsoft.com/azure/ai-services/content-understanding/service-limits#input-file-limits
+
+Environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL             — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+"""
+
+SAMPLE_PDF_PATH = Path(__file__).resolve().parents[1] / "shared" / "sample_assets" / "invoice.pdf"
+
+
+async def main() -> None:
+    # 1. Set up credentials and LLM client
+    credential = AzureCliCredential()
+
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+
+    # 2. Get the async OpenAI client from FoundryChatClient for vector store operations
+    openai_client = client.client
+
+    # 3. Create vector store and file_search tool
+    vector_store = await openai_client.vector_stores.create(
+        name="cu_large_doc_demo",
+        expires_after={"anchor": "last_active_at", "days": 1},
+    )
+    file_search_tool = client.get_file_search_tool(vector_store_ids=[vector_store.id])
+
+    # 4. Configure CU provider with file_search integration
+    # When file_search is set, CU-extracted markdown is automatically uploaded
+    # to the vector store and the file_search tool is registered on the context.
+    cu = ContentUnderstandingContextProvider(
+        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+        credential=credential,
+        analyzer_id="prebuilt-documentSearch",
+        max_wait=None,  # wait until CU analysis + vector store upload finishes
+        file_search=FileSearchConfig.from_foundry(
+            openai_client,
+            vector_store_id=vector_store.id,
+            file_search_tool=file_search_tool,
+        ),
+    )
+
+    pdf_bytes = SAMPLE_PDF_PATH.read_bytes()
+
+    # The provider handles everything: CU extraction + vector store upload + file_search tool
+    async with cu:
+        agent = Agent(
+            client=client,
+            name="LargeDocAgent",
+            instructions=(
+                "You are a document analyst. Use the file_search tool to find "
+                "relevant sections from the document and answer precisely. "
+                "Cite specific sections when answering."
+            ),
+            context_providers=[cu],
+        )
+
+        session = AgentSession()
+
+        # Turn 1: Upload — CU extracts and uploads to vector store automatically
+        print("--- Turn 1: Upload document ---")
+        response = await agent.run(
+            Message(
+                role="user",
+                contents=[
+                    Content.from_text("What are the key points in this document?"),
+                    Content.from_data(
+                        pdf_bytes,
+                        "application/pdf",
+                        additional_properties={"filename": SAMPLE_PDF_PATH.name},
+                    ),
+                ],
+            ),
+            session=session,
+        )
+        print(f"Agent: {response}\n")
+
+        # Turn 2: Follow-up — file_search retrieves relevant chunks (token efficient)
+        print("--- Turn 2: Follow-up (RAG) ---")
+        response = await agent.run(
+            "What numbers or financial metrics are mentioned?",
+            session=session,
+        )
+        print(f"Agent: {response}\n")
+
+    # Explicitly delete the vector store created for this sample
+    await openai_client.vector_stores.delete(vector_store.id)
+    print("Done. Vector store deleted.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+"""
+Sample output:
+
+--- Turn 1: Upload document ---
+Agent: An invoice from Contoso Ltd. to Microsoft Corporation (INV-100).
+  Line items: Consulting Services $60, Document Fee $30, Printing Fee $10.
+  Subtotal $100, Sales tax $10, Total $110, Previous balance $500, Amount due $610.
+
+--- Turn 2: Follow-up (RAG) ---
+Agent: Subtotal $100.00, Sales tax $10.00, Total $110.00,
+  Previous unpaid balance $500.00, Amount due $610.00.
+  Line items: 2 hours @ $30 = $60, 3 @ $10 = $30, 10 pages @ $1 = $10.
+
+Done. Vector store cleaned up automatically.
+"""
@@ -0,0 +1,33 @@
+# DevUI Multi-Modal Agent
+
+Interactive web UI for uploading and chatting with documents, images, audio, and video using Azure Content Understanding.
+
+## Setup
+
+1. Set environment variables (or create a `.env` file in `python/`):
+   ```bash
+   FOUNDRY_PROJECT_ENDPOINT=https://your-project.api.azureml.ms
+   AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=gpt-4.1
+   AZURE_CONTENTUNDERSTANDING_ENDPOINT=https://your-cu-resource.cognitiveservices.azure.com/
+   ```
+
+2. Log in with Azure CLI:
+   ```bash
+   az login
+   ```
+
+3. Run with DevUI:
+   ```bash
+   uv run poe devui --agent packages/azure-contentunderstanding/samples/devui_multimodal_agent
+   ```
+
+4. Open the DevUI URL in your browser and start uploading files.
+
+## What You Can Do
+
+- **Upload PDFs** — including scanned/image-based PDFs that LLM vision struggles with
+- **Upload images** — handwritten notes, infographics, charts
+- **Upload audio** — meeting recordings, call center calls (transcription with speaker ID)
+- **Upload video** — product demos, training videos (frame extraction + transcription)
+- **Ask questions** across all uploaded documents
+- **Check status** — "which documents are ready?" uses the auto-registered `list_documents()` tool
@@ -0,0 +1,6 @@
+# Copyright (c) Microsoft. All rights reserved.
+"""DevUI Multi-Modal Agent with Azure Content Understanding."""
+
+from .agent import agent
+
+__all__ = ["agent"]
@@ -0,0 +1,68 @@
+# Copyright (c) Microsoft. All rights reserved.
+"""DevUI Multi-Modal Agent — file upload + CU-powered analysis.
+
+This agent uses Azure Content Understanding to analyze uploaded files
+(PDFs, scanned documents, handwritten images, audio recordings, video)
+and answer questions about them through the DevUI web interface.
+
+Unlike the standard azure_responses_agent which sends files directly to the LLM,
+this agent uses CU for structured extraction — superior for scanned PDFs,
+handwritten content, audio transcription, and video analysis.
+
+Required environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                 — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL                            — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+
+Run with DevUI:
+  uv run poe devui --agent packages/azure-contentunderstanding/samples/devui_multimodal_agent
+"""
+
+import os
+
+from agent_framework import Agent
+from agent_framework.foundry import FoundryChatClient
+from azure.core.credentials import AzureKeyCredential
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+from agent_framework.foundry import ContentUnderstandingContextProvider
+
+load_dotenv()
+
+# --- Auth ---
+_credential = AzureCliCredential()
+_cu_api_key = os.environ.get("AZURE_CONTENTUNDERSTANDING_API_KEY")
+_cu_credential = AzureKeyCredential(_cu_api_key) if _cu_api_key else _credential
+
+cu = ContentUnderstandingContextProvider(
+    endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+    credential=_cu_credential,
+    # max_wait controls how long before_run() waits for CU analysis before
+    # deferring to background.  For interactive DevUI use, a short timeout
+    # (e.g. 5s) keeps the chat responsive — the agent tells the user the
+    # file is still being analyzed and resolves it on the next turn.
+    # Use max_wait=None to always wait for analysis to complete.
+    max_wait=5.0,
+)
+
+client = FoundryChatClient(
+    project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+    model=os.environ["FOUNDRY_MODEL"],
+    credential=_credential,
+)
+
+agent = Agent(
+    client=client,
+    name="MultiModalDocAgent",
+    instructions=(
+        "You are a helpful document analysis assistant. "
+        "When a user uploads files, they are automatically analyzed using Azure Content Understanding. "
+        "Use list_documents() to check which documents are ready, pending, or failed "
+        "and to see which files are available for answering questions. "
+        "Tell the user if any documents are still being analyzed. "
+        "You can process PDFs, scanned documents, handwritten images, audio recordings, and video files. "
+        "When answering, cite specific content from the documents."
+    ),
+    context_providers=[cu],
+)
@@ -0,0 +1,51 @@
+# DevUI File Search Agent
+
+Interactive web UI for uploading and chatting with documents, images, audio, and video using Azure Content Understanding + OpenAI file_search RAG.
+
+## How It Works
+
+1. **Upload** any supported file (PDF, image, audio, video) via the DevUI chat
+2. **CU analyzes** the file — auto-selects the right analyzer per media type
+3. **Markdown extracted** by CU is uploaded to an OpenAI vector store
+4. **file_search** tool is registered — LLM retrieves top-k relevant chunks
+5. **Ask questions** across all uploaded documents with token-efficient RAG
+
+## Setup
+
+1. Set environment variables (or create a `.env` file in `python/`):
+   ```bash
+   FOUNDRY_PROJECT_ENDPOINT=https://your-project.services.ai.azure.com/
+   AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME=gpt-4.1
+   AZURE_CONTENTUNDERSTANDING_ENDPOINT=https://your-cu-resource.services.ai.azure.com/
+   ```
+
+2. Log in with Azure CLI:
+   ```bash
+   az login
+   ```
+
+3. Run with DevUI:
+   ```bash
+   devui packages/azure-contentunderstanding/samples/devui_azure_openai_file_search_agent
+   ```
+
+4. Open the DevUI URL in your browser and start uploading files.
+
+## Supported File Types
+
+| Type | Formats | CU Analyzer (auto-detected) |
+|------|---------|----------------------------|
+| Documents | PDF, DOCX, XLSX, PPTX, HTML, TXT, Markdown | `prebuilt-documentSearch` |
+| Images | JPEG, PNG, TIFF, BMP | `prebuilt-documentSearch` |
+| Audio | WAV, MP3, FLAC, OGG, M4A | `prebuilt-audioSearch` |
+| Video | MP4, MOV, AVI, WebM | `prebuilt-videoSearch` |
+
+## vs. devui_multimodal_agent
+
+| Feature | multimodal_agent | file_search_agent |
+|---------|-----------------|-------------------|
+| CU extraction | ✅ Full content injected | ✅ Content indexed in vector store |
+| RAG | ❌ | ✅ file_search retrieves top-k chunks |
+| Large docs (100+ pages) | ⚠️ May exceed context window | ✅ Token-efficient |
+| Multiple large files | ⚠️ Context overflow risk | ✅ All indexed, searchable |
+| Best for | Small docs, quick inspection | Large docs, multi-file Q&A |
@@ -0,0 +1,6 @@
+# Copyright (c) Microsoft. All rights reserved.
+"""DevUI Multi-Modal Agent with CU + file_search RAG."""
+
+from .agent import agent
+
+__all__ = ["agent"]
@@ -0,0 +1,105 @@
+# Copyright (c) Microsoft. All rights reserved.
+"""DevUI Multi-Modal Agent — CU extraction + file_search RAG.
+
+This agent combines Azure Content Understanding with OpenAI file_search
+for token-efficient RAG over large or multi-modal documents.
+
+Upload flow:
+  1. CU extracts high-quality markdown (handles scanned PDFs, audio, video)
+  2. Extracted markdown is auto-uploaded to an OpenAI vector store
+  3. file_search tool is registered so the LLM retrieves top-k chunks
+  4. Vector store is configured to auto-expire after inactivity
+
+This is ideal for large documents (100+ pages), long audio recordings,
+or multiple files in the same conversation where full-context injection
+would exceed the LLM's context window.
+
+Analyzer auto-detection:
+  When no analyzer_id is specified, the provider auto-selects the
+  appropriate CU analyzer based on media type:
+    - Documents/images → prebuilt-documentSearch
+    - Audio            → prebuilt-audioSearch
+    - Video            → prebuilt-videoSearch
+
+Required environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                 — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL                            — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+
+Run with DevUI:
+  devui packages/azure-contentunderstanding/samples/devui_azure_openai_file_search_agent
+"""
+
+import os
+
+from agent_framework import Agent
+from agent_framework.foundry import (
+    ContentUnderstandingContextProvider,
+    FileSearchConfig,
+    FoundryChatClient,
+)
+from azure.ai.projects import AIProjectClient
+from azure.core.credentials import AzureKeyCredential
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# --- Auth ---
+_credential = AzureCliCredential()
+_cu_api_key = os.environ.get("AZURE_CONTENTUNDERSTANDING_API_KEY")
+_cu_credential = AzureKeyCredential(_cu_api_key) if _cu_api_key else _credential
+
+_endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
+
+# --- LLM client + sync vector store setup ---
+# DevUI loads agent modules synchronously at startup while an event loop is already
+# running, so we cannot use async APIs here. A sync AIProjectClient is used for
+# one-time vector store creation; runtime file uploads use client.client (async).
+client = FoundryChatClient(
+    project_endpoint=_endpoint,
+    model=os.environ["FOUNDRY_MODEL"],
+    credential=_credential,
+)
+
+_sync_project = AIProjectClient(endpoint=_endpoint, credential=_credential)  # type: ignore[arg-type]
+_sync_openai = _sync_project.get_openai_client()
+_vector_store = _sync_openai.vector_stores.create(
+    name="devui_cu_file_search",
+    expires_after={"anchor": "last_active_at", "days": 1},
+)
+_sync_openai.close()
+
+_file_search_tool = client.get_file_search_tool(
+    vector_store_ids=[_vector_store.id],
+    max_num_results=3,  # limit chunks to reduce input token usage
+)
+
+# --- CU context provider with file_search ---
+# client.client is the async OpenAI client used for runtime file uploads.
+# No analyzer_id → auto-selects per media type (documents, audio, video)
+cu = ContentUnderstandingContextProvider(
+    endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+    credential=_cu_credential,
+    file_search=FileSearchConfig.from_foundry(
+        client.client,  # reuse the LLM client's internal AsyncAzureOpenAI for file uploads
+        vector_store_id=_vector_store.id,
+        file_search_tool=_file_search_tool,
+    ),
+)
+
+agent = Agent(
+    client=client,
+    name="FileSearchDocAgent",
+    instructions=(
+        "You are a helpful document analysis assistant with RAG capabilities. "
+        "When a user uploads files, they are automatically analyzed using Azure Content Understanding "
+        "and indexed in a vector store for efficient retrieval. "
+        "Analysis takes time (seconds for documents, longer for audio/video) — if a document "
+        "is still pending, let the user know and suggest they ask again shortly. "
+        "You can process PDFs, scanned documents, handwritten images, audio recordings, and video files. "
+        "Multiple files can be uploaded and queried in the same conversation. "
+        "When answering, cite specific content from the documents."
+    ),
+    context_providers=[cu],
+)
@@ -0,0 +1,34 @@
+# DevUI Foundry File Search Agent
+
+Interactive web UI for uploading and chatting with documents, images, audio, and video using Azure Content Understanding + Foundry file_search RAG.
+
+This is the **Foundry** variant. For the Azure OpenAI Responses API variant, see `devui_azure_openai_file_search_agent`.
+
+## How It Works
+
+1. **Upload** any supported file (PDF, image, audio, video) via the DevUI chat
+2. **CU analyzes** the file — auto-selects the right analyzer per media type
+3. **Markdown extracted** by CU is uploaded to a Foundry vector store
+4. **file_search** tool is registered — LLM retrieves top-k relevant chunks
+5. **Ask questions** across all uploaded documents with token-efficient RAG
+
+## Setup
+
+1. Set environment variables (or create a `.env` file in `python/`):
+   ```bash
+   FOUNDRY_PROJECT_ENDPOINT=https://your-project.services.ai.azure.com/
+   FOUNDRY_MODEL=gpt-4.1
+   AZURE_CONTENTUNDERSTANDING_ENDPOINT=https://your-cu-resource.services.ai.azure.com/
+   ```
+
+2. Log in with Azure CLI:
+   ```bash
+   az login
+   ```
+
+3. Run with DevUI:
+   ```bash
+   devui packages/azure-contentunderstanding/samples/devui_foundry_file_search_agent
+   ```
+
+4. Open the DevUI URL in your browser and start uploading files.
@@ -0,0 +1 @@
+# Copyright (c) Microsoft. All rights reserved.
@@ -0,0 +1,110 @@
+# Copyright (c) Microsoft. All rights reserved.
+"""DevUI Multi-Modal Agent — CU extraction + file_search RAG via Azure AI Foundry.
+
+This agent combines Azure Content Understanding with Foundry's file_search
+for token-efficient RAG over large or multi-modal documents.
+
+Upload flow:
+  1. CU extracts high-quality markdown (handles scanned PDFs, audio, video)
+  2. Extracted markdown is uploaded to a Foundry vector store
+  3. file_search tool is registered so the LLM retrieves top-k chunks
+  4. Uploaded files are cleaned up on server shutdown
+
+This sample uses ``FoundryChatClient`` and ``FoundryFileSearchBackend``.
+For the OpenAI Responses API variant, see ``devui_azure_openai_file_search_agent``.
+
+Analyzer auto-detection:
+  When no analyzer_id is specified, the provider auto-selects the
+  appropriate CU analyzer based on media type:
+    - Documents/images → prebuilt-documentSearch
+    - Audio            → prebuilt-audioSearch
+    - Video            → prebuilt-videoSearch
+
+Required environment variables:
+  FOUNDRY_PROJECT_ENDPOINT                 — Azure AI Foundry project endpoint
+  FOUNDRY_MODEL                            — Model deployment name (e.g. gpt-4.1)
+  AZURE_CONTENTUNDERSTANDING_ENDPOINT      — CU endpoint URL
+
+Run with DevUI:
+  devui packages/azure-contentunderstanding/samples/devui_foundry_file_search_agent
+"""
+
+import os
+
+from agent_framework import Agent
+from agent_framework.foundry import FoundryChatClient
+from azure.core.credentials import AzureKeyCredential
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+from openai import AzureOpenAI
+
+from agent_framework.foundry import (
+    ContentUnderstandingContextProvider,
+    FileSearchConfig,
+)
+
+load_dotenv()
+
+# --- Auth ---
+# AzureCliCredential for Foundry. CU API key optional if on a different resource.
+_credential = AzureCliCredential()
+_cu_api_key = os.environ.get("AZURE_CONTENTUNDERSTANDING_API_KEY")
+_cu_credential = AzureKeyCredential(_cu_api_key) if _cu_api_key else _credential
+
+# --- Foundry LLM client ---
+client = FoundryChatClient(
+    project_endpoint=os.environ.get("FOUNDRY_PROJECT_ENDPOINT", ""),
+    model=os.environ.get("FOUNDRY_MODEL", ""),
+    credential=_credential,
+)
+
+# --- Create vector store (sync client to avoid event loop conflicts in DevUI) ---
+_token = _credential.get_token("https://ai.azure.com/.default").token
+_sync_openai = AzureOpenAI(
+    azure_endpoint=os.environ.get("FOUNDRY_PROJECT_ENDPOINT", ""),
+    azure_ad_token=_token,
+    api_version="2025-04-01-preview",
+)
+_vector_store = _sync_openai.vector_stores.create(
+    name="devui_cu_foundry_file_search",
+    expires_after={"anchor": "last_active_at", "days": 1},
+)
+_sync_openai.close()
+
+_file_search_tool = client.get_file_search_tool(
+    vector_store_ids=[_vector_store.id],
+    max_num_results=3,  # limit chunks to reduce input token usage
+)
+
+# --- CU context provider with file_search ---
+# No analyzer_id → auto-selects per media type (documents, audio, video)
+cu = ContentUnderstandingContextProvider(
+    endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
+    credential=_cu_credential,
+    # max_wait is the combined budget for CU analysis + vector store upload.
+    # For file_search mode, 10s gives enough time for small documents to be
+    # analyzed and indexed in one turn.  Larger files (audio, video) will
+    # be deferred to background and resolved on the next turn.
+    max_wait=10.0,
+    file_search=FileSearchConfig.from_foundry(
+        client.client,
+        vector_store_id=_vector_store.id,
+        file_search_tool=_file_search_tool,
+    ),
+)
+
+agent = Agent(
+    client=client,
+    name="FoundryFileSearchDocAgent",
+    instructions=(
+        "You are a helpful document analysis assistant with RAG capabilities. "
+        "When a user uploads files, they are automatically analyzed using Azure Content Understanding "
+        "and indexed in a vector store for efficient retrieval. "
+        "Analysis takes time (seconds for documents, longer for audio/video) — if a document "
+        "is still pending, let the user know and suggest they ask again shortly. "
+        "You can process PDFs, scanned documents, handwritten images, audio recordings, and video files. "
+        "Multiple files can be uploaded and queried in the same conversation. "
+        "When answering, cite specific content from the documents."
+    ),
+    context_providers=[cu],
+)
@@ -0,0 +1,39 @@
+# Azure Content Understanding Samples
+
+These samples demonstrate how to use the `agent-framework-azure-contentunderstanding` package to add document, image, audio, and video understanding to your agents.
+
+## Prerequisites
+
+1. Azure CLI logged in: `az login`
+2. Environment variables set (or `.env` file in the `python/` directory):
+   ```
+   FOUNDRY_PROJECT_ENDPOINT=https://your-project.services.ai.azure.com
+   FOUNDRY_MODEL=gpt-4.1
+   AZURE_CONTENTUNDERSTANDING_ENDPOINT=https://your-cu-resource.cognitiveservices.azure.com/
+   ```
+
+## Samples
+
+### 01-get-started — Script samples (easy → advanced)
+
+| # | Sample | Description | Run |
+|---|--------|-------------|-----|
+| 01 | [Document Q&A](01-get-started/01_document_qa.py) | Upload a PDF, ask questions with CU-powered extraction | `uv run samples/01-get-started/01_document_qa.py` |
+| 02 | [Multi-Turn Session](01-get-started/02_multi_turn_session.py) | AgentSession persistence across turns | `uv run samples/01-get-started/02_multi_turn_session.py` |
+| 03 | [Multi-Modal Chat](01-get-started/03_multimodal_chat.py) | PDF + audio + video parallel analysis | `uv run samples/01-get-started/03_multimodal_chat.py` |
+| 04 | [Invoice Processing](01-get-started/04_invoice_processing.py) | Structured field extraction with prebuilt-invoice | `uv run samples/01-get-started/04_invoice_processing.py` |
+| 05 | [Large Doc + file_search](01-get-started/05_large_doc_file_search.py) | CU extraction + OpenAI vector store RAG | `uv run samples/01-get-started/05_large_doc_file_search.py` |
+
+### 02-devui — Interactive web UI samples
+
+| # | Sample | Description | Run |
+|---|--------|-------------|-----|
+| 01 | [Multi-Modal Agent](02-devui/01-multimodal_agent/) | Web UI for file upload + CU-powered chat | `devui samples/02-devui/01-multimodal_agent` |
+| 02a | [file_search (Azure OpenAI backend)](02-devui/02-file_search_agent/azure_openai_backend/) | DevUI with CU + Azure OpenAI vector store | `devui samples/02-devui/02-file_search_agent/azure_openai_backend` |
+| 02b | [file_search (Foundry backend)](02-devui/02-file_search_agent/foundry_backend/) | DevUI with CU + Foundry vector store | `devui samples/02-devui/02-file_search_agent/foundry_backend` |
+
+## Install (preview)
+
+```bash
+pip install --pre agent-framework-azure-contentunderstanding
+```
				`@@ -0,0 +1 @@`
				`# Copyright (c) Microsoft. All rights reserved.`