mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
[Python] Add agent-framework-azure-ai-contentunderstanding package (#4829)
* feat: add agent-framework-azure-contentunderstanding package
Add Azure Content Understanding integration as a context provider for the
Agent Framework. The package automatically analyzes file attachments
(documents, images, audio, video) using Azure CU and injects structured
results (markdown, fields) into the LLM context.
Key features:
- Multi-document session state with status tracking (pending/ready/failed)
- Configurable timeout with async background fallback for large files
- Output filtering via AnalysisSection enum
- Auto-registered list_documents() and get_analyzed_document() tools
- Supports all CU modalities: documents, images, audio, video
- Content limits enforcement (pages, file size, duration)
- Binary stripping of supported files from input messages
Public API:
- ContentUnderstandingContextProvider (main class)
- AnalysisSection (output section selector enum)
- ContentLimits (configurable limits dataclass)
Tests: 46 unit tests, 91% coverage, all linting and type checks pass.
* fix: update CU fixtures with real API data, fix test assertions
- Replace synthetic fixtures with real CU API responses (sanitized)
- Update test assertions to match real data (Contoso vs CONTOSO,
TotalAmount vs InvoiceTotal, field values from real analysis)
- Add --pre install note in README (preview package)
- Document unenforced ContentLimits fields (max_pages, duration)
* chore: add connector .gitignore, update uv.lock
* refactor: rename to azure-ai-contentunderstanding, fix CI issues
Align naming with Azure SDK convention and AF pattern:
- Directory: azure-contentunderstanding -> azure-ai-contentunderstanding
- PyPI: agent-framework-azure-contentunderstanding -> agent-framework-azure-ai-contentunderstanding
- Module: agent_framework_azure_contentunderstanding -> agent_framework_azure_ai_contentunderstanding
CI fixes:
- Inline conftest helpers to avoid cross-package import collision in xdist
- Remove PyPI badge and dead API reference link from README (package not published yet)
* feat: add samples (document_qa, invoice_processing, multimodal_chat)
- document_qa.py: Single PDF upload, CU context provider, follow-up Q&A
- invoice_processing.py: Structured field extraction with prebuilt-invoice
- multimodal_chat.py: Multi-file session with status tracking
- Add ruff per-file-ignores for samples/ directory
- Update README with samples section, env vars, and run instructions
* feat: add remaining samples (devui_multimodal_agent, large_doc_file_search)
- S3: devui_multimodal_agent/ — DevUI web UI with CU-powered file analysis
- S4: large_doc_file_search.py — CU extraction + OpenAI vector store RAG
- Update README and samples/README.md with all 5 samples
* feat: add file_search integration for large document RAG
Add FileSearchConfig — when provided, CU-extracted markdown is automatically
uploaded to an OpenAI vector store and a file_search tool is registered on
the context. This enables token-efficient RAG retrieval for large documents
without users needing to manage vector stores manually.
- FileSearchConfig dataclass (openai_client, vector_store_name)
- Auto-create vector store, upload markdown, register file_search tool
- Auto-cleanup on close()
- When file_search is enabled, skip full content injection (use RAG instead)
- Update large_doc_file_search sample to use the integration
- 4 new tests (50 total, 90% coverage)
* fix: add key-based auth support to all samples
Follow established AF pattern: check for API key env var first,
fall back to AzureCliCredential. Supports AZURE_OPENAI_API_KEY and
AZURE_CONTENTUNDERSTANDING_API_KEY environment variables.
* FEATURE(python): add analyzer auto-detection, file_search RAG, and lazy init
_context_provider.py:
- Make analyzer_id optional (default None) with auto-detection by media
type prefix: audio->audioSearch, video->videoSearch, else documentSearch
- Add _ensure_initialized() for lazy client creation in before_run()
- Add FileSearchConfig-based vector store upload
- Fix: background-completed docs in file_search mode now upload to vector
store instead of injecting full markdown into context messages
- Add _pending_uploads queue for deferred vector store uploads
devui_file_search_agent/ (new sample):
- DevUI agent combining CU extraction + OpenAI file_search RAG
azure_responses_agent (existing sample fix):
- Add AzureCliCredential support and AZURE_AI_PROJECT_ENDPOINT fallback
Tests (19 new), Docs updated (AGENTS.md, README.md)
* feat(cu): MIME sniffing, media-aware formatting, unified timeout, vector store expiration
- Add three-layer MIME detection (fast path → filetype binary sniff → filename
fallback) to handle unreliable upstream MIME types (e.g. mp4 sent as
application/octet-stream). Adds filetype>=1.2,<2 dependency.
- Media-aware output formatting: video shows duration/resolution + all fields
as JSON; audio promotes Summary as prose; document unchanged.
- Unified timeout for all media types (removed file_search special-case that
waited indefinitely for video/audio). All files use max_wait with background
polling fallback.
- Vector store created with expires_after=1 day as crash safety net.
- Add 8 MIME sniffing tests (TestMimeSniffing class).
* fix: merge all CU content segments for video/audio analysis
CU's prebuilt-videoSearch and prebuilt-audioSearch analyzers split long
media files into multiple `contents[]` segments. Previously,
`_extract_sections()` only read `contents[0]`, causing truncated
duration, missing transcript, and incomplete fields for any video/audio
longer than a single scene.
Now iterates all segments and merges:
- duration: global min(startTimeMs) → max(endTimeMs)
- markdown: concatenated with `---` separators
- fields: same-named fields collected into per-segment list
- metadata (kind, resolution): taken from first segment
Single-segment results (documents, short audio) are unaffected.
Update test fixture to realistic 3-segment video structure and expand
assertions to verify multi-segment merging. Add documentation for
multi-segment processing and speaker diarization limitation.
* refactor: improve CU context provider docs and remove ContentLimits
- Improve class docstring: clarify endpoint (Azure AI Foundry URL with
example), credential (AzureKeyCredential vs Entra ID), and analyzer_id
(prebuilt/custom with auto-selection behavior and reference links)
- Add SUPPORTED_MEDIA_TYPES comments explaining MIME-based matching
behavior and add missing file types per CU service docs
- Use namespaced logger to align with other packages
- Remove ContentLimits and related code/tests
- Rename DEFAULT_MAX_WAIT to DEFAULT_MAX_WAIT_SECONDS for clarity
* feat: support user-provided vector store in FileSearchConfig
- Add vector_store_id field to FileSearchConfig (None = auto-create)
- Track _owns_vector_store to only delete auto-created stores on close()
- Remove vector_store_name; use internal _DEFAULT_VECTOR_STORE_NAME
- Add inline comments for private state fields
- Document output_sections default in docstring
- Update AGENTS.md, samples, and tests
* fix: remove ContentLimits from README code block
* refactor: create CU client in __init__ instead of __aenter__
Follow Azure AI Search provider pattern: create the client eagerly in
__init__, make __aenter__ a no-op. This ensures __aexit__/close() is
always safe to call and eliminates the _ensure_initialized() workaround.
* docs: add file_search param to class docstring
* feat: introduce FileSearchBackend abstraction for cross-client support
Replace direct OpenAI client usage with FileSearchBackend ABC:
- OpenAIFileSearchBackend: for OpenAIChatClient (Responses API)
- FoundryFileSearchBackend: for FoundryChatClient (Azure Foundry)
- Shared base _OpenAICompatBackend for common vector store CRUD
FileSearchConfig now takes a backend instead of openai_client.
Factory methods from_openai() and from_foundry() for convenience.
BREAKING: FileSearchConfig(openai_client=...) -> FileSearchConfig.from_openai(...)
* refactor: FileSearchBackend abstraction + caller-owned vector store
* fix: file_search reliability and sample improvements
- Poll vector store indexing (create_and_poll) to ensure file_search
returns results immediately after upload
- Set status to failed when vector store upload fails
- Skip get_analyzed_document tool in file_search mode to prevent
LLM from bypassing RAG
- Simplify sample auth: single credential, direct parameters
- Use from_foundry backend for Foundry project endpoints
* perf: set max_num_results=10 for file_search to reduce token usage
* fix: move import to top of file (E402 lint)
* chore: remove unused imports
* fix: align azure-ai-contentunderstanding with MAF coding conventions
- Add module-level docstrings to __init__.py and _context_provider.py
- Use Self return type for __aenter__ (with typing_extensions fallback)
- Use explicit typed params for __aexit__ signature
- Add sync TokenCredential to AzureCredentialTypes union
- Pass AGENT_FRAMEWORK_USER_AGENT to ContentUnderstandingClient
- Remove unused ContentLimits from public API and tests
- Fix FileSearchConfig tests to match refactored backend API
- Fix lifecycle tests to match eager client initialization
* refactor: improve CU context provider API surface and fix CI
- Refactor _analyze_file to return DocumentEntry instead of mutating dict
- Remove TokenCredential from AzureCredentialTypes (fixes mypy/pyright CI)
- Remove OpenAIFileSearchBackend/FoundryFileSearchBackend from public API
(internal to FileSearchConfig factory methods)
- Remove DocumentStatus from public exports (implementation detail)
- Update file_search comments to reflect backend-agnostic design
- Add DocumentStatus enum, analysis/upload duration tracking
- Add combined timeout for CU analysis + vector store upload
* fix: improve file_search samples and move tool guidelines to context provider
- Delete redundant devui_file_search_agent sample (duplicate of azure_openai variant)
- Move tool usage guidelines from sample agent instructions into context provider
(extend_instructions in step 6, applied automatically for all file_search users)
- Fix file_search purpose: use from_foundry() for Azure OpenAI (purpose="assistants")
- Add filename hint in upload instructions for targeted file_search queries
- Reduce max_num_results from 10 to 3 in both devui samples
- Simplify agent instructions in both samples (remove tool-specific guidance)
* feat: improve source_id, integration tests, and content assertions
- Rename DEFAULT_SOURCE_ID to "azure_ai_contentunderstanding" (matches
azure_ai_search convention)
- Improve source_id docstring to describe default value
- Clarify _detect_and_strip_files docstring (CU-supported files)
- Add invoice.pdf test fixture from Azure CU samples repo
- Refactor integration tests to use invoice.pdf directly (assert instead
of skip when fixture missing)
- Add URI content test (Content.from_uri with external URL)
- Add "CONTOSO LTD." content assertion to all integration tests
- Use max_wait=None in integration tests (wait until complete)
* feat: reject duplicate filenames, add integration tests and sample comments
- Reject duplicate document keys in before_run (skip + warn LLM to rename)
- Update _derive_doc_key docstring to document uniqueness constraint
- Add unit tests for duplicate filename rejection (cross-turn and same-turn)
- Add integration test for data URI content (from_uri with base64)
- Add integration test for background analysis (max_wait timeout + resolve)
- Add filename recommendation comments to all samples' Content.from_data()
* chore: improve doc key derivation, comments, and README
- Replace hash-based doc key with uuid4 for anonymous uploads (O(1), no payload traversal)
- Remove hashlib import (no longer needed)
- Add File Naming section to README (filename importance, duplicate rejection)
- Improve inline comments (_derive_doc_key, _extract_binary, URL parsing)
* test: strengthen _format_result assertions with exact expected strings
- Replace loose 'in' checks with exact 'assert formatted == expected'
for both multi-segment and single-segment format tests
- Add object-type fields (ShippingAddress, Speakers) to test data
to cover nested dict/list serialization
- Add position-based ordering assertions to verify structural
correctness (header -> markdown -> fields across segments)
* refactor: move invoice.pdf to shared sample_assets directory
- Move invoice.pdf from tests/cu/test_data/ to
python/samples/shared/sample_assets/ as single source of truth
- Add INVOICE_PDF_PATH constant in test_integration.py pointing
to the shared location
- Update document_qa.py, invoice_processing.py, large_doc_file_search.py
to use invoice.pdf instead of sample.pdf
* refactor: reorganize samples into numbered dirs and simplify auth
- Move script samples into 01-get-started/ with numbered prefixes
(01_document_qa, 02_multimodal_chat, 03_invoice_processing,
04_large_doc_file_search)
- Move devui samples into 02-devui/ with 01-multimodal_agent and
02-file_search_agent/{azure_openai_backend,foundry_backend}
- Move invoice.pdf to CU package-local samples/shared/sample_assets/
- Replace kwargs dicts with direct constructor calls; support both
API key (AZURE_OPENAI_API_KEY) and AzureCliCredential
- Update README sample table with new paths
* fix: resolve CI lint errors (D205, RUF001, E501)
- Fix D205: single-line docstring summary for _detect_and_strip_files
- Fix RUF001: replace EN DASH with HYPHEN-MINUS in segment headers
- Fix E501: wrap long assertion lines in tests
- Also includes samples reorg and auth simplification
* refactor: overhaul samples — FoundryChatClient, sessions, remove get_analyzed_document
Samples:
- Switch all samples from deprecated AzureOpenAIResponsesClient to FoundryChatClient
- Add 02_multi_turn_session.py showing AgentSession persistence across turns
- Rewrite 03_multimodal_chat.py with real PDF + audio + video (parallel
analysis), per-modality follow-ups, cross-document question, elapsed
time, user prompts, and input token counts
- Renumber: 02->03 multimodal, 03->04 invoice, 04->05 file_search
Context provider:
- Remove get_analyzed_document tool -- full content is in conversation
history via InMemoryHistoryProvider, no retrieval tool needed
- Remove follow-up turn instructions about tools
- Only list_documents tool remains (for status queries)
- Update README to reflect tool removal
* feat: add 05_background_analysis sample and fix 04 session/max_wait
- Add 05_background_analysis.py demonstrating non-blocking CU analysis
with max_wait=1s, status tracking via list_documents(), and automatic
background task resolution on subsequent turns
- Fix 04_invoice_processing.py: add max_wait=None and AgentSession
- Rename 05→06 large_doc_file_search
- Update README sample table
* docs: update README and fix sample 06
README:
- Switch Quick Start from AzureOpenAIResponsesClient to FoundryChatClient
- Add AgentSession to Quick Start example
- Fix status values: pending -> analyzing/uploading/ready/failed
- Fix env var: AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME -> AZURE_OPENAI_DEPLOYMENT_NAME
- Update samples section with new paths, link to samples/README.md
- Update multi-segment description to reflect per-segment fields
Sample 06:
- Fix from_openai -> from_foundry for Azure endpoints
- Add AgentSession and max_wait=None
* docs: rewrite README — concise format, prerequisites, CU link
* fix: resolve pyright errors in _format_result segment cast
* docs: add numbered section comments and fresh sample output to all samples
- Add numbered section comments (# 1. ..., # 2. ...) per SAMPLE_GUIDELINES
- Re-run all 6 samples and update expected output with real results
- Fix duplicate sample output blocks in 04 and 05
- Update README code example to use public invoice URL
* feat: add load_settings support for env var configuration
- Make endpoint optional in constructor — auto-loads from
AZURE_CONTENTUNDERSTANDING_ENDPOINT env var via load_settings()
- Add ContentUnderstandingSettings TypedDict
- Add env_file_path/env_file_encoding params for .env file support
- Add 4 unit tests: env var loading, explicit override, missing
endpoint error, missing credential error
- Update README with env var auto-resolution docs
- Follows framework convention used by all other packages
* docs: polish README — fix duplicate env var, add Next steps, service limits link
* chore: trim invoice fixture from 199K to 33 lines
Keep only VendorName, InvoiceTotal, DueDate, InvoiceDate, InvoiceId
fields and first 500 chars of markdown. Strip spans/source/coordinates.
Reduces fixture from 6.6MB to 1.2KB.
* feat: per-file analyzer_id override via additional_properties
- Read analyzer_id from Content.additional_properties for per-file override
- Resolution order: per-file > provider-level > auto-detect by media type
- Update class docstring documenting filename and analyzer_id properties
- Update sample 04 to demonstrate per-file override (prebuilt-invoice)
- Add unit test for per-file analyzer override
* Trim PDF test fixture and clarify unique filename requirement
- Trim analyze_pdf_result.json from 4427 to 23 lines by removing
pages, words, lines, paragraphs, sections, spans, and source
fields that are not used by any unit test.
- Add docstring note that filename must be unique within a session;
duplicate filenames are rejected and the file will not be analyzed.
* Update python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_context_provider.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_context_provider.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/azure_openai_backend/agent.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update python/packages/azure-ai-contentunderstanding/samples/02-devui/01-multimodal_agent/agent.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update python/packages/azure-ai-contentunderstanding/samples/01-get-started/06_large_doc_file_search.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Fix AGENTS.md to match implementation; remove unused variable in test helper
AGENTS.md:
- Remove _ensure_initialized() reference (client is created in __init__)
- Fix multi-segment docs: segments kept as list, not merged into fields
- Remove get_analyzed_document() reference (only list_documents registered)
- Update sample names to match current directory structure
test_context_provider.py:
- Simplify _make_data_uri() — remove unused 'encoded' variable
* Fix premature file_search instruction for background-completed docs
- Change _resolve_pending_tasks() instruction from 'Use file_search'
to 'being indexed' since the upload hasn't completed yet at that point.
- Add LLM instruction on upload failure in step 1b so the agent can
inform the user the document isn't searchable.
* fix: wrap long line in devui agent instructions (E501)
* Fix Copilot review: unused logger, stray code in README, await cancelled tasks
- _file_search.py: Remove unused logger and logging import
- 01-multimodal_agent/README.md: Remove accidentally pasted Python script
- _context_provider.py close(): Await cancelled tasks before closing
client to prevent 'Task destroyed but pending' warnings
* Sanitize doc keys and fix duplicate filename re-injection
- Add _sanitize_doc_key() to strip control characters, collapse
whitespace, and cap length at 255 chars — prevents prompt injection
via crafted filenames in extend_instructions() calls.
- Track accepted doc_keys in step 3 so step 5 only injects content
for files actually analyzed this turn, not pre-existing duplicates.
- Soften duplicate upload instruction wording (remove IMPORTANT/caps).
* fix: add type annotation to tasks_to_cancel for pyright
* Move per-session mutable state to state dict for session isolation
Previously _pending_tasks, _pending_uploads, and _uploaded_file_ids
were stored on self, shared across all sessions. This caused
cross-session leakage: Session A's background task results could be
injected into Session B's context.
Now these are stored in the per-session state dict. Global copies
(_all_pending_tasks, _all_uploaded_file_ids) are kept on self only
for best-effort cleanup in close().
Add 2 new TestSessionIsolation tests verifying that background tasks
and resolved content stay within their originating session.
* Remove unused AnalysisSection enum values
Only MARKDOWN and FIELDS are handled by _extract_sections().
Remove FIELD_GROUNDING, TABLES, PARAGRAPHS, SECTIONS to avoid
exposing dead options to users.
* Recursively flatten object/array field values for cleaner LLM output
- Use SDK .value property with recursive extraction for object/array fields
- Object: AmountDue -> {Amount: 610, CurrencyCode: USD} (was raw SDK dict)
- Array: LineItems -> list of flattened items (was raw SDK list)
- Update invoice fixture with object/array fields from prebuilt-invoice
- Add 3 unit tests for object, array, and nested object field extraction
* Preserve sub-field confidence; compare full expected JSON in tests
* Remove incorrect MIME aliases (audio/mp4, video/x-matroska)
* feat: add AnalysisInput, content_range, warnings, and category support
- Use SDK AnalysisInput model instead of raw body dict for begin_analyze
- Forward content_range from additional_properties to CU (page/time ranges)
- Extract CU warnings with code/message/target (ODataV4Format) into output
- Include content-level category from classifier analyzers
- Add 5 new tests: warnings, category, content_range forwarding
- Fix pyright with explicit casts; fix en-dash lint (RUF002)
* fix: falsy-0 bug in duration calc; improve test coverage
- Fix start_time_ms=0 treated as falsy by 'or' short-circuit, use
'is None' checks instead for duration and segment time extraction
- Update warnings test to use RAI ContentFiltered codes
- Enrich warnings extraction to include code/message/target (ODataV4Format)
- Add multi-segment video category test with per-segment assertions
* refactor: split _context_provider.py into focused modules
- Extract _constants.py: SUPPORTED_MEDIA_TYPES, MIME_ALIASES, analyzer maps
- Extract _detection.py: file detection, MIME sniffing, doc key derivation
- Extract _extraction.py: result extraction, field flattening, LLM formatting
- _context_provider.py delegates via thin wrappers (793 lines, was 1255)
- Update test imports to use _constants.py for SUPPORTED_MEDIA_TYPES
* docs: update AGENTS.md with DocumentStatus, FileSearchBackend, and _file_search.py
* refactor: replace AnalysisSection enum with Literal type for simpler DX
- Remove AnalysisSection(str, Enum) class, replace with Literal["markdown", "fields"] type alias
- Users can now pass plain strings: output_sections=["markdown"] — no extra import needed
- AnalysisSection type alias still exported for type annotation use
- Update all samples, tests, and internal code to use string literals
- Address PR review feedback (eavanvalkenburg)
* refactor: replace asyncio.Task with continuation tokens for serializable state
- Replace state["_pending_tasks"] (asyncio.Task — not serializable) with
state["_pending_tokens"] (dict of continuation token strings) so the
framework can persist session state to disk/storage
- Resume pending analyses via Azure SDK continuation_token mechanism
- Fix: resumed pollers have stale cached status (done() always False),
use asyncio.wait_for(poller.result()) with 10s min timeout instead
- Remove _background_poll(), _all_pending_tasks, and task cancellation
- Address PR review feedback (eavanvalkenburg): state must be serializable
* fix: resolve CI lint (RUF052) and mypy (call-overload) errors
* feat: add structured output (Pydantic model) to invoice processing sample
- Use response_format=InvoiceResult for schema-constrained LLM output
- Use output_sections=["fields"] only (no markdown needed for structured output)
- Add LowConfidenceField model with confidence values
- Add comments about prebuilt-invoice extensive schema vs simplified model
- Address PR review feedback (eavanvalkenburg): use structured response
* fix: use FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL env vars in all samples
Replace AZURE_AI_PROJECT_ENDPOINT → FOUNDRY_PROJECT_ENDPOINT and
AZURE_OPENAI_DEPLOYMENT_NAME → FOUNDRY_MODEL across all sample .py and
README.md files. Address PR review feedback (eavanvalkenburg).
* refactor: remove background_analysis sample, use FoundryChatClient in DevUI
- Remove 05_background_analysis.py (per reviewer feedback — discuss max_wait
design separately from samples)
- Renumber 06_large_doc_file_search.py → 05_large_doc_file_search.py
- Replace AzureOpenAIResponsesClient with FoundryChatClient in all DevUI samples
- Replace client.as_agent() with Agent(client=client, ...) everywhere
- Add max_wait comments explaining interactive vs batch usage
- Update README.md and AGENTS.md
- Address PR review feedback (eavanvalkenburg)
* fix: vector_stores API moved from beta namespace in OpenAI SDK
* docs: add comments about multi-file support and CU service limits in file_search sample
* fix: broken markdown links after sample removal and renumbering
* fix: migrate BaseContextProvider to ContextProvider (non-deprecated)
* fix: Message(text=) -> Message(contents=[]) for API compatibility
* Inline _constants.py into consuming modules
Remove _constants.py and move constants to where they are used:
- SUPPORTED_MEDIA_TYPES, MIME_ALIASES → _detection.py
- MEDIA_TYPE_ANALYZER_MAP, DEFAULT_ANALYZER → _context_provider.py
Addresses review feedback to reduce file count.
* Mark package as alpha per package management skill
- Version: 1.0.0b260401 → 1.0.0a260401
- Classifier: Development Status 4 - Beta → 3 - Alpha
- Add to PACKAGE_STATUS.md as alpha
Follows the alpha package checklist from python-package-management skill.
* Replace extend_instructions with extend_messages for status notifications
Status/error/result notifications now use extend_messages (conversation
context) instead of extend_instructions (system prompt). This avoids
system prompt bloat and keeps behavioral directives separate from
event notifications.
- 11 extend_instructions calls → extend_messages (role='user')
- 1 extend_instructions retained: tool usage guidelines (behavioral)
- 6 test assertions updated to check context_messages
All 84 unit tests + 5 live integration tests pass.
* Fix lint: E402 import order, ISC004 implicit string concatenation
- Move constants after all imports to fix E402
- Wrap multi-line strings in parentheses inside contents=[] to fix ISC004
* Fix lint: remove unused json import in invoice sample
* Fix CI: apply ruff format + fix E501 line length after reformatting
ruff format expands Message() calls to multi-line, pushing string
indentation deeper. Break long strings to fit within 120 char limit
after formatting. Also removes unused json import in sample.
* Address review feedback: keyword-only args, accept pre-built client, remove wrappers
- All __init__ args now keyword-only (matches FoundryChatClient pattern)
- New 'client' param accepts pre-built ContentUnderstandingClient
- core dep bound: >=1.0.0rc5 → >=1.0.0,<2
- Self import moved after local imports
- Removed 9 static method wrappers; callsites use module functions directly
- Tests updated to import derive_doc_key and format_result directly
* fix: remove duplicate ContentUnderstandingClient instantiation
The client was being created twice — once inside the if/else block and
again unconditionally after it. The second instantiation overwrote the
pre-built client path and failed type checking when credential was None.
* rename: azure-ai-contentunderstanding → azure-contentunderstanding
Package: agent-framework-azure-ai-contentunderstanding → agent-framework-azure-contentunderstanding
Module: agent_framework_azure_ai_contentunderstanding → agent_framework_azure_contentunderstanding
Directory: packages/azure-ai-contentunderstanding → packages/azure-contentunderstanding
Per agreement with PM and MAF team to drop 'AI' from the package name.
* feat: add ContentUnderstanding re-export to agent_framework.foundry namespace
Enables: from agent_framework.foundry import ContentUnderstandingContextProvider
Exports: ContentUnderstandingContextProvider, FileSearchConfig,
FileSearchBackend, AnalysisSection, DocumentStatus
Updates all samples and README to use the foundry namespace import.
* fix: add missing copyright headers to standalone sample scripts
* chore: remove .vscode/settings.json and add to .gitignore
* refactor: reuse FoundryChatClient.client for vector store ops in file_search sample
Address review feedback from TaoChenOSU:
- 05_large_doc_file_search.py: use client.client instead of manually
constructing AsyncAzureOpenAI; remove openai dependency
- azure_openai_backend/agent.py: import reorder only (AIProjectClient
kept — required for sync vector store creation in DevUI)
* fix: skip closing client when caller passes pre-built client
When a ContentUnderstandingClient is passed via client=, the caller
owns its lifecycle. Added _owns_client flag so close() only closes
the client when we created it internally.
---------
Co-authored-by: yungshinlin <yungshin@msn.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Yung-Shin Lin
co-authored by
yungshinlin
Copilot
yungshinlin
Copilot
parent
3a463b8bf6
commit
1e1eda65ce
+28
@@ -0,0 +1,28 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Azure Content Understanding integration for Microsoft Agent Framework.
|
||||
|
||||
Provides a context provider that analyzes file attachments (documents, images,
|
||||
audio, video) using Azure Content Understanding and injects structured results
|
||||
into the LLM context.
|
||||
"""
|
||||
|
||||
import importlib.metadata
|
||||
|
||||
from ._context_provider import ContentUnderstandingContextProvider
|
||||
from ._file_search import FileSearchBackend
|
||||
from ._models import AnalysisSection, DocumentStatus, FileSearchConfig
|
||||
|
||||
try:
|
||||
__version__ = importlib.metadata.version(__name__)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
__version__ = "0.0.0"
|
||||
|
||||
__all__ = [
|
||||
"AnalysisSection",
|
||||
"ContentUnderstandingContextProvider",
|
||||
"DocumentStatus",
|
||||
"FileSearchBackend",
|
||||
"FileSearchConfig",
|
||||
"__version__",
|
||||
]
|
||||
+858
@@ -0,0 +1,858 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Azure Content Understanding context provider using ContextProvider.
|
||||
|
||||
This module provides ``ContentUnderstandingContextProvider``, built on the
|
||||
:class:`ContextProvider` hooks pattern. It automatically detects file
|
||||
attachments, analyzes them via the Azure Content Understanding API, and
|
||||
injects structured results into the LLM context.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING, Any, ClassVar, TypedDict
|
||||
|
||||
from agent_framework import (
|
||||
AGENT_FRAMEWORK_USER_AGENT,
|
||||
Content,
|
||||
ContextProvider,
|
||||
FunctionTool,
|
||||
Message,
|
||||
SessionContext,
|
||||
)
|
||||
from agent_framework._sessions import AgentSession
|
||||
from agent_framework._settings import load_settings
|
||||
from azure.ai.contentunderstanding.aio import ContentUnderstandingClient
|
||||
from azure.ai.contentunderstanding.models import AnalysisInput, AnalysisResult
|
||||
from azure.core.credentials import AzureKeyCredential
|
||||
from azure.core.credentials_async import AsyncTokenCredential
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from agent_framework._agents import SupportsAgentRun
|
||||
|
||||
from ._detection import (
|
||||
detect_and_strip_files,
|
||||
)
|
||||
from ._extraction import extract_sections, format_result
|
||||
from ._models import AnalysisSection, DocumentEntry, DocumentStatus, FileSearchConfig
|
||||
|
||||
if sys.version_info >= (3, 11):
|
||||
from typing import Self # pragma: no cover
|
||||
else:
|
||||
from typing_extensions import Self # pragma: no cover
|
||||
|
||||
logger = logging.getLogger("agent_framework.azure_contentunderstanding")
|
||||
|
||||
AzureCredentialTypes = AzureKeyCredential | AsyncTokenCredential
|
||||
|
||||
# Mapping from media type prefix to the appropriate prebuilt CU analyzer.
|
||||
# Used when analyzer_id is None (auto-detect mode).
|
||||
MEDIA_TYPE_ANALYZER_MAP: dict[str, str] = {
|
||||
"audio/": "prebuilt-audioSearch",
|
||||
"video/": "prebuilt-videoSearch",
|
||||
}
|
||||
DEFAULT_ANALYZER: str = "prebuilt-documentSearch"
|
||||
|
||||
|
||||
class ContentUnderstandingSettings(TypedDict, total=False):
|
||||
"""Settings for ContentUnderstandingContextProvider with auto-loading from environment.
|
||||
|
||||
Settings are resolved in this order: explicit keyword arguments, values from an
|
||||
explicitly provided .env file, then environment variables with the prefix
|
||||
``AZURE_CONTENTUNDERSTANDING_``.
|
||||
|
||||
Keys:
|
||||
endpoint: Azure AI Foundry endpoint URL.
|
||||
Can be set via environment variable ``AZURE_CONTENTUNDERSTANDING_ENDPOINT``.
|
||||
"""
|
||||
|
||||
endpoint: str | None
|
||||
|
||||
|
||||
class ContentUnderstandingContextProvider(ContextProvider):
|
||||
"""Context provider that analyzes file attachments using Azure Content Understanding.
|
||||
|
||||
Automatically detects supported file attachments in the agent's input,
|
||||
analyzes them via CU, and injects the structured results (markdown, fields)
|
||||
into the LLM context. Supports multiple documents per session with background
|
||||
processing for long-running analyses. Optionally integrates with a vector
|
||||
store backend for ``file_search``-based RAG retrieval on LLM clients that
|
||||
support it.
|
||||
|
||||
Args:
|
||||
endpoint: Azure AI Foundry endpoint URL
|
||||
(e.g., ``"https://<your-foundry-resource>.services.ai.azure.com/"``).
|
||||
Can also be set via environment variable
|
||||
``AZURE_CONTENTUNDERSTANDING_ENDPOINT``.
|
||||
credential: An ``AzureKeyCredential`` for API key auth or an
|
||||
``AsyncTokenCredential`` (e.g., ``DefaultAzureCredential``) for
|
||||
Microsoft Entra ID auth.
|
||||
analyzer_id: A prebuilt or custom CU analyzer ID. When ``None``
|
||||
(default), a prebuilt analyzer is chosen automatically based on
|
||||
the file's media type: ``prebuilt-documentSearch`` for documents
|
||||
and images, ``prebuilt-audioSearch`` for audio, and
|
||||
``prebuilt-videoSearch`` for video.
|
||||
Analyzer reference: https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/analyzer-reference
|
||||
Prebuilt analyzers: https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/prebuilt-analyzers
|
||||
max_wait: Max seconds to wait for analysis before deferring to background.
|
||||
``None`` waits until complete.
|
||||
output_sections: Which CU output sections to pass to LLM.
|
||||
Defaults to ``["markdown", "fields"]``.
|
||||
file_search: Optional configuration for uploading CU-extracted markdown to
|
||||
a vector store for token-efficient RAG retrieval. When provided, full
|
||||
content injection is replaced by ``file_search`` tool registration.
|
||||
The ``FileSearchConfig`` abstraction is backend-agnostic — use
|
||||
``FileSearchConfig.from_openai()`` or ``FileSearchConfig.from_foundry()``
|
||||
for supported providers, or supply a custom ``FileSearchBackend``
|
||||
implementation for other vector store services.
|
||||
source_id: Unique identifier for this provider instance, used for message
|
||||
attribution and tool registration. Defaults to ``"azure_contentunderstanding"``.
|
||||
env_file_path: Path to a ``.env`` file for loading settings.
|
||||
env_file_encoding: Encoding of the ``.env`` file.
|
||||
|
||||
Per-file ``additional_properties`` on ``Content`` objects:
|
||||
The provider reads the following keys from
|
||||
``Content.additional_properties`` (passed via ``Content.from_data()``
|
||||
or ``Content.from_uri()``):
|
||||
|
||||
``filename`` (str):
|
||||
The document key used for tracking, status, and LLM references.
|
||||
Without a filename, a UUID-based key is generated.
|
||||
Must be unique within a session — uploading a file with a
|
||||
duplicate filename will be rejected and the file will not be
|
||||
analyzed.
|
||||
|
||||
``analyzer_id`` (str):
|
||||
Per-file analyzer override. Takes priority over the provider-level
|
||||
``analyzer_id``. Useful for mixing analyzers in the same turn
|
||||
(e.g., ``prebuilt-invoice`` for invoices alongside
|
||||
``prebuilt-documentSearch`` for general documents).
|
||||
|
||||
``content_range`` (str):
|
||||
Subset of the input to analyze. For documents, use 1-based page
|
||||
numbers (e.g., ``"1-3"`` for pages 1-3, ``"1,3,5-"`` for pages
|
||||
1, 3, and 5 onward). For audio/video, use milliseconds
|
||||
(e.g., ``"0-60000"`` for the first 60 seconds).
|
||||
|
||||
Example::
|
||||
|
||||
Content.from_data(
|
||||
pdf_bytes,
|
||||
"application/pdf",
|
||||
additional_properties={
|
||||
"filename": "invoice.pdf",
|
||||
"analyzer_id": "prebuilt-invoice",
|
||||
"content_range": "1-3",
|
||||
},
|
||||
)
|
||||
"""
|
||||
|
||||
DEFAULT_SOURCE_ID: ClassVar[str] = "azure_contentunderstanding"
|
||||
DEFAULT_MAX_WAIT_SECONDS: ClassVar[float] = 5.0
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
endpoint: str | None = None,
|
||||
credential: AzureCredentialTypes | None = None,
|
||||
client: ContentUnderstandingClient | None = None,
|
||||
analyzer_id: str | None = None,
|
||||
max_wait: float | None = DEFAULT_MAX_WAIT_SECONDS,
|
||||
output_sections: list[AnalysisSection] | None = None,
|
||||
file_search: FileSearchConfig | None = None,
|
||||
source_id: str = DEFAULT_SOURCE_ID,
|
||||
env_file_path: str | None = None,
|
||||
env_file_encoding: str | None = None,
|
||||
) -> None:
|
||||
super().__init__(source_id)
|
||||
|
||||
if client is not None:
|
||||
# Use the pre-built client directly — endpoint/credential are ignored.
|
||||
self._client = client
|
||||
self._owns_client = False
|
||||
self._endpoint = ""
|
||||
self._credential = None
|
||||
else:
|
||||
# Build a new client from endpoint + credential.
|
||||
settings = load_settings(
|
||||
ContentUnderstandingSettings,
|
||||
env_prefix="AZURE_CONTENTUNDERSTANDING_",
|
||||
required_fields=["endpoint"],
|
||||
endpoint=endpoint,
|
||||
env_file_path=env_file_path,
|
||||
env_file_encoding=env_file_encoding,
|
||||
)
|
||||
resolved_endpoint: str = settings["endpoint"] # type: ignore[assignment] # validated by load_settings
|
||||
|
||||
if credential is None:
|
||||
raise ValueError(
|
||||
"Azure credential is required. Provide a 'credential' keyword argument "
|
||||
"(e.g., AzureKeyCredential or AzureCliCredential), or pass a pre-built "
|
||||
"'client' (ContentUnderstandingClient) instead."
|
||||
)
|
||||
|
||||
self._endpoint = resolved_endpoint
|
||||
self._credential = credential
|
||||
self._client = ContentUnderstandingClient(
|
||||
self._endpoint, self._credential, user_agent=AGENT_FRAMEWORK_USER_AGENT
|
||||
)
|
||||
self._owns_client = True
|
||||
self.analyzer_id = analyzer_id
|
||||
self.max_wait = max_wait
|
||||
self.output_sections: list[AnalysisSection] = output_sections or ["markdown", "fields"]
|
||||
self.file_search = file_search
|
||||
# Global list of uploaded file IDs — used only by close() for
|
||||
# best-effort cleanup. The authoritative per-session copy lives in
|
||||
# state["_uploaded_file_ids"] (populated in before_run). This global
|
||||
# list may contain entries from multiple sessions; that is intentional
|
||||
# for cleanup.
|
||||
self._all_uploaded_file_ids: list[str] = []
|
||||
|
||||
async def __aenter__(self) -> Self:
|
||||
"""Async context manager entry."""
|
||||
return self
|
||||
|
||||
async def __aexit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: Any,
|
||||
) -> None:
|
||||
"""Async context manager exit — cleanup clients."""
|
||||
await self.close()
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close the underlying CU client and clean up resources.
|
||||
|
||||
Uses global tracking lists for best-effort cleanup across all
|
||||
sessions that used this provider instance.
|
||||
"""
|
||||
# Clean up uploaded files; the vector store itself is caller-managed.
|
||||
if self.file_search and self._all_uploaded_file_ids:
|
||||
await self._cleanup_uploaded_files()
|
||||
# Only close the client if we created it internally.
|
||||
# When a pre-built client was passed in, the caller owns its lifecycle.
|
||||
if self._owns_client:
|
||||
await self._client.close()
|
||||
|
||||
async def before_run(
|
||||
self,
|
||||
*,
|
||||
agent: SupportsAgentRun,
|
||||
session: AgentSession,
|
||||
context: SessionContext,
|
||||
state: dict[str, Any],
|
||||
) -> None:
|
||||
"""Analyze file attachments and inject results into the LLM context.
|
||||
|
||||
This method is called automatically by the framework before each LLM invocation.
|
||||
"""
|
||||
documents: dict[str, DocumentEntry] = state.setdefault("documents", {})
|
||||
|
||||
# Per-session mutable state — isolated per session to prevent cross-session leakage.
|
||||
# _pending_tokens stores serializable continuation tokens (not asyncio.Task objects)
|
||||
# so that state can be persisted to disk/storage by the framework.
|
||||
# Structure: {doc_key: {"continuation_token": <opaque Azure SDK string>,
|
||||
# "analyzer_id": <CU analyzer used for this file>}}
|
||||
pending_tokens: dict[str, dict[str, str]] = state.setdefault("_pending_tokens", {})
|
||||
pending_uploads: list[tuple[str, DocumentEntry]] = state.setdefault("_pending_uploads", [])
|
||||
|
||||
# 1. Resolve pending background analyses via continuation tokens
|
||||
await self._resolve_pending_tokens(pending_tokens, pending_uploads, documents, context)
|
||||
|
||||
# 1b. Upload any documents that completed in the background (file_search mode)
|
||||
if pending_uploads:
|
||||
# Use a bounded timeout so before_run() stays responsive and does not block
|
||||
# indefinitely on slow vector store indexing.
|
||||
upload_timeout = getattr(self, "max_wait", None)
|
||||
remaining_uploads: list[tuple[str, DocumentEntry]] = []
|
||||
for upload_key, upload_entry in pending_uploads:
|
||||
try:
|
||||
if upload_timeout is not None:
|
||||
await asyncio.wait_for(
|
||||
self._upload_to_vector_store(upload_key, upload_entry, state=state),
|
||||
timeout=upload_timeout,
|
||||
)
|
||||
else:
|
||||
await self._upload_to_vector_store(upload_key, upload_entry, state=state)
|
||||
except asyncio.TimeoutError:
|
||||
# Leave timed-out uploads pending so they can be retried on a later turn.
|
||||
logger.warning(
|
||||
"Timed out while uploading document '%s' to vector store; will retry later.",
|
||||
upload_key,
|
||||
)
|
||||
remaining_uploads.append((upload_key, upload_entry))
|
||||
except Exception:
|
||||
# Log unexpected failures and drop the upload entry; this matches prior
|
||||
# behavior where all pending uploads were cleared regardless of outcome.
|
||||
logger.exception(
|
||||
"Error while uploading document '%s' to vector store; dropping from pending list.",
|
||||
upload_key,
|
||||
)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"Document '{upload_key}' was analyzed but failed to upload "
|
||||
"to the vector store. The document content is not available for search."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
state["_pending_uploads"] = remaining_uploads
|
||||
pending_uploads = remaining_uploads
|
||||
|
||||
# 2. Detect CU-supported file attachments, strip them from input, and return for analysis
|
||||
new_files = detect_and_strip_files(context)
|
||||
|
||||
# 3. Analyze new files using CU (track elapsed time for combined timeout)
|
||||
file_start_times: dict[str, float] = {}
|
||||
accepted_keys: set[str] = set() # doc_keys successfully accepted for analysis this turn
|
||||
for doc_key, content_item, binary_data in new_files:
|
||||
# Reject duplicate filenames — re-analyzing would orphan vector store entries
|
||||
if doc_key in documents:
|
||||
logger.warning("Duplicate document key '%s' — skipping (already exists in session).", doc_key)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"The user tried to upload '{doc_key}', but a file with that name "
|
||||
"was already uploaded earlier in this session. The new upload was rejected "
|
||||
"and was not analyzed. Tell the user that a file with the same name "
|
||||
"already exists and they need to rename the file before uploading again."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
continue
|
||||
file_start_times[doc_key] = time.monotonic()
|
||||
doc_entry = await self._analyze_file(doc_key, content_item, binary_data, context, pending_tokens)
|
||||
if doc_entry:
|
||||
documents[doc_key] = doc_entry
|
||||
accepted_keys.add(doc_key)
|
||||
|
||||
# 4. Inject content for ready documents and register tools
|
||||
if documents:
|
||||
self._register_tools(documents, context)
|
||||
|
||||
# 5. On upload turns, inject content for docs accepted this turn
|
||||
for doc_key in accepted_keys:
|
||||
entry = documents.get(doc_key)
|
||||
if entry and entry["status"] == DocumentStatus.READY and entry["result"]:
|
||||
# Upload to vector store if file_search is configured
|
||||
if self.file_search:
|
||||
# Combined timeout: subtract CU analysis time from max_wait
|
||||
remaining: float | None = None
|
||||
if self.max_wait is not None:
|
||||
elapsed = time.monotonic() - file_start_times.get(doc_key, time.monotonic())
|
||||
remaining = max(0.0, self.max_wait - elapsed)
|
||||
uploaded = await self._upload_to_vector_store(doc_key, entry, timeout=remaining, state=state)
|
||||
if uploaded:
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"The user just uploaded '{entry['filename']}'. It has been analyzed "
|
||||
"using Azure Content Understanding and indexed in a vector store. "
|
||||
f"When using file_search, include '{entry['filename']}' in your query "
|
||||
"to retrieve content from this specific document."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
elif entry.get("error"):
|
||||
# Upload failed (not timeout — actual error)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"Document '{entry['filename']}' was analyzed but failed to upload "
|
||||
"to the vector store. The document content is not available for search."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
else:
|
||||
# Upload deferred to background (timeout)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"Document '{entry['filename']}' has been analyzed and is being indexed. "
|
||||
"Ask about it again in a moment."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
else:
|
||||
# Without file_search, inject full content into context
|
||||
context.extend_messages(
|
||||
self,
|
||||
[
|
||||
Message(role="user", contents=[format_result(entry["filename"], entry["result"])]),
|
||||
],
|
||||
)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
(
|
||||
f"The user just uploaded '{entry['filename']}'."
|
||||
" It has been analyzed using Azure Content Understanding."
|
||||
" The document content (markdown) and extracted fields"
|
||||
" (JSON) are provided above."
|
||||
" If the user's question is ambiguous,"
|
||||
" prioritize this most recently uploaded document."
|
||||
" Use specific field values and cite page numbers"
|
||||
" when answering."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
# 6. Register file_search tool (for LLM clients that support it)
|
||||
if self.file_search:
|
||||
context.extend_tools(
|
||||
self.source_id,
|
||||
[self.file_search.file_search_tool],
|
||||
)
|
||||
context.extend_instructions(
|
||||
self.source_id,
|
||||
"Tool usage guidelines:\n"
|
||||
"- Use file_search ONLY when answering questions about document content.\n"
|
||||
"- Use list_documents() for status queries (e.g. 'list docs', 'what's uploaded?').\n"
|
||||
"- Do NOT call file_search for status queries — it wastes tokens.",
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Analyzer Resolution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _resolve_analyzer_id(self, media_type: str) -> str:
|
||||
"""Return the analyzer ID to use for the given media type.
|
||||
|
||||
When ``self.analyzer_id`` is set, it is always returned (explicit
|
||||
override). Otherwise the media type prefix is matched against the
|
||||
known mapping, falling back to ``prebuilt-documentSearch``.
|
||||
"""
|
||||
if self.analyzer_id is not None:
|
||||
return self.analyzer_id
|
||||
for prefix, analyzer in MEDIA_TYPE_ANALYZER_MAP.items():
|
||||
if media_type.startswith(prefix):
|
||||
return analyzer
|
||||
return DEFAULT_ANALYZER
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Analysis
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _analyze_file(
|
||||
self,
|
||||
doc_key: str,
|
||||
content: Content,
|
||||
binary_data: bytes | None,
|
||||
context: SessionContext,
|
||||
pending_tokens: dict[str, dict[str, str]] | None = None,
|
||||
) -> DocumentEntry | None:
|
||||
"""Analyze a single file via CU with timeout handling.
|
||||
|
||||
The analyzer is resolved in priority order:
|
||||
1. Per-file override via ``content.additional_properties["analyzer_id"]``
|
||||
2. Provider-level default via ``self.analyzer_id``
|
||||
3. Auto-detect by media type (document/audio/video)
|
||||
|
||||
Returns:
|
||||
A ``DocumentEntry`` (ready, analyzing, or failed), or ``None`` if
|
||||
file data could not be extracted.
|
||||
"""
|
||||
media_type = content.media_type or "application/octet-stream"
|
||||
filename = doc_key
|
||||
|
||||
# Per-file analyzer override from additional_properties
|
||||
props = content.additional_properties or {}
|
||||
per_file_analyzer = props.get("analyzer_id")
|
||||
content_range = props.get("content_range")
|
||||
resolved_analyzer = per_file_analyzer or self._resolve_analyzer_id(media_type)
|
||||
t0 = time.monotonic()
|
||||
|
||||
try:
|
||||
# Start CU analysis
|
||||
if content.type == "uri" and content.uri and not content.uri.startswith("data:"):
|
||||
poller = await self._client.begin_analyze(
|
||||
resolved_analyzer,
|
||||
inputs=[AnalysisInput(url=content.uri, content_range=content_range)],
|
||||
)
|
||||
elif binary_data:
|
||||
poller = await self._client.begin_analyze_binary(
|
||||
resolved_analyzer,
|
||||
binary_input=binary_data,
|
||||
content_type=media_type,
|
||||
)
|
||||
else:
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[Message(role="user", contents=[f"Could not extract file data from '{filename}'."])],
|
||||
)
|
||||
return None
|
||||
|
||||
# Wait with timeout; defer to background polling on timeout.
|
||||
try:
|
||||
result = await asyncio.wait_for(poller.result(), timeout=self.max_wait)
|
||||
except asyncio.TimeoutError:
|
||||
# Save continuation token for resuming on next before_run().
|
||||
# Continuation tokens are serializable strings, so state can
|
||||
# be persisted to disk/storage without issues.
|
||||
token = poller.continuation_token()
|
||||
logger.info("Analysis of '%s' timed out; deferring to background via continuation token.", filename)
|
||||
if pending_tokens is not None:
|
||||
pending_tokens[doc_key] = {
|
||||
"continuation_token": token,
|
||||
"analyzer_id": resolved_analyzer,
|
||||
}
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[f"Document '{filename}' is being analyzed. Ask about it again in a moment."],
|
||||
)
|
||||
],
|
||||
)
|
||||
return DocumentEntry(
|
||||
status=DocumentStatus.ANALYZING,
|
||||
filename=filename,
|
||||
media_type=media_type,
|
||||
analyzer_id=resolved_analyzer,
|
||||
analyzed_at=None,
|
||||
analysis_duration_s=None,
|
||||
upload_duration_s=None,
|
||||
result=None,
|
||||
error=None,
|
||||
)
|
||||
|
||||
# Analysis completed within timeout
|
||||
analysis_duration = round(time.monotonic() - t0, 2)
|
||||
extracted = self._extract_sections(result)
|
||||
logger.info("Analyzed '%s' with analyzer '%s' in %.1fs.", filename, resolved_analyzer, analysis_duration)
|
||||
return DocumentEntry(
|
||||
status=DocumentStatus.READY,
|
||||
filename=filename,
|
||||
media_type=media_type,
|
||||
analyzer_id=resolved_analyzer,
|
||||
analyzed_at=datetime.now(tz=timezone.utc).isoformat(),
|
||||
analysis_duration_s=analysis_duration,
|
||||
upload_duration_s=None,
|
||||
result=extracted,
|
||||
error=None,
|
||||
)
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning("CU analysis error for '%s': %s", filename, e)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[Message(role="user", contents=[f"Could not analyze '{filename}': {e}"])],
|
||||
)
|
||||
return DocumentEntry(
|
||||
status=DocumentStatus.FAILED,
|
||||
filename=filename,
|
||||
media_type=media_type,
|
||||
analyzer_id=resolved_analyzer,
|
||||
analyzed_at=datetime.now(tz=timezone.utc).isoformat(),
|
||||
analysis_duration_s=round(time.monotonic() - t0, 2),
|
||||
upload_duration_s=None,
|
||||
result=None,
|
||||
error=str(e),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Pending Token Resolution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _resolve_pending_tokens(
|
||||
self,
|
||||
pending_tokens: dict[str, dict[str, str]],
|
||||
pending_uploads: list[tuple[str, DocumentEntry]],
|
||||
documents: dict[str, DocumentEntry],
|
||||
context: SessionContext,
|
||||
) -> None:
|
||||
"""Resume pending CU analyses using serializable continuation tokens.
|
||||
|
||||
When a file's CU analysis exceeds ``max_wait``, a continuation token
|
||||
(an opaque string from the Azure SDK) is saved in ``state`` instead of
|
||||
an ``asyncio.Task``. This keeps state fully serializable — it can be
|
||||
persisted to disk/storage by the framework.
|
||||
|
||||
On the next ``before_run()`` call, this method resumes each pending
|
||||
operation by passing the token back to ``begin_analyze()``. If the
|
||||
server-side operation has completed, the result is available
|
||||
immediately; otherwise the token is kept for the next turn.
|
||||
"""
|
||||
if not pending_tokens:
|
||||
return
|
||||
logger.info("Resolving %d pending analysis token(s).", len(pending_tokens))
|
||||
completed_keys: list[str] = []
|
||||
|
||||
for doc_key, token_info in pending_tokens.items():
|
||||
entry = documents.get(doc_key)
|
||||
if not entry:
|
||||
completed_keys.append(doc_key)
|
||||
continue
|
||||
|
||||
try:
|
||||
poller = await self._client.begin_analyze( # type: ignore[call-overload, reportUnknownVariableType]
|
||||
token_info["analyzer_id"],
|
||||
continuation_token=token_info["continuation_token"], # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
# Use wait_for to avoid blocking before_run indefinitely.
|
||||
# poller.done() always returns False for resumed pollers (stale
|
||||
# cached status), so we call poller.result() which polls the server.
|
||||
#
|
||||
# Timeout: at least 10s regardless of max_wait. The upload-turn
|
||||
# max_wait can be very short (e.g. 5s) for responsiveness, but
|
||||
# on resolution turns the resumed poller needs a network round-trip
|
||||
# to fetch the result. If the analysis is still running after 10s,
|
||||
# the token is kept and retried on the next turn.
|
||||
MIN_RESOLUTION_TIMEOUT = 10.0
|
||||
resolution_timeout = max(self.max_wait or MIN_RESOLUTION_TIMEOUT, MIN_RESOLUTION_TIMEOUT)
|
||||
try:
|
||||
result: AnalysisResult = await asyncio.wait_for(
|
||||
poller.result(), # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType]
|
||||
timeout=resolution_timeout,
|
||||
) # pyright: ignore[reportUnknownVariableType]
|
||||
except asyncio.TimeoutError:
|
||||
# Still running — update token and keep for next turn
|
||||
new_token: str = poller.continuation_token() # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
|
||||
token_info["continuation_token"] = new_token
|
||||
logger.info("Analysis for '%s' still running; keeping token for next turn.", doc_key)
|
||||
continue
|
||||
|
||||
completed_keys.append(doc_key)
|
||||
extracted = self._extract_sections(result) # pyright: ignore[reportUnknownArgumentType]
|
||||
entry["status"] = DocumentStatus.READY
|
||||
entry["analyzed_at"] = datetime.now(tz=timezone.utc).isoformat()
|
||||
entry["result"] = extracted
|
||||
entry["error"] = None
|
||||
logger.info("Background analysis of '%s' completed.", entry["filename"])
|
||||
|
||||
# Inject newly ready content
|
||||
if self.file_search:
|
||||
pending_uploads.append((doc_key, entry))
|
||||
else:
|
||||
context.extend_messages(
|
||||
self,
|
||||
[
|
||||
Message(role="user", contents=[format_result(entry["filename"], extracted)]),
|
||||
],
|
||||
)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[
|
||||
Message(
|
||||
role="user",
|
||||
contents=[
|
||||
f"Document '{entry['filename']}' analysis is now complete."
|
||||
+ (
|
||||
" The document is being indexed in the vector store and will become"
|
||||
" searchable via file_search shortly."
|
||||
if self.file_search
|
||||
else " The content is provided above."
|
||||
)
|
||||
],
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
completed_keys.append(doc_key)
|
||||
logger.warning("Background analysis of '%s' failed: %s", entry.get("filename", doc_key), e)
|
||||
entry["status"] = DocumentStatus.FAILED
|
||||
entry["analyzed_at"] = datetime.now(tz=timezone.utc).isoformat()
|
||||
entry["error"] = str(e)
|
||||
context.extend_messages(
|
||||
self.source_id,
|
||||
[Message(role="user", contents=[f"Document '{entry['filename']}' analysis failed: {e}"])],
|
||||
)
|
||||
|
||||
for key in completed_keys:
|
||||
del pending_tokens[key]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Output Extraction & Formatting (delegates to _extraction module)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _extract_sections(self, result: AnalysisResult) -> dict[str, object]:
|
||||
return extract_sections(result, self.output_sections)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tool Registration
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _register_tools(
|
||||
self,
|
||||
documents: dict[str, DocumentEntry],
|
||||
context: SessionContext,
|
||||
) -> None:
|
||||
"""Register document tools on the context.
|
||||
|
||||
Only ``list_documents`` is registered — the full document content is
|
||||
already injected into conversation history on the upload turn, so a
|
||||
separate retrieval tool is not needed.
|
||||
"""
|
||||
context.extend_tools(
|
||||
self.source_id,
|
||||
[self._make_list_documents_tool(documents)],
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _make_list_documents_tool(documents: dict[str, DocumentEntry]) -> FunctionTool:
|
||||
"""Create a tool that lists all tracked documents with their status."""
|
||||
docs_ref = documents
|
||||
|
||||
def list_documents() -> str:
|
||||
"""List all documents that have been uploaded and their analysis status."""
|
||||
entries: list[dict[str, object]] = []
|
||||
for name, entry in docs_ref.items():
|
||||
entries.append({
|
||||
"name": name,
|
||||
"status": entry["status"],
|
||||
"media_type": entry["media_type"],
|
||||
"analyzed_at": entry["analyzed_at"],
|
||||
"analysis_duration_s": entry["analysis_duration_s"],
|
||||
"upload_duration_s": entry["upload_duration_s"],
|
||||
})
|
||||
return json.dumps(entries, indent=2, default=str)
|
||||
|
||||
return FunctionTool(
|
||||
name="list_documents",
|
||||
description=(
|
||||
"List all documents that have been uploaded in this session "
|
||||
"with their analysis status (analyzing, uploading, ready, or failed)."
|
||||
),
|
||||
func=list_documents,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# file_search Vector Store Integration
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _upload_to_vector_store(
|
||||
self,
|
||||
doc_key: str,
|
||||
entry: DocumentEntry,
|
||||
*,
|
||||
timeout: float | None = None,
|
||||
state: dict[str, Any] | None = None,
|
||||
) -> bool:
|
||||
"""Upload CU-extracted markdown to the caller's vector store.
|
||||
|
||||
Delegates to the configured ``FileSearchBackend`` (OpenAI, Foundry,
|
||||
or a custom implementation). The upload includes file upload **and**
|
||||
vector store indexing (embedding + ingestion) — ``create_and_poll``
|
||||
waits for the index to be fully ready before returning.
|
||||
|
||||
Args:
|
||||
doc_key: Document identifier.
|
||||
entry: The document entry with extracted results.
|
||||
timeout: Max seconds to wait for upload + indexing. ``None`` waits
|
||||
indefinitely. On timeout the upload is deferred to the
|
||||
per-session ``_pending_uploads`` queue for the next
|
||||
``before_run()`` call.
|
||||
state: Per-session state dict for tracking uploaded file IDs and
|
||||
pending uploads.
|
||||
|
||||
Returns:
|
||||
True if the upload succeeded, False otherwise.
|
||||
"""
|
||||
if not self.file_search:
|
||||
return False
|
||||
|
||||
result = entry.get("result")
|
||||
if not result:
|
||||
return False
|
||||
|
||||
# Upload the full formatted content (markdown + fields + segments),
|
||||
# not just raw markdown — consistent with what non-file_search mode injects.
|
||||
formatted = format_result(entry["filename"], result)
|
||||
if not formatted:
|
||||
return False
|
||||
|
||||
entry["status"] = DocumentStatus.UPLOADING
|
||||
t0 = time.monotonic()
|
||||
|
||||
try:
|
||||
upload_coro = self.file_search.backend.upload_file(
|
||||
self.file_search.vector_store_id, f"{doc_key}.md", formatted.encode("utf-8")
|
||||
)
|
||||
file_id = await asyncio.wait_for(upload_coro, timeout=timeout)
|
||||
upload_duration = round(time.monotonic() - t0, 2)
|
||||
# Track in per-session state and global list (for close() cleanup)
|
||||
if state is not None:
|
||||
state.setdefault("_uploaded_file_ids", []).append(file_id)
|
||||
self._all_uploaded_file_ids.append(file_id)
|
||||
entry["status"] = DocumentStatus.READY
|
||||
entry["upload_duration_s"] = upload_duration
|
||||
logger.info("Uploaded '%s' to vector store in %.1fs (%s bytes).", doc_key, upload_duration, len(formatted))
|
||||
return True
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
logger.info("Vector store upload for '%s' timed out; deferring to background.", doc_key)
|
||||
entry["status"] = DocumentStatus.UPLOADING
|
||||
if state is not None:
|
||||
state.setdefault("_pending_uploads", []).append((doc_key, entry))
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.warning("Failed to upload '%s' to vector store: %s", doc_key, e)
|
||||
entry["status"] = DocumentStatus.FAILED
|
||||
entry["upload_duration_s"] = round(time.monotonic() - t0, 2)
|
||||
entry["error"] = f"Vector store upload failed: {e}"
|
||||
return False
|
||||
|
||||
async def _cleanup_uploaded_files(self) -> None:
|
||||
"""Delete files uploaded by this provider via the configured backend.
|
||||
|
||||
The vector store itself is caller-managed and is not deleted here.
|
||||
"""
|
||||
if not self.file_search:
|
||||
return
|
||||
|
||||
backend = self.file_search.backend
|
||||
|
||||
try:
|
||||
for file_id in self._all_uploaded_file_ids:
|
||||
await backend.delete_file(file_id)
|
||||
self._all_uploaded_file_ids.clear()
|
||||
|
||||
except Exception as e:
|
||||
logger.warning("Failed to clean up uploaded files: %s", e)
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""File detection utilities for Azure Content Understanding context provider.
|
||||
|
||||
Functions for scanning input messages, sniffing MIME types, deriving
|
||||
document keys, and extracting binary data from content items.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import logging
|
||||
import mimetypes
|
||||
import re
|
||||
import uuid
|
||||
|
||||
import filetype
|
||||
from agent_framework import Content, SessionContext
|
||||
|
||||
logger = logging.getLogger("agent_framework.azure_contentunderstanding")
|
||||
|
||||
# MIME types used to match against the resolved media type for routing files to CU analysis.
|
||||
# The media type may be provided via Content.media_type or inferred (e.g., via sniffing or filename)
|
||||
# when missing or generic (such as application/octet-stream). Only files whose resolved media type is
|
||||
# in this set will be processed; others are skipped.
|
||||
#
|
||||
# Supported input file types:
|
||||
# https://learn.microsoft.com/azure/ai-services/content-understanding/service-limits#input-file-limits
|
||||
SUPPORTED_MEDIA_TYPES: frozenset[str] = frozenset({
|
||||
# Documents and images
|
||||
"application/pdf",
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/tiff",
|
||||
"image/bmp",
|
||||
"image/heif",
|
||||
"image/heic",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
# Text
|
||||
"text/plain",
|
||||
"text/html",
|
||||
"text/markdown",
|
||||
"text/rtf",
|
||||
"text/xml",
|
||||
"application/xml",
|
||||
"message/rfc822",
|
||||
"application/vnd.ms-outlook",
|
||||
# Audio
|
||||
"audio/wav",
|
||||
"audio/mpeg",
|
||||
"audio/mp3",
|
||||
"audio/mp4",
|
||||
"audio/m4a",
|
||||
"audio/flac",
|
||||
"audio/ogg",
|
||||
"audio/opus",
|
||||
"audio/webm",
|
||||
"audio/x-ms-wma",
|
||||
"audio/aac",
|
||||
"audio/amr",
|
||||
"audio/3gpp",
|
||||
# Video
|
||||
"video/mp4",
|
||||
"video/quicktime",
|
||||
"video/x-msvideo",
|
||||
"video/webm",
|
||||
"video/x-flv",
|
||||
"video/x-ms-wmv",
|
||||
"video/x-ms-asf",
|
||||
"video/x-matroska",
|
||||
})
|
||||
|
||||
# Mapping from filetype's MIME output to our canonical SUPPORTED_MEDIA_TYPES values.
|
||||
# filetype uses some x-prefixed variants that differ from our set.
|
||||
MIME_ALIASES: dict[str, str] = {
|
||||
"audio/x-wav": "audio/wav",
|
||||
"audio/x-flac": "audio/flac",
|
||||
"video/x-m4v": "video/mp4",
|
||||
}
|
||||
|
||||
|
||||
def detect_and_strip_files(
|
||||
context: SessionContext,
|
||||
) -> list[tuple[str, Content, bytes | None]]:
|
||||
"""Scan input messages for supported file content and prepare for CU analysis.
|
||||
|
||||
Scans for type ``data`` or ``uri`` content supported by Azure Content
|
||||
Understanding, strips them from messages to prevent raw binary being sent
|
||||
to the LLM, and returns metadata for CU analysis.
|
||||
|
||||
Detected files are tracked via ``doc_key`` (derived from filename, URL,
|
||||
or UUID) and their analysis status is managed in session state.
|
||||
|
||||
When the upstream MIME type is unreliable (``application/octet-stream``
|
||||
or missing), binary content sniffing via ``filetype`` is used to
|
||||
determine the real media type, with ``mimetypes.guess_type`` as a
|
||||
filename-based fallback.
|
||||
|
||||
Returns:
|
||||
List of (doc_key, content_item, binary_data) tuples for files to analyze.
|
||||
"""
|
||||
results: list[tuple[str, Content, bytes | None]] = []
|
||||
strip_ids: set[int] = set()
|
||||
|
||||
for msg in context.input_messages:
|
||||
for c in msg.contents:
|
||||
if c.type not in ("data", "uri"):
|
||||
continue
|
||||
|
||||
media_type = c.media_type
|
||||
# Fast path: already a known supported type
|
||||
if media_type and media_type in SUPPORTED_MEDIA_TYPES:
|
||||
binary_data = extract_binary(c)
|
||||
results.append((derive_doc_key(c), c, binary_data))
|
||||
strip_ids.add(id(c))
|
||||
continue
|
||||
|
||||
# Slow path: unreliable MIME — sniff binary content
|
||||
if (not media_type) or (media_type == "application/octet-stream"):
|
||||
binary_data = extract_binary(c)
|
||||
resolved = sniff_media_type(binary_data, c)
|
||||
if resolved and (resolved in SUPPORTED_MEDIA_TYPES):
|
||||
c.media_type = resolved
|
||||
results.append((derive_doc_key(c), c, binary_data))
|
||||
strip_ids.add(id(c))
|
||||
|
||||
# Strip detected files from input so raw binary isn't sent to LLM
|
||||
msg.contents = [c for c in msg.contents if id(c) not in strip_ids]
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def sniff_media_type(binary_data: bytes | None, content: Content) -> str | None:
|
||||
"""Sniff the actual MIME type from binary data, with filename fallback.
|
||||
|
||||
Uses ``filetype`` (magic-bytes) first, then ``mimetypes.guess_type``
|
||||
on the filename. Normalizes filetype's variant MIME values (e.g.
|
||||
``audio/x-wav`` -> ``audio/wav``) via ``MIME_ALIASES``.
|
||||
"""
|
||||
# 1. Binary sniffing via filetype (needs only first 261 bytes)
|
||||
if binary_data:
|
||||
kind = filetype.guess(binary_data[:262]) # type: ignore[reportUnknownMemberType]
|
||||
if kind:
|
||||
mime: str = kind.mime # type: ignore[reportUnknownMemberType]
|
||||
return MIME_ALIASES.get(mime, mime)
|
||||
|
||||
# 2. Filename extension fallback — try additional_properties first,
|
||||
# then extract basename from external URL path
|
||||
filename: str | None = None
|
||||
if content.additional_properties:
|
||||
filename = content.additional_properties.get("filename")
|
||||
if not filename and content.uri and not content.uri.startswith("data:"):
|
||||
# Extract basename from URL path (e.g. "https://example.com/report.pdf?v=1" -> "report.pdf")
|
||||
filename = content.uri.split("?")[0].split("#")[0].rsplit("/", 1)[-1]
|
||||
if filename:
|
||||
guessed, _ = mimetypes.guess_type(filename) # uses file extension to guess MIME type
|
||||
if guessed:
|
||||
return MIME_ALIASES.get(guessed, guessed)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def is_supported_content(content: Content) -> bool:
|
||||
"""Check if a content item is a supported file type for CU analysis."""
|
||||
if content.type not in ("data", "uri"):
|
||||
return False
|
||||
media_type = content.media_type
|
||||
if not media_type:
|
||||
return False
|
||||
return media_type in SUPPORTED_MEDIA_TYPES
|
||||
|
||||
|
||||
def sanitize_doc_key(raw: str) -> str:
|
||||
"""Sanitize a document key to prevent prompt injection.
|
||||
|
||||
Removes control characters (newlines, tabs, etc.), collapses
|
||||
whitespace, strips surrounding whitespace, and caps length at
|
||||
255 characters.
|
||||
"""
|
||||
# Remove control characters (C0/C1 controls, including \n, \r, \t)
|
||||
cleaned = re.sub(r"[\x00-\x1f\x7f-\x9f]", "", raw)
|
||||
# Collapse whitespace
|
||||
cleaned = " ".join(cleaned.split())
|
||||
# Cap length
|
||||
return cleaned[:255] if cleaned else f"doc_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
def derive_doc_key(content: Content) -> str:
|
||||
"""Derive a unique document key from content metadata.
|
||||
|
||||
The key is used to track documents in session state. Duplicate keys
|
||||
within a session are rejected (not re-analyzed) to prevent orphaned
|
||||
vector store entries.
|
||||
|
||||
The returned key is sanitized to prevent prompt injection via
|
||||
crafted filenames (control characters removed, length capped).
|
||||
|
||||
Priority: filename > URL basename > generated UUID.
|
||||
"""
|
||||
# 1. Filename from additional_properties
|
||||
if content.additional_properties:
|
||||
filename = content.additional_properties.get("filename")
|
||||
if filename and isinstance(filename, str):
|
||||
return sanitize_doc_key(filename)
|
||||
|
||||
# 2. URL path basename for external URIs (e.g. "https://example.com/report.pdf" -> "report.pdf")
|
||||
if content.type == "uri" and content.uri and not content.uri.startswith("data:"):
|
||||
path = content.uri.split("?")[0].split("#")[0] # strip query params and fragments
|
||||
# rstrip("/") handles trailing slashes (e.g. ".../files/" -> ".../files")
|
||||
# rsplit("/", 1)[-1] splits from the right once to get the last path segment
|
||||
basename = path.rstrip("/").rsplit("/", 1)[-1]
|
||||
if basename:
|
||||
return sanitize_doc_key(basename)
|
||||
|
||||
# 3. Fallback: generate a unique ID for anonymous uploads (no filename, no URL)
|
||||
return f"doc_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
def extract_binary(content: Content) -> bytes | None:
|
||||
"""Extract binary data from a data URI content item.
|
||||
|
||||
Only handles ``data:`` URIs (base64-encoded). Returns ``None`` for
|
||||
external URLs -- those are passed directly to CU via ``begin_analyze``.
|
||||
"""
|
||||
if content.uri and content.uri.startswith("data:"):
|
||||
try:
|
||||
_, data_part = content.uri.split(",", 1)
|
||||
return base64.b64decode(data_part)
|
||||
except Exception:
|
||||
logger.warning("Failed to decode base64 data URI")
|
||||
return None
|
||||
return None
|
||||
+297
@@ -0,0 +1,297 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Output extraction and formatting for Azure Content Understanding results.
|
||||
|
||||
Converts CU ``AnalysisResult`` objects into plain Python dicts suitable
|
||||
for LLM consumption, and formats them as human-readable text.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, cast
|
||||
|
||||
from azure.ai.contentunderstanding.models import AnalysisResult
|
||||
|
||||
from ._models import AnalysisSection
|
||||
|
||||
|
||||
def extract_sections(
|
||||
result: AnalysisResult,
|
||||
output_sections: list[AnalysisSection],
|
||||
) -> dict[str, object]:
|
||||
"""Extract configured sections from a CU analysis result.
|
||||
|
||||
For single-segment results (documents, images, short audio), returns a flat
|
||||
dict with ``markdown`` and ``fields`` at the top level.
|
||||
|
||||
For multi-segment results (e.g. video split into scenes), fields are kept
|
||||
with their respective segments in a ``segments`` list so the LLM can see
|
||||
which fields belong to which part of the content:
|
||||
- ``segments``: list of per-segment dicts with ``markdown``, ``fields``,
|
||||
``start_time_s``, and ``end_time_s``
|
||||
- ``markdown``: still concatenated at top level for file_search uploads
|
||||
- ``duration_seconds``: computed from the global time span
|
||||
- ``kind`` / ``resolution``: taken from the first segment
|
||||
"""
|
||||
extracted: dict[str, object] = {}
|
||||
contents = result.contents
|
||||
if not contents:
|
||||
return extracted
|
||||
|
||||
# --- Warnings from the CU service (ODataV4Format with code/message/target) ---
|
||||
if result.warnings:
|
||||
warnings_out: list[dict[str, str]] = []
|
||||
for w in result.warnings:
|
||||
entry: dict[str, str] = {}
|
||||
code = getattr(w, "code", None)
|
||||
if code:
|
||||
entry["code"] = code
|
||||
msg = getattr(w, "message", None)
|
||||
entry["message"] = msg if msg else str(w)
|
||||
target = getattr(w, "target", None)
|
||||
if target:
|
||||
entry["target"] = target
|
||||
warnings_out.append(entry)
|
||||
extracted["warnings"] = warnings_out
|
||||
|
||||
# --- Media metadata (from first segment) ---
|
||||
first = contents[0]
|
||||
kind = getattr(first, "kind", None)
|
||||
if kind:
|
||||
extracted["kind"] = kind
|
||||
width = getattr(first, "width", None)
|
||||
height = getattr(first, "height", None)
|
||||
if width and height:
|
||||
extracted["resolution"] = f"{width}x{height}"
|
||||
|
||||
# Compute total duration from the global time span of all segments.
|
||||
global_start: int | None = None
|
||||
global_end: int | None = None
|
||||
for content in contents:
|
||||
s = getattr(content, "start_time_ms", None)
|
||||
if s is None:
|
||||
s = getattr(content, "startTimeMs", None)
|
||||
e = getattr(content, "end_time_ms", None)
|
||||
if e is None:
|
||||
e = getattr(content, "endTimeMs", None)
|
||||
if s is not None:
|
||||
global_start = s if global_start is None else min(global_start, s)
|
||||
if e is not None:
|
||||
global_end = e if global_end is None else max(global_end, e)
|
||||
if global_start is not None and global_end is not None:
|
||||
extracted["duration_seconds"] = round((global_end - global_start) / 1000, 1)
|
||||
|
||||
is_multi_segment = len(contents) > 1
|
||||
|
||||
# --- Single-segment: flat output (documents, images, short audio) ---
|
||||
if not is_multi_segment:
|
||||
if "markdown" in output_sections and contents[0].markdown:
|
||||
extracted["markdown"] = contents[0].markdown
|
||||
if "fields" in output_sections and contents[0].fields:
|
||||
fields: dict[str, object] = {}
|
||||
for name, field in contents[0].fields.items():
|
||||
entry_dict: dict[str, object] = {
|
||||
"type": getattr(field, "type", None),
|
||||
"value": extract_field_value(field),
|
||||
}
|
||||
confidence = getattr(field, "confidence", None)
|
||||
if confidence is not None:
|
||||
entry_dict["confidence"] = confidence
|
||||
fields[name] = entry_dict
|
||||
if fields:
|
||||
extracted["fields"] = fields
|
||||
# Content-level category (e.g. from classifier analyzers)
|
||||
category = getattr(contents[0], "category", None)
|
||||
if category:
|
||||
extracted["category"] = category
|
||||
return extracted
|
||||
|
||||
# --- Multi-segment: per-segment output (video scenes, long audio) ---
|
||||
# Each segment keeps its own markdown + fields together so the LLM can
|
||||
# see which fields (e.g. Summary) belong to which part of the content.
|
||||
segments_out: list[dict[str, object]] = []
|
||||
md_parts: list[str] = [] # also collect for top-level concatenated markdown
|
||||
|
||||
for content in contents:
|
||||
seg: dict[str, object] = {}
|
||||
|
||||
# Time range for this segment
|
||||
s = getattr(content, "start_time_ms", None)
|
||||
if s is None:
|
||||
s = getattr(content, "startTimeMs", None)
|
||||
e = getattr(content, "end_time_ms", None)
|
||||
if e is None:
|
||||
e = getattr(content, "endTimeMs", None)
|
||||
if s is not None:
|
||||
seg["start_time_s"] = round(s / 1000, 1)
|
||||
if e is not None:
|
||||
seg["end_time_s"] = round(e / 1000, 1)
|
||||
|
||||
# Per-segment markdown
|
||||
if "markdown" in output_sections and content.markdown:
|
||||
seg["markdown"] = content.markdown
|
||||
md_parts.append(content.markdown)
|
||||
|
||||
# Per-segment fields
|
||||
if "fields" in output_sections and content.fields:
|
||||
seg_fields: dict[str, object] = {}
|
||||
for name, field in content.fields.items():
|
||||
seg_entry: dict[str, object] = {
|
||||
"type": getattr(field, "type", None),
|
||||
"value": extract_field_value(field),
|
||||
}
|
||||
confidence = getattr(field, "confidence", None)
|
||||
if confidence is not None:
|
||||
seg_entry["confidence"] = confidence
|
||||
seg_fields[name] = seg_entry
|
||||
if seg_fields:
|
||||
seg["fields"] = seg_fields
|
||||
|
||||
# Per-segment category (e.g. from classifier analyzers)
|
||||
category = getattr(content, "category", None)
|
||||
if category:
|
||||
seg["category"] = category
|
||||
|
||||
segments_out.append(seg)
|
||||
|
||||
extracted["segments"] = segments_out
|
||||
|
||||
# Top-level concatenated markdown (used by file_search for vector store upload)
|
||||
if md_parts:
|
||||
extracted["markdown"] = "\n\n---\n\n".join(md_parts)
|
||||
|
||||
return extracted
|
||||
|
||||
|
||||
def extract_field_value(field: Any) -> object:
|
||||
"""Extract the plain Python value from a CU ``ContentField``.
|
||||
|
||||
Uses the SDK's ``.value`` convenience property, which dynamically
|
||||
reads the correct ``value_*`` attribute for each field type.
|
||||
Object and array types are recursively flattened so that the
|
||||
output contains only plain Python primitives (str, int, float,
|
||||
date, dict, list) -- no SDK model objects or raw wire format
|
||||
(``valueNumber``, ``spans``, ``source``, etc.).
|
||||
"""
|
||||
field_type = getattr(field, "type", None)
|
||||
raw = getattr(field, "value", None)
|
||||
|
||||
# Object fields -> recursively resolve nested sub-fields
|
||||
if field_type == "object" and raw is not None and isinstance(raw, dict):
|
||||
return {str(k): flatten_field(v) for k, v in cast(dict[str, Any], raw).items()}
|
||||
|
||||
# Array fields -> list of flattened items (each with value + optional confidence)
|
||||
if field_type == "array" and raw is not None and isinstance(raw, list):
|
||||
return [flatten_field(item) for item in cast(list[Any], raw)]
|
||||
|
||||
# Scalar fields (string, number, date, etc.) -- .value returns native Python type
|
||||
return raw
|
||||
|
||||
|
||||
def flatten_field(field: Any) -> object:
|
||||
"""Flatten a CU ``ContentField`` into a ``{type, value, confidence}`` dict.
|
||||
|
||||
Used for sub-fields inside object and array types to preserve
|
||||
per-field confidence scores. Confidence is omitted when ``None``
|
||||
to reduce token usage.
|
||||
"""
|
||||
field_type = getattr(field, "type", None)
|
||||
value = extract_field_value(field)
|
||||
confidence = getattr(field, "confidence", None)
|
||||
|
||||
result: dict[str, object] = {"type": field_type, "value": value}
|
||||
if confidence is not None:
|
||||
result["confidence"] = confidence
|
||||
return result
|
||||
|
||||
|
||||
def format_result(filename: str, result: dict[str, object]) -> str:
|
||||
"""Format extracted CU result for LLM consumption.
|
||||
|
||||
For multi-segment results (video/audio with ``segments``), each segment's
|
||||
markdown and fields are grouped together so the LLM can see which fields
|
||||
belong to which part of the content.
|
||||
"""
|
||||
kind = result.get("kind")
|
||||
is_video = kind == "audioVisual"
|
||||
is_audio = kind == "audio"
|
||||
|
||||
# Header -- media-aware label
|
||||
if is_video:
|
||||
label = "Video analysis"
|
||||
elif is_audio:
|
||||
label = "Audio analysis"
|
||||
else:
|
||||
label = "Document analysis"
|
||||
parts: list[str] = [f'{label} of "{filename}":']
|
||||
|
||||
# Media metadata line (duration, resolution)
|
||||
meta_items: list[str] = []
|
||||
duration = result.get("duration_seconds")
|
||||
if duration is not None:
|
||||
mins, secs = divmod(int(duration), 60) # type: ignore[call-overload]
|
||||
meta_items.append(f"Duration: {mins}:{secs:02d}")
|
||||
resolution = result.get("resolution")
|
||||
if resolution:
|
||||
meta_items.append(f"Resolution: {resolution}")
|
||||
if meta_items:
|
||||
parts.append(" | ".join(meta_items))
|
||||
|
||||
# --- Multi-segment: format each segment with its own content + fields ---
|
||||
raw_segments = result.get("segments")
|
||||
segments: list[dict[str, object]] = (
|
||||
cast(list[dict[str, object]], raw_segments) if isinstance(raw_segments, list) else []
|
||||
)
|
||||
if segments:
|
||||
for i, seg in enumerate(segments):
|
||||
# Segment header with time range
|
||||
start = seg.get("start_time_s")
|
||||
end = seg.get("end_time_s")
|
||||
if start is not None and end is not None:
|
||||
s_min, s_sec = divmod(int(start), 60) # type: ignore[call-overload]
|
||||
e_min, e_sec = divmod(int(end), 60) # type: ignore[call-overload]
|
||||
parts.append(f"\n### Segment {i + 1} ({s_min}:{s_sec:02d} - {e_min}:{e_sec:02d})")
|
||||
else:
|
||||
parts.append(f"\n### Segment {i + 1}")
|
||||
|
||||
# Segment markdown
|
||||
seg_md = seg.get("markdown")
|
||||
if seg_md:
|
||||
parts.append(f"\n```markdown\n{seg_md}\n```")
|
||||
|
||||
# Segment fields
|
||||
seg_fields = seg.get("fields")
|
||||
if isinstance(seg_fields, dict) and seg_fields:
|
||||
fields_json = json.dumps(seg_fields, indent=2, default=str)
|
||||
parts.append(f"\n**Fields:**\n```json\n{fields_json}\n```")
|
||||
|
||||
return "\n".join(parts)
|
||||
|
||||
# --- Single-segment: flat format ---
|
||||
fields_raw = result.get("fields")
|
||||
fields: dict[str, object] = cast(dict[str, object], fields_raw) if isinstance(fields_raw, dict) else {}
|
||||
|
||||
# For audio: promote Summary field as prose before markdown
|
||||
if is_audio and fields:
|
||||
summary_field = fields.get("Summary")
|
||||
if isinstance(summary_field, dict):
|
||||
sf = cast(dict[str, object], summary_field)
|
||||
if sf.get("value"):
|
||||
parts.append(f"\n## Summary\n\n{sf['value']}")
|
||||
|
||||
# Markdown content
|
||||
markdown = result.get("markdown")
|
||||
if markdown:
|
||||
parts.append(f"\n## Content\n\n```markdown\n{markdown}\n```")
|
||||
|
||||
# Fields section
|
||||
if fields:
|
||||
remaining = dict(fields)
|
||||
if is_audio:
|
||||
remaining = {k: v for k, v in remaining.items() if k != "Summary"}
|
||||
if remaining:
|
||||
fields_json = json.dumps(remaining, indent=2, default=str)
|
||||
parts.append(f"\n## Extracted Fields\n\n```json\n{fields_json}\n```")
|
||||
|
||||
return "\n".join(parts)
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""File search backend abstraction for vector store file operations.
|
||||
|
||||
Provides a unified interface for uploading CU-extracted content to
|
||||
vector stores across different LLM clients. Two implementations:
|
||||
|
||||
- ``OpenAIFileSearchBackend`` — for ``OpenAIChatClient`` (Responses API)
|
||||
- ``FoundryFileSearchBackend`` — for ``FoundryChatClient`` (Responses API via Azure)
|
||||
|
||||
Both share the same OpenAI-compatible vector store file API but differ
|
||||
in the file upload ``purpose`` value.
|
||||
|
||||
Vector store creation, tool construction, and lifecycle management are
|
||||
the caller's responsibility — the backend only handles file upload/delete.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
|
||||
class FileSearchBackend(ABC):
|
||||
"""Abstract interface for vector store file operations.
|
||||
|
||||
Implementations handle the differences between OpenAI and Foundry
|
||||
file upload APIs (e.g., different ``purpose`` values).
|
||||
|
||||
Vector store creation, deletion, and ``file_search`` tool construction
|
||||
are **not** part of this interface — those are managed by the caller.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
async def upload_file(self, vector_store_id: str, filename: str, content: bytes) -> str:
|
||||
"""Upload a file to a vector store and return the file ID."""
|
||||
|
||||
@abstractmethod
|
||||
async def delete_file(self, file_id: str) -> None:
|
||||
"""Delete a previously uploaded file by ID."""
|
||||
|
||||
|
||||
class _OpenAICompatBackend(FileSearchBackend):
|
||||
"""Shared base for OpenAI-compatible file upload backends.
|
||||
|
||||
Both OpenAI and Foundry use the same ``client.files.*`` and
|
||||
``client.vector_stores.files.*`` API surface. Subclasses only
|
||||
override the file upload ``purpose``.
|
||||
"""
|
||||
|
||||
_FILE_PURPOSE: str # Subclasses must set this
|
||||
|
||||
def __init__(self, client: Any) -> None:
|
||||
self._client = client
|
||||
|
||||
async def upload_file(self, vector_store_id: str, filename: str, content: bytes) -> str:
|
||||
uploaded = await self._client.files.create(
|
||||
file=(filename, io.BytesIO(content)),
|
||||
purpose=self._FILE_PURPOSE,
|
||||
)
|
||||
# Use create_and_poll to wait for indexing to complete before returning.
|
||||
# Without this, file_search queries may return no results immediately
|
||||
# after upload because the vector store index isn't ready yet.
|
||||
await self._client.vector_stores.files.create_and_poll(
|
||||
vector_store_id=vector_store_id,
|
||||
file_id=uploaded.id,
|
||||
)
|
||||
return uploaded.id # type: ignore[no-any-return]
|
||||
|
||||
async def delete_file(self, file_id: str) -> None:
|
||||
await self._client.files.delete(file_id)
|
||||
|
||||
|
||||
class OpenAIFileSearchBackend(_OpenAICompatBackend):
|
||||
"""File search backend for OpenAI Responses API.
|
||||
|
||||
Use with ``OpenAIChatClient`` or ``AzureOpenAIResponsesClient``.
|
||||
Requires an ``AsyncOpenAI`` or ``AsyncAzureOpenAI`` client.
|
||||
|
||||
Args:
|
||||
client: An async OpenAI client (``AsyncOpenAI`` or ``AsyncAzureOpenAI``)
|
||||
that supports ``client.files.*`` and ``client.vector_stores.*`` APIs.
|
||||
"""
|
||||
|
||||
_FILE_PURPOSE = "user_data"
|
||||
|
||||
|
||||
class FoundryFileSearchBackend(_OpenAICompatBackend):
|
||||
"""File search backend for Azure AI Foundry.
|
||||
|
||||
Use with ``FoundryChatClient``. Requires the OpenAI-compatible client
|
||||
obtained from ``FoundryChatClient.client`` (i.e.,
|
||||
``project_client.get_openai_client()``).
|
||||
|
||||
Args:
|
||||
client: The OpenAI-compatible async client from a ``FoundryChatClient``
|
||||
(access via ``foundry_client.client``).
|
||||
"""
|
||||
|
||||
_FILE_PURPOSE = "assistants"
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any, Literal, TypedDict
|
||||
|
||||
from ._file_search import FileSearchBackend, FoundryFileSearchBackend, OpenAIFileSearchBackend
|
||||
|
||||
|
||||
class DocumentStatus(str, Enum):
|
||||
"""Analysis lifecycle state of a tracked document."""
|
||||
|
||||
ANALYZING = "analyzing"
|
||||
"""CU analysis is in progress (deferred to background)."""
|
||||
|
||||
UPLOADING = "uploading"
|
||||
"""Analysis complete; vector store upload + indexing is in progress."""
|
||||
|
||||
READY = "ready"
|
||||
"""Analysis (and upload, if applicable) completed successfully."""
|
||||
|
||||
FAILED = "failed"
|
||||
"""Analysis or upload failed."""
|
||||
|
||||
|
||||
AnalysisSection = Literal["markdown", "fields"]
|
||||
"""Which sections of the CU output to pass to the LLM.
|
||||
|
||||
- ``"markdown"``: Full document text with tables as HTML, reading order preserved.
|
||||
- ``"fields"``: Extracted typed fields with confidence scores (when available).
|
||||
"""
|
||||
|
||||
|
||||
class DocumentEntry(TypedDict):
|
||||
"""Tracks the analysis state of a single document in session state."""
|
||||
|
||||
status: DocumentStatus
|
||||
filename: str
|
||||
media_type: str
|
||||
analyzer_id: str
|
||||
analyzed_at: str | None
|
||||
analysis_duration_s: float | None
|
||||
upload_duration_s: float | None
|
||||
result: dict[str, object] | None
|
||||
error: str | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileSearchConfig:
|
||||
"""Configuration for uploading CU-extracted content to an existing vector store.
|
||||
|
||||
When provided to ``ContentUnderstandingContextProvider``, analyzed document
|
||||
markdown is automatically uploaded to the specified vector store and the
|
||||
given ``file_search`` tool is registered on the context. This enables
|
||||
token-efficient RAG retrieval on follow-up turns for large documents.
|
||||
|
||||
The caller is responsible for creating and managing the vector store and
|
||||
the ``file_search`` tool. Use :meth:`from_openai` or :meth:`from_foundry`
|
||||
factory methods for convenience.
|
||||
|
||||
Args:
|
||||
backend: A ``FileSearchBackend`` that handles file upload/delete
|
||||
operations for the target vector store.
|
||||
vector_store_id: The ID of a pre-existing vector store to upload to.
|
||||
file_search_tool: A ``file_search`` tool object created via the LLM
|
||||
client's ``get_file_search_tool()`` factory method. This is
|
||||
registered on the context via ``extend_tools`` so the LLM can
|
||||
retrieve uploaded content.
|
||||
"""
|
||||
|
||||
backend: FileSearchBackend
|
||||
vector_store_id: str
|
||||
file_search_tool: Any
|
||||
|
||||
@staticmethod
|
||||
def from_openai(
|
||||
client: Any,
|
||||
*,
|
||||
vector_store_id: str,
|
||||
file_search_tool: Any,
|
||||
) -> FileSearchConfig:
|
||||
"""Create a config for OpenAI Responses API (``OpenAIChatClient``).
|
||||
|
||||
Args:
|
||||
client: An ``AsyncOpenAI`` or ``AsyncAzureOpenAI`` client.
|
||||
vector_store_id: The ID of the vector store to upload to.
|
||||
file_search_tool: Tool from ``OpenAIChatClient.get_file_search_tool()``.
|
||||
"""
|
||||
return FileSearchConfig(
|
||||
backend=OpenAIFileSearchBackend(client),
|
||||
vector_store_id=vector_store_id,
|
||||
file_search_tool=file_search_tool,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def from_foundry(
|
||||
client: Any,
|
||||
*,
|
||||
vector_store_id: str,
|
||||
file_search_tool: Any,
|
||||
) -> FileSearchConfig:
|
||||
"""Create a config for Azure AI Foundry (``FoundryChatClient``).
|
||||
|
||||
Args:
|
||||
client: The OpenAI-compatible client from ``FoundryChatClient.client``.
|
||||
vector_store_id: The ID of the vector store to upload to.
|
||||
file_search_tool: Tool from ``FoundryChatClient.get_file_search_tool()``.
|
||||
"""
|
||||
return FileSearchConfig(
|
||||
backend=FoundryFileSearchBackend(client),
|
||||
vector_store_id=vector_store_id,
|
||||
file_search_tool=file_search_tool,
|
||||
)
|
||||
Reference in New Issue
Block a user