1 Commits

• Python: feat(python): Add embedding abstractions and OpenAI implementation (Phase 1) (#4153)

  * feat(python): Add embedding abstractions and OpenAI implementation (Phase 1)
  
  This PR contains two parts:
  
  1. **Overall migration plan** for porting vector stores and embeddings from
     Semantic Kernel to Agent Framework (docs/features/vector-stores-and-embeddings/README.md)
     covering all 10 phases from core abstractions through connectors and TextSearch.
  
  2. **Phase 1 implementation** — core embedding abstractions and OpenAI/Azure OpenAI
     embedding clients:
  
     Core types (_types.py):
     - EmbeddingGenerationOptions TypedDict (total=False)
     - Embedding[EmbeddingT] generic class with model_id, dimensions, created_at
     - GeneratedEmbeddings[EmbeddingT, EmbeddingOptionsT] list container with options, usage
     - EmbeddingInputT (default str) and EmbeddingT (default list[float]) TypeVars
  
     Protocol + base class (_clients.py):
     - SupportsGetEmbeddings protocol — Generic[EmbeddingInputT, EmbeddingT, OptionsContraT]
     - BaseEmbeddingClient ABC — Generic[EmbeddingInputT, EmbeddingT, OptionsCoT]
  
     Telemetry (observability.py):
     - EmbeddingTelemetryLayer with gen_ai.operation.name = "embeddings"
  
     OpenAI implementation (openai/_embedding_client.py):
     - RawOpenAIEmbeddingClient, OpenAIEmbeddingClient, OpenAIEmbeddingOptions
     - Uses _ensure_client() factory pattern
  
     Azure OpenAI implementation (azure/_embedding_client.py):
     - AzureOpenAIEmbeddingClient following AzureOpenAIChatClient pattern
     - Supports API key, Entra ID credentials, env var configuration
  
     Tests:
     - 47 unit tests for types, protocol, base class, OpenAI, and Azure clients
     - 6 integration tests (gated behind RUN_INTEGRATION_TESTS + credentials)
  
     Samples:
     - samples/02-agents/embeddings/openai_embeddings.py
     - samples/02-agents/embeddings/azure_openai_embeddings.py
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Add AzureOpenAIEmbeddingClient to azure __init__.pyi stub
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * ci: Add embedding env vars to Python integration tests
  
  Map OPENAI_EMBEDDING_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
  from GitHub vars to the integration test environment.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Handle base64 encoding_format in OpenAI embedding client
  
  When encoding_format='base64' is used, the OpenAI API returns base64-encoded
  floats instead of a JSON array. Decode these automatically to list[float]
  so the return type stays consistent regardless of encoding format.
  
  Also adds a unit test for base64 decoding and fixes minor docstring/import issues.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Only record INPUT_TOKENS for embedding telemetry
  
  Embeddings have no output/completion tokens. Remove OUTPUT_TOKENS recording
  which was double-counting prompt_tokens via the total_tokens fallback.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Resolve mypy variance error and lint warning
  
  Use contravariant/covariant TypeVars for SupportsGetEmbeddings Protocol.
  Combine nested if into single statement in telemetry layer.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Make EmbeddingCoT invariant for mypy compatibility
  
  GeneratedEmbeddings is invariant in its type param, so the Protocol
  TypeVar cannot be covariant.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Address PR review - empty values guard, service_url for telemetry
  
  - Add early return for empty values in get_embeddings to avoid unnecessary API calls
  - Add service_url() method to RawOpenAIEmbeddingClient for proper telemetry endpoint reporting
  - Add test for empty values behavior
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Python: Fix OpenAI chat client compatibility with third-party endpoints and OTel 0.4.14 (#4161)
  
  * Fix system message content sent as list instead of string
  
  Some OpenAI-compatible endpoints (e.g. NVIDIA NIM) reject system messages
  when content is a list of content parts. This change flattens system and
  developer message content to a plain string in the Chat Completions client.
  
  Fixes https://github.com/microsoft/agent-framework/issues/1407
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix compatibility with opentelemetry-semantic-conventions-ai 0.4.14
  
  Version 0.4.14 removed several LLM_* attributes from SpanAttributes
  (LLM_SYSTEM, LLM_REQUEST_MODEL, LLM_RESPONSE_MODEL, LLM_REQUEST_MAX_TOKENS,
  LLM_REQUEST_TEMPERATURE, LLM_REQUEST_TOP_P, LLM_TOKEN_TYPE).
  
  Move these to the OtelAttr enum with their well-known gen_ai.* string values
  and update all references in observability.py and tests.
  
  Fixes https://github.com/microsoft/agent-framework/issues/4160
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Flatten text-only message content to string for all roles
  
  Extend the system/developer fix to all message roles. Text-only content
  lists are now post-processed into plain strings, while multimodal content
  (text + images/audio) remains as a list. This fixes compatibility with
  OpenAI-like endpoints that cannot deserialize list content (e.g. Foundry
  Local's Neutron backend).
  
  Partially fixes https://github.com/microsoft/agent-framework/issues/4084
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix streaming text lost when usage data in same chunk
  
  Some providers (e.g. Gemini) include both usage data and text content
  in the same streaming chunk. The early return on chunk.usage caused
  text and tool call parsing to be skipped entirely. Remove the early
  return and process usage alongside text/tool calls.
  
  Fixes https://github.com/microsoft/agent-framework/issues/3434
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * Fix mypy errors in _chat_client.py
  
  Rename shadowed variable 'args' in system/developer branch to 'sys_args'
  and rename loop variable 'content' to 'msg_content' to avoid type conflict.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  ---------
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * reorder imports
  
  * fix: Use OtelAttr.REQUEST_MODEL instead of removed SpanAttributes.LLM_REQUEST_MODEL
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * docs: Add score_threshold to vector store plan
  
  Reference SK .NET PR #13501 for score threshold filtering semantics.
  Include score_threshold in SearchOptions from Phase 3.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * docs: Add reference to roji's SK .NET MEVD work for SQL connectors
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  * fix: Clear env vars in construction tests to avoid CI leakage
  
  Tests for missing API key / model ID now use monkeypatch.delenv to ensure
  env vars from the integration test environment don't prevent the expected
  ValueError from being raised.
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
  
  ---------
  
  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

  Eduard van Valkenburg · 2026-02-24 07:40:20 +00:00
  6305e3e092