agent-framework/python/samples/05-end-to-end/chatkit-integration/README.md

ChatKit Integration Sample with Weather Agent and Image Analysis

This sample demonstrates how to integrate Microsoft Agent Framework with OpenAI ChatKit. It provides a complete implementation of a weather assistant with interactive widget visualization, image analysis, and file upload support.

Features:

• Weather information with interactive widgets
• Image analysis using vision models
• Current time queries
• File upload with attachment storage
• Chat interface with streaming responses
• City selector widget with one-click weather

Architecture

graph TB
    subgraph Frontend["React Frontend (ChatKit UI)"]
        UI[ChatKit Components]
        Upload[File Upload]
    end

    subgraph Backend["FastAPI Server"]
        FastAPI[FastAPI Endpoints]

        subgraph ChatKit["WeatherChatKitServer"]
            Respond[respond method]
            Action[action method]
        end

        subgraph Stores["Data & Storage Layer"]
            SQLite[SQLiteStore<br/>Store Protocol]
            AttStore[FileBasedAttachmentStore<br/>AttachmentStore Protocol]
            DB[(SQLite DB<br/>chatkit_demo.db)]
            Files[/uploads directory/]
        end

        subgraph Integration["Agent Framework Integration"]
            Converter[ThreadItemConverter]
            Streamer[stream_agent_response]
            Agent[Agent]
        end

        Widgets[Widget Rendering<br/>render_weather_widget<br/>render_city_selector_widget]
    end

    subgraph Azure["Azure AI"]
        Foundry[GPT-5<br/>with Vision]
    end

    UI -->|HTTP POST /chatkit| FastAPI
    Upload -->|HTTP POST /upload/id| FastAPI

    FastAPI --> ChatKit

    ChatKit -->|save/load threads| SQLite
    ChatKit -->|save/load attachments| AttStore
    ChatKit -->|convert messages| Converter

    SQLite -.->|persist| DB
    AttStore -.->|save files| Files
    AttStore -.->|save metadata| SQLite

    Converter -->|Message array| Agent
    Agent -->|AgentResponseUpdate| Streamer
    Streamer -->|ThreadStreamEvent| ChatKit

    ChatKit --> Widgets
    Widgets -->|WidgetItem| ChatKit

    Agent <-->|Chat Completions API| Foundry

    ChatKit -->|ThreadStreamEvent| FastAPI
    FastAPI -->|SSE Stream| UI

    style ChatKit fill:#e1f5ff
    style Stores fill:#fff4e1
    style Integration fill:#f0e1ff
    style Azure fill:#e1ffe1

Server Implementation

The sample implements a ChatKit server using the ChatKitServer base class from the chatkit package:

Core Components:

• WeatherChatKitServer: Custom ChatKit server implementation that:

  • Extends ChatKitServer[dict[str, Any]]
  • Uses Agent Framework's Agent with Azure OpenAI
  • Converts ChatKit messages to Agent Framework format using ThreadItemConverter
  • Streams responses back to ChatKit using stream_agent_response
  • Creates and streams interactive widgets after agent responses

• SQLiteStore: Data persistence layer that:

  • Implements the Store[dict[str, Any]] protocol from ChatKit
  • Persists threads, messages, and attachment metadata in SQLite
  • Provides thread management and item history
  • Stores attachment metadata for the upload lifecycle

• FileBasedAttachmentStore: File storage implementation that:

  • Implements the AttachmentStore[dict[str, Any]] protocol from ChatKit
  • Stores uploaded files on the local filesystem (in ./uploads directory)
  • Generates upload URLs for two-phase file upload
  • Saves attachment metadata to the data store for upload tracking
  • Provides preview URLs for images

Key Integration Points:

# Converting ChatKit messages to Agent Framework
converter = ThreadItemConverter(
    attachment_data_fetcher=self._fetch_attachment_data
)
agent_messages = await converter.to_agent_input(user_message_item)

# Running agent and streaming back to ChatKit
async for event in stream_agent_response(
    self.weather_agent.run(agent_messages, stream=True),
    thread_id=thread.id,
):
    yield event

# Streaming widgets
widget = render_weather_widget(weather_data)
async for event in stream_widget(thread_id=thread.id, widget=widget):
    yield event

Installation and Setup

Prerequisites

• Python 3.10+
• Node.js 18.18+ and npm 9+
• Azure OpenAI service configured
• Azure CLI for authentication (az login)

Network Requirements

Important: This sample uses the OpenAI ChatKit frontend, which requires internet connectivity to OpenAI services.

The frontend makes outbound requests to:

• cdn.platform.openai.com - ChatKit UI library (required)
• chatgpt.com - Configuration endpoint
• api-js.mixpanel.com - Telemetry

This sample is not suitable for air-gapped or network-restricted environments. The ChatKit frontend library cannot be self-hosted. See Limitations for details.

Domain Key Configuration

For local development, the sample uses a default domain key (domain_pk_localhost_dev).

For production deployment:

1. Register your domain at platform.openai.com

2. Create a .env file in the frontend directory:

   VITE_CHATKIT_API_DOMAIN_KEY=your_domain_key_here
   

Backend Setup

1. Install Python packages:

cd python/samples/05-end-to-end/chatkit-integration
pip install agent-framework-chatkit fastapi uvicorn azure-identity

2. Configure Azure OpenAI:

export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_VERSION="2024-06-01"
export AZURE_OPENAI_CHAT_DEPLOYMENT_NAME="gpt-4o"

3. Authenticate with Azure:

az login

Frontend Setup

Install the Node.js dependencies:

cd frontend
npm install

How to Run

Start the Backend Server

From the chatkit-integration directory:

python app.py

Or with auto-reload for development:

uvicorn app:app --host 127.0.0.1 --port 8001 --reload

The backend will start on http://localhost:8001

Start the Frontend Development Server

In a new terminal, from the frontend directory:

npm run dev

The frontend will start on http://localhost:5171

Access the Application

Open your browser and navigate to:

http://localhost:5171

You can now:

• Ask about weather in any location (weather widgets display automatically)
• Upload images for analysis using the attachment button
• Get the current time
• Ask to see available cities and click city buttons for instant weather

Project Structure

chatkit-integration/
├── app.py                    # FastAPI backend with ChatKitServer implementation
├── store.py                  # SQLiteStore implementation
├── attachment_store.py       # FileBasedAttachmentStore implementation
├── weather_widget.py         # Widget rendering functions
├── chatkit_demo.db          # SQLite database (auto-created)
├── uploads/                  # Uploaded files directory (auto-created)
└── frontend/
    ├── package.json
    ├── vite.config.ts
    ├── index.html
    └── src/
        ├── main.tsx
        └── App.tsx           # ChatKit UI integration

Configuration

You can customize the application by editing constants at the top of app.py:

# Server configuration
SERVER_HOST = "127.0.0.1"  # Bind to localhost only for security (local dev)
SERVER_PORT = 8001
SERVER_BASE_URL = f"http://localhost:{SERVER_PORT}"

# Database configuration
DATABASE_PATH = "chatkit_demo.db"

# File storage configuration
UPLOADS_DIRECTORY = "./uploads"

# User context
DEFAULT_USER_ID = "demo_user"

Sample Conversations

Try these example queries:

• "What's the weather like in Tokyo?"
• "Show me available cities" (displays interactive city selector)
• "What's the current time?"
• Upload an image and ask "What do you see in this image?"

Limitations

Air-Gapped / Regulated Environments

The ChatKit frontend (chatkit.js) is loaded from OpenAI's CDN and cannot be self-hosted. This means:

• Not suitable for air-gapped environments where *.openai.com is blocked
• Not suitable for regulated environments that prohibit external telemetry
• Requires domain registration with OpenAI for production use

What you CAN self-host:

• The Python backend (FastAPI server, ChatKitServer, stores)
• The agent-framework-chatkit integration layer
• Your LLM infrastructure (Azure OpenAI, local models, etc.)

What you CANNOT self-host:

• The ChatKit frontend UI library

For more details, see:

• openai/chatkit-js#57 - Self-hosting feature request
• openai/chatkit-js#76 - Domain key requirements

Learn More

• Agent Framework Documentation
• ChatKit Documentation
• Azure OpenAI Documentation