61 Commits

• AgentInterface sets default streamFn and getApiKey

  - Make Agent.streamFn and Agent.getApiKey public
  - AgentInterface sets defaults if not already set
  - Example no longer needs to configure these

  Mario Zechner · 2025-12-30 22:42:21 +01:00
  4d1936d3df
• Clear streaming container on agent_end
  Mario Zechner · 2025-12-30 22:42:21 +01:00
  6972ce4e87
• Fix AgentInterface to requestUpdate on all message lifecycle events

  Was only handling message_update, now also handles message_start,
  message_end, turn_start, turn_end, agent_start, agent_end.

  Mario Zechner · 2025-12-30 22:42:21 +01:00
  347d4cf729
• Add defaultConvertToLlm to web-ui, simplify example

  - web-ui exports: defaultConvertToLlm, convertAttachments, isUserMessageWithAttachments, isArtifactMessage
  - defaultConvertToLlm handles UserMessageWithAttachments and filters ArtifactMessage
  - Example's customMessageTransformer now extends defaultConvertToLlm
  - Removes duplicated attachment conversion logic from example

  Mario Zechner · 2025-12-30 22:42:20 +01:00
  13a1991ec2
• Fix web-ui and example for new agent API

  - AgentInterface composes UserMessageWithAttachments for attachments
  - Updated example to use convertToLlm instead of transport/messageTransformer
  - Fixed declaration merging: target pi-agent-core, add path to example tsconfig
  - Fixed typo in CustomAgentMessages key (user-with-attachment -> user-with-attachments)
  - customMessageTransformer properly converts UserMessageWithAttachments to content blocks

  Mario Zechner · 2025-12-30 22:42:20 +01:00
  7a39f9eb11
• WIP: Major cleanup - move Attachment to consumers, simplify agent API

  - Removed Attachment from agent package (now in web-ui/coding-agent)
  - Agent.prompt now takes (text, images?: ImageContent[])
  - Removed transports from web-ui (duplicate of agent package)
  - Updated coding-agent to use local message types
  - Updated mom package for new agent API
  
  Remaining: Fix AgentInterface.ts to compose UserMessageWithAttachments

  Mario Zechner · 2025-12-30 22:42:20 +01:00
  6ddc7418da
• Enable more biome lints and fix things
  Mario Zechner · 2025-12-21 22:56:20 +01:00
  d5fd685901
• Add totalTokens field to Usage type

  - Added totalTokens field to Usage interface in pi-ai
  - Anthropic: computed as input + output + cacheRead + cacheWrite
  - OpenAI/Google: uses native total_tokens/totalTokenCount
  - Fixed openai-completions to compute totalTokens when reasoning tokens present
  - Updated calculateContextTokens() to use totalTokens field
  - Added comprehensive test covering 13 providers
  
  fixes #130

  Mario Zechner · 2025-12-06 22:46:02 +01:00
  86e5a70ec4
• Add lifecycle callbacks to SandboxRuntimeProvider for abort signal support

  - Added onExecutionStart(sandboxId, signal) - called when sandbox execution begins
  - Added onExecutionEnd(sandboxId) - called when sandbox execution ends
  - Integrated callbacks in SandboxedIframe.execute()
  - Enables providers to track and cancel async operations (e.g., userScript executions)

  Mario Zechner · 2025-11-16 00:56:20 +01:00
  3932c5e04c
• Fix Lit dependency duplication and update mini-lit to 0.2.0

  - Move lit from dependencies to peerDependencies in web-ui to prevent multiple Lit instances being loaded
  - Update mini-lit from 0.1.8/0.1.10 to 0.2.0 in root package.json and web-ui/package.json
  - Install @tailwindcss/vite in web-ui/example
  - Run biome formatting fixes across codebase
  
  This resolves HMR custom element re-registration errors caused by duplicate Lit registries.

  Mario Zechner · 2025-11-12 16:03:43 +01:00
  e533aebacd
• Add image support in tool results across all providers

  Tool results now use content blocks and can include both text and images.
  All providers (Anthropic, Google, OpenAI Completions, OpenAI Responses)
  correctly pass images from tool results to LLMs.
  
  - Update ToolResultMessage type to use content blocks
  - Add placeholder text for image-only tool results in Google/Anthropic
  - OpenAI providers send tool result + follow-up user message with images
  - Fix Anthropic JSON parsing for empty tool arguments
  - Add comprehensive tests for image-only and text+image tool results
  - Update README with tool result content blocks API

  Mario Zechner · 2025-11-12 10:45:56 +01:00
  84dcab219b
• Custom provider WIP
  Mario Zechner · 2025-11-10 21:47:21 +01:00
  1f9a3a00cc
• Improve web-ui message and tool UX

  - Add collapsible thinking blocks with shimmer animation during streaming
  - Update user messages to use orange gradient pill styling (matching sitegeist)
  - Fix cost display to only show for completed messages, not while streaming
  - Update tool renderers to use ChevronsUpDown/ChevronUp icons instead of rotating ChevronRight
  - Export ThinkingBlock component from public API

  Mario Zechner · 2025-10-30 20:42:04 +01:00
  389c80d7a8
• Fix ProviderKeyInput UX: improve save button state and remove clear button

  - Add inputChanged state to track when user modifies input after save
  - Remove Clear button and removeKey() method entirely
  - Save button now disables after successful save until input changes
  - Save button stays enabled on test failure (allows immediate retry)
  - Keep input field showing stars (••••) after successful save instead of clearing
  
  This provides clearer feedback and prevents accidental key deletion.

  Mario Zechner · 2025-10-29 21:14:19 +01:00
  9e50bb2c37
• Fix message editor height adjustment, requires Chrome +123
  Mario Zechner · 2025-10-29 16:11:05 +01:00
  92f8801864
• Better proxy handling.
  Mario Zechner · 2025-10-28 00:21:54 +01:00
  b6b64dff86
• Add timestamp to messages
  Mario Zechner · 2025-10-26 00:43:43 +02:00
  55dc0b6e08
• Updates to prompts
  Mario Zechner · 2025-10-17 22:44:03 +02:00
  2a7ccf0fcb
• Agent package + coding agent WIP, refactored web-ui prompts
  Mario Zechner · 2025-10-17 11:47:01 +02:00
  ffc9be8867
• Fix console.log duplication bug in ConsoleRuntimeProvider

  Issue:
  Each browser_javascript execution wrapped console methods, but captured
  the current (already wrapped) console as "original". This created a chain
  of wrappers that accumulated across executions:
  - Execution 1: 1x console.log (wrapper1 → real console)
  - Execution 2: 2x console.log (wrapper2 → wrapper1 → real console)
  - Execution 3: 3x console.log (wrapper3 → wrapper2 → wrapper1 → real console)
  - Execution 4: 4x console.log (and so on...)
  
  Fix:
  Store the truly original console methods in window.__originalConsole on
  first wrap only. All subsequent executions use these stored original methods
  instead of capturing the current console. This prevents wrapper accumulation.
  
  Changes:
  - Check if window.__originalConsole exists before wrapping
  - Store original console methods with .bind() to preserve context
  - Always use window.__originalConsole for local logging
  - Now each execution logs exactly 1x regardless of execution count

  Mario Zechner · 2025-10-13 00:20:23 +02:00
  949cd4efd8
• Add paste image support to MessageEditor

  - Add handlePaste method to detect and process pasted images
  - Automatically attach pasted images as attachments
  - Support multiple images in single paste
  - Prevent default paste behavior for images (text paste still works normally)
  - Use same loadAttachment utility as drag-and-drop for consistency

  Mario Zechner · 2025-10-12 19:47:38 +02:00
  f66d577eb2
• Add standalone mode for HTML artifact downloads

  When downloading HTML artifacts, the generated HTML now works standalone
  without requiring the extension runtime.
  
  Changes:
  - Add PrepareHtmlOptions config object to SandboxedIframe.prepareHtmlDocument()
  - Add isStandalone flag to skip runtime bridge and navigation interceptor
  - When isStandalone=true:
    - window.sendRuntimeMessage is NOT defined
    - Navigation interceptor is NOT injected
    - Artifact runtime providers fall back to embedded data
  - HtmlArtifact download button now uses isStandalone: true
  
  This fixes the issue where downloaded HTML artifacts would:
  - Try to call window.sendRuntimeMessage (which would fail silently)
  - Try to postMessage to non-existent parent window
  - Not work when opened via file:// protocol
  
  Now downloaded artifacts work completely standalone with embedded data.

  Mario Zechner · 2025-10-12 00:39:26 +02:00
  26b774bb04
• Fix up context check
  Mario Zechner · 2025-10-11 21:41:24 +02:00
  63bfe95c18
• Revert: Fix extension context detection - original logic was correct

  The negated approach broke browser_script which runs in page context.
  browser_script injects into web pages (http://, https://) where
  chrome.runtime.sendMessage doesn't work. The original whitelist
  approach correctly identifies extension contexts only.
  
  The real issue with artifacts in sandbox iframes needs a different fix.

  Mario Zechner · 2025-10-11 19:32:33 +02:00
  8a96741441
• Fix module script validation and extension context detection

  1. Skip JavaScript validation for <script type="module"> tags
     - new Function() cannot validate ES module syntax (import/export)
     - Module scripts now execute without validation errors
     - Fixes issue where HTML artifacts with modules would fail to load
  
  2. Improve __isExtensionContext detection logic
     - Changed from whitelist (chrome-extension://, moz-extension://)
     - To blacklist approach: negate http://, https://, file://
     - More robust - handles all extension protocols (about:srcdoc, etc.)
     - Fixes "offline mode" errors when using browser_script artifacts
  
  Resolves artifacts not working in extension context after document.write()

  Mario Zechner · 2025-10-11 19:19:19 +02:00
  372be18657
• Intercept iframe navigation and open links externally

  - Add navigation interceptor script that prevents all iframe navigation
  - Intercept link clicks, form submissions, and window.location changes
  - Send open-external-url messages to parent window
  - Use chrome.tabs.create() in extension context, window.open() fallback
  - Fix font-size workaround to use 'initial' instead of hardcoded 16px
  - Document Chrome extension font-size bug with Stack Overflow reference
  
  This prevents HTML artifacts from navigating away when users click links,
  which would break the sandbox message system. All links now open in new
  Chrome tabs instead.

  Mario Zechner · 2025-10-11 19:13:04 +02:00
  98ea763d64
• Fix font-size inheritance in HTML artifact iframes

  Set explicit font-size: 16px on the <html> element in sandboxed iframes
  to prevent browser default inheritance quirks that can cause unexpected
  font sizes (e.g., 75% shown in devtools).
  
  This establishes a consistent base font size while still allowing user
  HTML content to override it on body or specific elements as needed.
  
  Fixes the issue where HTML artifacts displayed with inconsistent font
  sizes due to iframe inheritance behavior.

  Mario Zechner · 2025-10-11 18:46:55 +02:00
  2a67556e5d
• Fix extension context detection in runtime providers

  Replace window.sendRuntimeMessage existence check with URL-based detection.
  The sendRuntimeMessage function exists in both extension and non-extension
  contexts (e.g., downloaded HTML artifacts), causing incorrect behavior.
  
  Changes:
  - Add window.__isExtensionContext() helper to RuntimeMessageBridge that checks:
    - chrome-extension:// URLs (Chrome)
    - moz-extension:// URLs (Firefox)
    - about:srcdoc (sandbox iframes)
  - Update all runtime providers to use __isExtensionContext() instead:
    - FileDownloadRuntimeProvider: Correctly falls back to browser download
    - ConsoleRuntimeProvider: Only sends messages in extension context
    - ArtifactsRuntimeProvider: Properly detects offline/read-only mode
  
  This fixes the issue where downloaded HTML artifacts incorrectly try to
  communicate with the extension when opened from disk.

  Mario Zechner · 2025-10-11 18:39:45 +02:00
  b5648eaabd
• Add ToolRenderResult interface for custom tool rendering

  - Changed ToolRenderer return type from TemplateResult to ToolRenderResult
  - ToolRenderResult = { content: TemplateResult, isCustom: boolean }
  - isCustom: true = no card wrapper, false = wrap in card
  - Updated all existing tool renderers to return new format
  - Updated Messages.ts to handle custom rendering
  
  This enables tools to render without default card chrome when needed.

  Mario Zechner · 2025-10-11 04:40:42 +02:00
  b129154cc8
• Replace hasArtifact with listArtifacts

  LLMs don't need to check existence - they can just list all artifacts.
  Simpler API that returns all filenames at once.
  
  Changes:
  - Replace hasArtifact(filename) with listArtifacts() returning string[]
  - Add 'list' action handler that returns all artifact keys
  - Update examples in prompt to use listArtifacts()

  Mario Zechner · 2025-10-10 01:49:03 +02:00
  2e56f77599
• Merge createArtifact and updateArtifact into createOrUpdateArtifact

  LLMs get confused about when to use create vs update. The single function
  automatically detects if the artifact exists and chooses the right operation.
  
  Changes:
  - Replace createArtifact/updateArtifact with createOrUpdateArtifact in runtime
  - Update handler to check existence and use appropriate command (create/rewrite)
  - Simplify prompt documentation and examples

  Mario Zechner · 2025-10-10 01:43:36 +02:00
  7a859cbe52
• Fix abort detection to use message stopReason instead of global isStreaming

  When abort happens during tool call streaming, the tool result should show as aborted.
  Previously used global isStreaming state which would flip when new messages streamed in,
  causing spinner to reappear incorrectly.
  
  Changes:
  - Use message.stopReason === "aborted" to detect aborted tool calls
  - Create synthetic error result for aborted tool calls in ToolMessage component
  - Fix Ollama provider key test to return true (can't know which model to test)
  - Add newline before HTML execution logs in artifacts update

  Mario Zechner · 2025-10-10 01:40:59 +02:00
  961112ff83
• Simplify ArtifactsRuntimeProvider constructor

  - Take artifactsPanel and agent directly instead of 5 separate function parameters
  - Define minimal ArtifactsPanelLike and AgentLike interfaces to avoid circular deps
  - Update all call sites (ChatPanel, browser-javascript) to use simplified constructor
  - Much cleaner and easier to use

  Mario Zechner · 2025-10-09 23:37:14 +02:00
  695bcf9b6f
• Fix console logging and message routing

  Console Logging Improvements:
  - Changed ConsoleRuntimeProvider to send logs immediately instead of batching
  - Track pending send promises and await them in onCompleted callback
  - Ensures REPL gets all logs before execution-complete
  - Enables real-time console logging for HTML artifacts
  
  Message Routing Fixes:
  - Remove "handled" concept from message routing - broadcast all messages to all providers/consumers
  - Change handleMessage return type from Promise<boolean> to Promise<void>
  - Add debug logging to RuntimeMessageRouter to trace message flow
  - Fix duplicate error logging (window error handler now only tracks errors, doesn't log them)
  
  Output Formatting Consistency:
  - Remove [LOG], [ERROR] prefixes from console output in both tools
  - Show console logs before error messages
  - Use "=> value" format for return values in both javascript-repl and browser-javascript
  - Remove duplicate "Error:" prefix and extra formatting
  
  Bug Fixes:
  - Fix race condition where execution-complete arrived before console logs
  - Fix ConsoleRuntimeProvider blocking execution-complete from reaching consumers
  - Remove duplicate console log collection from SandboxedIframe
  - Fix return value capture by wrapping user code in async function

  Mario Zechner · 2025-10-09 23:20:14 +02:00
  4ac774cbbb
• Remove debug console.log statements causing duplicate error messages

  - console.log('Reporting execution error:', finalError) was logging the error
  - This caused duplicate error message in output
  - Removed all debug console.log statements from window.complete()
  - Error is only shown via execution-error message now

  Mario Zechner · 2025-10-09 20:53:32 +02:00
  38aaaee968
• Display return values and fix duplicate error messages in REPL

  - javascript-repl now displays return values with => prefix
  - Objects are JSON.stringify'd with formatting
  - Removed console.error from wrapper to prevent duplicate error messages
  - Error is already captured and sent via execution-error message
  - Fixes missing return value display and duplicate error output

  Mario Zechner · 2025-10-09 20:51:25 +02:00
  8747e01c31
• Pass return value through execution-complete message instead of logging

  - Return value now passed to window.complete(error, returnValue)
  - execution-complete message includes returnValue field
  - SandboxResult interface updated to include returnValue
  - executionConsumer passes returnValue in resolved promise
  - Return values properly captured and available to callers

  Mario Zechner · 2025-10-09 20:47:26 +02:00
  5fc3857666
• Fix REPL timeout with return statements by wrapping user code in function

  - User code with return statement was exiting the async IIFE early
  - Completion callbacks and window.complete() were never reached
  - Now wrap user code in userCodeFunc to capture return value
  - Return statement returns from userCodeFunc, not outer IIFE
  - Completion callbacks and window.complete() always execute
  - Return value is logged to console output
  - Fixes 30-second timeout when using return statements in REPL

  Mario Zechner · 2025-10-09 20:43:39 +02:00
  9a3f01fcb8
• Fix message routing by broadcasting to all providers and consumers

  - Router was stopping propagation after first handler returned true
  - This prevented consumers from seeing messages that providers handled
  - executionConsumer never received execution-complete because ConsoleRuntimeProvider handled it first
  - Now all providers and consumers receive all messages
  - Fixes javascript_repl never completing

  Mario Zechner · 2025-10-09 20:41:22 +02:00
  fdd4e24246
• Fix ConsoleRuntimeProvider blocking execution-complete from reaching executionConsumer

  - ConsoleRuntimeProvider was handling execution-complete and returning true
  - This stopped message propagation before executionConsumer could handle it
  - ExecutionConsumer never got execution-complete, so promise never resolved
  - Now ConsoleRuntimeProvider responds but returns false to allow propagation
  - Fixes javascript_repl never completing (30s timeout)

  Mario Zechner · 2025-10-09 20:39:04 +02:00
  3f29ce9cb6
• Fix race condition by making window.complete() async and awaiting it

  - window.complete() was fire-and-forget, causing execution-complete to arrive before console messages
  - This caused sandbox to unregister before console message responses arrived
  - Made complete() async and await sendRuntimeMessage()
  - SandboxedIframe wrapper now awaits window.complete()
  - Ensures all messages are processed before cleanup/unregister
  - Fixes 30-second timeout on javascript_repl

  Mario Zechner · 2025-10-09 20:31:48 +02:00
  5dcb5ecc89
• Fix ConsoleRuntimeProvider hanging by calling respond()

  - ConsoleRuntimeProvider.handleMessage() was not calling respond()
  - This caused sendRuntimeMessage() to hang waiting for response (30s timeout)
  - Now properly acknowledges console, execution-complete, and execution-error messages
  - Fixes javascript_repl hanging on simple console.log() calls

  Mario Zechner · 2025-10-09 20:27:02 +02:00
  bb138307b1
• Unify completion callback pattern across browser-javascript and javascript-repl

  - Remove fallback timeout from ConsoleRuntimeProvider (was causing 2s delays)
  - Add completion callback support to SandboxedIframe REPL wrapper
  - Call completion callbacks before window.complete() in both success/error paths
  - Both browser-javascript and javascript-repl now use identical completion pattern
  - Ensures console logs are batched and sent before execution completes

  Mario Zechner · 2025-10-09 20:24:20 +02:00
  33418d9dea
• Simplify ConsoleRuntimeProvider to batch-send logs at completion only
  Mario Zechner · 2025-10-09 20:16:32 +02:00
  6983ad4eaa
• Add onCompleted callback system for guaranteed console log delivery

  - Add onCompleted() callback registration in RuntimeMessageBridge
  - Modify wrapperFunction to call completion callbacks before returning
  - Update ConsoleRuntimeProvider to immediate send + completion batch pattern
  - Extract DOWNLOADABLE_FILE_RUNTIME_DESCRIPTION from ATTACHMENTS_RUNTIME_DESCRIPTION
  - Logs sent immediately (fire-and-forget), unsent logs batched at completion
  - Ensures all console logs arrive before tool execution completes

  Mario Zechner · 2025-10-09 20:10:20 +02:00
  af0297cd16
• Add debug logging to RuntimeMessageRouter and increase API key validation tokens

  - Add console.log for user script messages in RuntimeMessageRouter
  - Increase ProviderKeyInput maxTokens from 10 to 200 for better validation

  Mario Zechner · 2025-10-09 18:57:43 +02:00
  b288cd9448
• Add runtime bridge architecture and fix HTML escaping

  Major refactoring to unify runtime providers across sandbox and user script contexts:
  
  1. Runtime Bridge & Router
     - Add RuntimeMessageBridge for unified messaging abstraction
     - Rename SandboxMessageRouter → RuntimeMessageRouter
     - Router now handles both iframe and user script messages
     - Guard for non-extension environments
  
  2. Provider Refactoring
     - ArtifactsRuntimeProvider: Add offline mode with snapshot fallback
     - AttachmentsRuntimeProvider: Remove returnDownloadableFile (moved to dedicated provider)
     - ConsoleRuntimeProvider: Add message collection, remove lifecycle logic
     - FileDownloadRuntimeProvider: New provider for file downloads
  
  3. HTML Escaping Fix
     - Escape </script> in JSON.stringify output to prevent premature tag closure
     - Applies when injecting provider data into <script> tags
     - JavaScript engine automatically unescapes, no runtime changes needed
  
  4. Function Renaming
     - listFiles → listAttachments
     - readTextFile → readTextAttachment
     - readBinaryFile → readBinaryAttachment
     - returnFile → returnDownloadableFile
  
  5. Updated Exports
     - Export new RuntimeMessageBridge and RuntimeMessageRouter
     - Export FileDownloadRuntimeProvider
     - Update all cross-references
  
  This sets the foundation for reusing providers in browser-javascript tool.

  Mario Zechner · 2025-10-09 17:32:45 +02:00
  c2793d8017
• Add artifact message persistence for session reconstruction

  - Add ArtifactMessage type as core part of AppMessage union (not CustomMessages)
  - ArtifactsRuntimeProvider appends artifact messages on create/update/delete
  - MessageList filters out artifact messages (UI display only)
  - artifacts.ts reconstructFromMessages handles artifact messages
  - Export ARTIFACTS_RUNTIME_PROVIDER_DESCRIPTION from main index
  - Fix artifact creation bug: pass filename as title instead of mimeType
  
  Changes:
  - web-ui/src/components/Messages.ts: Add ArtifactMessage to BaseMessage union
  - web-ui/src/components/MessageList.ts: Skip artifact messages in render
  - web-ui/src/components/sandbox/ArtifactsRuntimeProvider.ts: Append messages, fix title parameter
  - web-ui/src/ChatPanel.ts: Pass agent.appendMessage callback
  - web-ui/src/tools/artifacts/artifacts.ts: Handle artifact messages in reconstructFromMessages
  - web-ui/src/index.ts: Export ARTIFACTS_RUNTIME_PROVIDER_DESCRIPTION
  - web-ui/example/src/custom-messages.ts: Update message transformer to filter artifacts

  Mario Zechner · 2025-10-09 04:07:59 +02:00
  4d2ca6ab2a
• Fix various sandbox issues.
  Mario Zechner · 2025-10-08 22:51:32 +02:00
  0eaa879d46
• Add ollama dependency and dialog backdrop blur

  - Add ollama package to web-ui dependencies for ModelSelector
  - Add backdrop blur to SettingsDialog (bg-black/50 backdrop-blur-sm)
  - Update mini-lit to 0.1.9 for backdropClassName prop support
  - Fix TypeScript errors in ModelSelector (ollama import, parameter types)
  - Add backward compatibility methods to SessionsStore (saveSession, loadSession, getLatestSessionId)

  Mario Zechner · 2025-10-08 17:49:56 +02:00
  91c1dc6475