agent/pi - pi - Penguin

Fix up prompt for image-tool-result for Gemini

Mario Zechner · 2025-11-20 17:09:32 +01:00

4afb3231e4

Release v0.7.17

Mario Zechner · 2025-11-18 17:49:12 +01:00

a11c1aa4ff

Add image support in tool results across all providers

Tool results now use content blocks and can include both text and images.
All providers (Anthropic, Google, OpenAI Completions, OpenAI Responses)
correctly pass images from tool results to LLMs.

- Update ToolResultMessage type to use content blocks
- Add placeholder text for image-only tool results in Google/Anthropic
- OpenAI providers send tool result + follow-up user message with images
- Fix Anthropic JSON parsing for empty tool arguments
- Add comprehensive tests for image-only and text+image tool results
- Update README with tool result content blocks API

Mario Zechner · 2025-11-12 10:45:56 +01:00

84dcab219b

Fix token statistics on abort for Anthropic provider

- Add handling for message_start event to capture initial token usage
- Fix message_delta to use assignment (=) instead of addition (+=)
  since Anthropic sends cumulative token counts, not incremental
- Add comprehensive tests for all providers (Google, OpenAI Completions,
  OpenAI Responses, Anthropic)
- Document OpenAI limitation: token stats only available at stream end

Fixes issue where aborted streams had zero token counts despite
Anthropic sending input tokens in the initial message_start event.

Mario Zechner · 2025-10-26 21:22:24 +01:00

bc8d994a7b

Add timestamp to messages

Mario Zechner · 2025-10-26 00:43:43 +02:00

55dc0b6e08

Add Unicode surrogate sanitization for all providers

Fixes issue where unpaired Unicode surrogates in tool results cause JSON serialization errors in API providers, particularly Anthropic.

- Add sanitizeSurrogates() utility function to remove unpaired surrogates
- Apply sanitization in all provider convertMessages() functions:
  - User message text content (string and text blocks)
  - Assistant message text and thinking blocks
  - Tool result output
  - System prompts
- Valid emoji (properly paired surrogates) are preserved
- Add comprehensive test suite covering all 8 providers

Previously only Google and Groq handled unpaired surrogates correctly.
Now all providers (Anthropic, OpenAI Completions/Responses, Google, xAI, Groq, Cerebras, zAI) sanitize text before API submission.

Mario Zechner · 2025-10-13 14:26:54 +02:00

4e7a340460

Add ToolRenderResult interface for custom tool rendering

- Changed ToolRenderer return type from TemplateResult to ToolRenderResult
- ToolRenderResult = { content: TemplateResult, isCustom: boolean }
- isCustom: true = no card wrapper, false = wrap in card
- Updated all existing tool renderers to return new format
- Updated Messages.ts to handle custom rendering

This enables tools to render without default card chrome when needed.

Mario Zechner · 2025-10-11 04:40:42 +02:00

b129154cc8

Remove tool calls for which there are no results in subsequent user messages.

Mario Zechner · 2025-10-01 22:18:30 +02:00

51f5448a5c

refactor(ai): improve error handling and stop reason types

- Add 'aborted' as a distinct stop reason separate from 'error'
- Change AssistantMessage.error to errorMessage for clarity
- Update error event to include reason field ('error' | 'aborted')
- Map provider-specific safety/refusal reasons to 'error' stop reason
- Reorganize utility functions into utils/ directory
- Rename agent.ts to agent-loop.ts for better clarity
- Fix error handling in all providers to properly distinguish abort from error

Mario Zechner · 2025-09-18 19:57:13 +02:00

2296dc4052

feat(ai): add partial JSON parsing for streaming tool calls

- Added partial-json package for parsing incomplete JSON during streaming
- Tool call arguments now contain partially parsed JSON during toolcall_delta events
- Enables progressive UI updates (e.g., showing file paths before content is complete)
- Arguments are always valid objects (minimum empty {}), never undefined
- Full validation still occurs at toolcall_end when arguments are complete
- Updated all providers (Anthropic, OpenAI Completions/Responses) to use parseStreamingJson
- Added comprehensive documentation and examples in README
- Added test to verify arguments are always defined during streaming

Mario Zechner · 2025-09-16 12:23:34 +02:00

39c626b6c9

Fix NodeJS compat

Mario Zechner · 2025-09-16 02:19:47 +02:00

197259c88a

Replace Zod with TypeBox for schema validation

- Switch from Zod to TypeBox for tool parameter schemas
- TypeBox schemas can be serialized/deserialized as JSON
- Use AJV for runtime validation instead of Zod's parse
- Add StringEnum helper for Google API compatibility (avoids anyOf/const patterns)
- Export Type and Static from main package for convenience
- Update all tests and documentation to reflect TypeBox usage

Mario Zechner · 2025-09-16 01:10:40 +02:00

e8370436d7

feat(ai): Implement Zod-based tool validation and improve Agent API

- Replace JSON Schema with Zod schemas for tool parameter definitions
- Add runtime validation for all tool calls at provider level
- Create shared validation module with detailed error formatting
- Update Agent API with comprehensive event system
- Add agent tests with calculator tool for multi-turn execution
- Add abort test to verify proper handling of aborted requests
- Update documentation with detailed event flow examples
- Rename generate.ts to stream.ts for clarity

Mario Zechner · 2025-09-09 14:58:54 +02:00

35fe8f21e9

Stop GPT-OSS 20b from being dumb ..

Mario Zechner · 2025-09-09 04:31:09 +02:00

594b0dac6c

Fix streaming for z-ai in anthropic provider, add preliminary support for tool call streaming. Only reporting argument string deltas, not partial JSON objects

Mario Zechner · 2025-09-09 04:26:56 +02:00

98a876f3a0

chore: bump version to 0.5.31

Mario Zechner · 2025-09-07 00:09:34 +02:00

2bdb87dfe7

feat(ai): Add zAI provider support

- Add 'zai' as a KnownProvider type
- Add ZAI_API_KEY environment variable mapping
- Generate 4 zAI models (glm-4.5-air, glm-4.5v, etc.) using anthropic-messages API
- Add comprehensive test coverage for zAI provider in generate.test.ts and empty.test.ts
- Models support reasoning/thinking capabilities and tool calling

Mario Zechner · 2025-09-07 00:09:15 +02:00

d073953ef7

fix(ai): Sanitize tool call IDs for Anthropic API compatibility

- Anthropic API requires tool call IDs to match pattern ^[a-zA-Z0-9_-]+$
- OpenAI Responses API generates IDs with pipe character (|) which breaks Anthropic
- Added sanitizeToolCallId() to replace invalid characters with underscores
- Fixes cross-provider handoffs from OpenAI Responses to Anthropic
- Added test to verify the fix works

Mario Zechner · 2025-09-04 05:17:08 +02:00

6679a83b32

Clean-up

Mario Zechner · 2025-09-03 00:01:32 +02:00

acf0f5aee2

Massive refactor of API

- Switch to function based API
- Anthropic SDK style async generator
- Fully typed with escape hatches for custom models

Mario Zechner · 2025-09-02 23:59:36 +02:00

66cefb236e

feat(ai): Add new streaming generate API with AsyncIterable interface

- Implement QueuedGenerateStream class that extends AsyncIterable with finalMessage() method
- Add new types: GenerateStream, GenerateOptions, GenerateOptionsUnified, GenerateFunction
- Create generateAnthropic function-based implementation replacing class-based approach
- Add comprehensive test suite for the new generate API
- Support streaming events with text, thinking, and tool call deltas
- Map ReasoningEffort to provider-specific options
- Include apiKey in options instead of constructor parameter

Mario Zechner · 2025-09-02 18:07:46 +02:00

004de3c9d0

test(ai): Add empty assistant message tests

- Test providers handling empty assistant messages in conversation flow
- Pattern: user message -> empty assistant -> user message
- All providers handle empty assistant messages gracefully
- Tests ensure providers can continue conversation after empty response

Mario Zechner · 2025-09-02 02:10:07 +02:00

be07c08a75

test(ai): Add empty message tests for all providers

- Test handling of empty content arrays
- Test handling of empty string content
- Test handling of whitespace-only content
- All providers handle these edge cases gracefully

Mario Zechner · 2025-09-02 02:03:06 +02:00

0ac05a0676

feat(ai): Fetch Anthropic, Google, and OpenAI models from models.dev instead of OpenRouter

- Updated generate-models.ts to fetch these providers directly from models.dev API
- OpenRouter now only used for xAI and other third-party providers
- Fixed test model IDs to match new model names from models.dev
- Removed unused import from google.ts

Mario Zechner · 2025-09-02 01:18:59 +02:00

efaa5cdb39

feat(ai): Add cross-provider message handoff support

- Add transformMessages utility to handle cross-provider compatibility
- Convert thinking blocks to <thinking> tagged text when switching providers
- Preserve native thinking blocks when staying with same provider/model
- Add comprehensive handoff tests verifying all provider combinations
- Fix OpenAI Completions to return partial results on abort
- Update tool call ID format for Anthropic compatibility
- Document cross-provider handoff capabilities in README

Mario Zechner · 2025-09-01 18:43:49 +02:00

46b5800d36

refactor(ai): Update API to support partial results on abort

- Anthropic, Google, and OpenAI Responses providers now return partial results when aborted
- Restructured streaming to accumulate content blocks incrementally
- Prevents submission of thinking/toolCall blocks from aborted completions in multi-turn conversations
- Makes UI development easier by providing partial content even when requests are interrupted

Mario Zechner · 2025-09-01 01:57:45 +02:00

bf1f410c2b

feat(ai): Add start event emission to all providers

- Emit start event with model and provider info after creating stream
- Add abort signal tests for all providers
- Update README abort signal section to reflect non-throwing API
- Fix model references in README examples

Mario Zechner · 2025-08-31 23:09:14 +02:00

a132b8140c

fix(ai): Fix OpenAI Responses provider multi-turn conversation support

- Collect complete output items during streaming instead of building blocks incrementally
- Handle reasoning summary parts with proper newline separation
- Support refusal content in message outputs
- Preserve full reasoning items and message IDs for multi-turn resubmission
- Emit proper streaming events for text and thinking deltas

Mario Zechner · 2025-08-31 22:11:08 +02:00

ee4c131873

refactor(ai): Update OpenAI Completions provider to new content block API

Mario Zechner · 2025-08-31 20:59:57 +02:00

a72e6d08d4

refactor(ai): Update API to support multiple thinking and text blocks

BREAKING CHANGE: AssistantMessage now uses content array instead of separate fields
- Changed AssistantMessage.content from string to array of content blocks
- Removed separate thinking, toolCalls, and signature fields
- Content blocks can be TextContent, ThinkingContent, or ToolCall types
- Updated streaming events to include start/end events for text and thinking
- Fixed multiTurn test to handle new content structure

Note: Currently only Anthropic provider is updated to work with new API
Other providers need to be updated to match the new interface

Mario Zechner · 2025-08-31 19:32:12 +02:00

f29752ac82

test(ai): Add image input test for Anthropic Haiku 3.5

- Added image handling test for Claude 3.5 Haiku
- Ensures vision capabilities are properly tested

Mario Zechner · 2025-08-30 23:00:30 +02:00

f8a81b9453

fix(ai): Fix OpenAI Responses provider multi-turn conversation support

- Added contentSignature tracking for assistant messages
- Fixed message format in convertToResponsesFormat (output_text instead of input_text)
- Properly preserve message IDs for multi-turn conversations
- Added proper ResponseOutputMessage type satisfaction
- Updated tests to cover more providers and multi-turn scenarios

Mario Zechner · 2025-08-30 22:55:29 +02:00

cff766d3e2

feat(ai): Add image input tests for vision-capable models

- Added image tests to OpenAI Completions (gpt-4o-mini)
- Added image tests to Anthropic (claude-sonnet-4-0)
- Added image tests to Google (gemini-2.5-flash)
- Tests verify models can process and describe the red circle test image

Mario Zechner · 2025-08-30 18:37:17 +02:00

796e48b80e

refactor(ai): Update LLM implementations to use Model objects

- LLM constructors now take Model objects instead of string IDs
- Added provider field to AssistantMessage interface
- Updated getModel function with type-safe model ID autocomplete
- Fixed Anthropic model ID mapping for proper API aliases
- Added baseUrl to Model interface for provider-specific endpoints
- Updated all tests to use getModel for model instantiation
- Removed deprecated models.json in favor of generated models

Mario Zechner · 2025-08-30 00:21:03 +02:00

f9d688d577

fix(ai): Deduplicate models and add Anthropic aliases

- Add proper Anthropic model aliases (claude-opus-4-1, claude-sonnet-4-0, etc.)
- Deduplicate models when same ID appears in both models.dev and OpenRouter
- models.dev takes priority over OpenRouter for duplicate IDs
- Fix test to use correct claude-3-5-haiku-latest alias
- Reduces Anthropic models from 11 to 10 (removed duplicate)

Mario Zechner · 2025-08-29 23:34:01 +02:00

d61d09b88d

refactor(ai): Implement unified model system with type-safe createLLM

- Add Model interface to types.ts with normalized structure
- Create type-safe generic createLLM function with provider-specific model constraints
- Generate models from OpenRouter API and models.dev data
- Strip provider prefixes for direct providers (google, openai, anthropic, xai)
- Keep full model IDs for OpenRouter-proxied models
- Clean separation: types.ts (Model interface), models.ts (factory logic), models.generated.ts (data)
- Remove old model scripts and unused dependencies
- Rename GeminiLLM to GoogleLLM for consistency
- Add tests for new providers (xAI, Groq, Cerebras, OpenRouter)
- Support 181 tool-capable models across 7 providers with full type safety

Mario Zechner · 2025-08-29 23:19:47 +02:00

c7618db3f7

feat(ai): Migrate tests to Vitest and add provider test coverage

- Switch from Node.js test runner to Vitest for better DX
- Add test suites for Grok, Groq, Cerebras, and OpenRouter providers
- Add Ollama test suite with automatic server lifecycle management
- Include thinking mode and multi-turn tests for all providers
- Remove example files (consolidated into test suite)
- Add VS Code test configuration

Mario Zechner · 2025-08-29 21:32:45 +02:00

3f36051bc6

fix(ai): Improve ModelInfo types based on actual data structure

- Remove catch-all [key: string]: any from ModelInfo
- Make all required fields non-optional (attachment, reasoning, etc.)
- Add proper union types for modalities (text, image, audio, video, pdf)
- Mark only cost and knowledge fields as optional
- Export ModalityInput and ModalityOutput types

Mario Zechner · 2025-08-25 20:18:34 +02:00

9b8ea585bd

feat(ai): Add models.dev data integration

- Add models script to download latest model information
- Create models.ts module to query model capabilities
- Include models.json in package distribution
- Export utilities to check model features (reasoning, tools)
- Update build process to copy models.json to dist

Mario Zechner · 2025-08-25 20:10:54 +02:00

02a9b4f09f

feat(ai): Add OpenAI-compatible provider examples for multiple services

- Add examples for Cerebras, Groq, Ollama, and OpenRouter
- Update OpenAI Completions provider to handle base URL properly
- Simplify README formatting
- All examples use the same OpenAICompletionsLLM provider with different base URLs

Mario Zechner · 2025-08-25 17:41:47 +02:00

4bb3a5ad02

test(ai): Add comprehensive E2E tests for all AI providers

- Add multi-turn test to verify thinking and tool calling work together
- Test thinkingSignature handling for proper multi-turn context
- Fix Gemini provider to generate base64 thinkingSignature when needed
- Handle multiple rounds of tool calls in tests (Gemini behavior)
- Make thinking tests more robust for model-dependent behavior
- All 18 tests passing across 4 providers

Mario Zechner · 2025-08-25 15:54:26 +02:00

7a6852081d

feat(ai): Add proper thinking support for Gemini 2.5 models

- Added thinkingConfig with includeThoughts and thinkingBudget support
- Use part.thought boolean flag to detect thinking content per API docs
- Capture and preserve thought signatures for multi-turn function calling
- Added supportsThinking() check for Gemini 2.5 series models
- Updated example to demonstrate thinking configuration
- Handle SDK type limitations with proper type assertions

Mario Zechner · 2025-08-25 10:26:23 +02:00

3e1422d3d7

feat(ai): Implement Gemini provider with streaming and tool support

- Added GeminiLLM provider implementation with GoogleGenerativeAI SDK
- Supports streaming with text/thinking content and completion signals
- Handles Gemini's parts-based content system (text, thought, functionCall)
- Implements tool/function calling with proper format conversion
- Maps between unified types and Gemini-specific formats (model vs assistant role)
- Added test example matching other provider patterns
- Fixed typo in AssistantMessage type (stopResaon -> stopReason) across all providers

Mario Zechner · 2025-08-24 20:41:32 +02:00

a8ba19f0b4

refactor(ai): Add completion signal to onText/onThinking callbacks

- Update LLMOptions interface to include completion boolean parameter
- Modify all providers to signal when text/thinking blocks are complete
- Update examples to handle the completion parameter
- Move documentation files to docs/ directory

Mario Zechner · 2025-08-24 20:33:26 +02:00

cb4c32faaa

feat(ai): Add OpenAI Completions and Responses API providers

- Implement OpenAICompletionsLLM for Chat Completions API with streaming
- Implement OpenAIResponsesLLM for Responses API with reasoning support
- Update types to use LLM/Context instead of AI/Request
- Add support for reasoning tokens, tool calls, and streaming
- Create test examples for both OpenAI providers
- Update Anthropic provider to match new interface

Mario Zechner · 2025-08-24 20:18:10 +02:00

8364ecde4a

feat(ai): Implement unified AI API with Anthropic provider

- Define clean API with complete() method and callbacks for streaming
- Add comprehensive type system for messages, tools, and usage
- Implement AnthropicAI provider with full feature support:
  - Thinking/reasoning with signatures
  - Tool calling with parallel execution
  - Streaming via callbacks (onText, onThinking)
  - Proper error handling and stop reasons
  - Cache tracking for input/output tokens
- Add working test/example demonstrating tool execution flow
- Support for system prompts, temperature, max tokens
- Proper message role types: user, assistant, toolResult

Mario Zechner · 2025-08-17 23:30:20 +02:00

e5aedfed29

46 Commits