Roger Barreto
c3eca2567a
* Fix FoundryAgents_Step15_ComputerUse sample for Azure Agents API The Azure Agents API rejects previous_response_id alongside computer_call_output items, unlike the vanilla OpenAI Responses API. This fix: - Send all prior response output items (reasoning, computer_call, etc.) as input items in follow-up calls so the API has full conversation context - Create a fresh session per call to avoid ConversationId/previous_response_id - Use currentCallId instead of initialCallId for computer_call_output - Clear ContinuationToken after polling to prevent stale tokens - Remove unused initialCallId tracking variable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address comments --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Using Computer Use Tool with AI Agents
This sample demonstrates how to use the computer use tool with AI agents. The computer use tool allows agents to interact with a computer environment by viewing the screen, controlling the mouse and keyboard, and performing various actions to help complete tasks.
Note
Azure Agents API vs. vanilla OpenAI Responses API behavior: The Azure Agents API rejects requests that include
previous_response_idalongsidecomputer_call_outputitems — unlike the vanilla OpenAI Responses API, which accepts them. This sample works around the limitation by creating a fresh session for each follow-up call (so noprevious_response_idis carried over) and re-sending all prior response output items (reasoning, computer_call, etc.) as input items to preserve full conversation context. Additionally, the sample uses the currentCallIdfrom each computer call response (not the initial one) and clears theContinuationTokenafter polling completes to prevent stale tokens from affecting subsequent requests.
What this sample demonstrates
- Creating agents with computer use capabilities
- Using HostedComputerTool (MEAI abstraction)
- Using native SDK computer use tools (ResponseTool.CreateComputerTool)
- Extracting computer action information from agent responses
- Handling computer tool results (text output and screenshots)
- Managing agent lifecycle (creation and deletion)
Prerequisites
Before you begin, ensure you have the following prerequisites:
- .NET 10 SDK or later
- Azure Foundry service endpoint and deployment configured
- Azure CLI installed and authenticated (for Azure credential authentication)
Note: This demo uses Azure CLI credentials for authentication. Make sure you're logged in with az login and have access to the Azure Foundry resource. For more information, see the Azure CLI documentation.
Set the following environment variables:
$env:AZURE_FOUNDRY_PROJECT_ENDPOINT="https://your-foundry-service.services.ai.azure.com/api/projects/your-foundry-project" # Replace with your Azure Foundry resource endpoint
$env:AZURE_FOUNDRY_PROJECT_DEPLOYMENT_NAME="computer-use-preview" # Optional, defaults to computer-use-preview
Run the sample
Navigate to the FoundryAgents sample directory and run:
cd dotnet/samples/GettingStarted/FoundryAgents
dotnet run --project .\FoundryAgents_Step15_ComputerUse
Expected behavior
The sample will:
- Create two agents with computer use capabilities:
- Option 1: Using HostedComputerTool (MEAI abstraction)
- Option 2: Using native SDK computer use tools
- Run the agent with a task: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."
- The agent will use the computer use tool to:
- Interpret the screenshots
- Issue action requests based on the task
- Analyze the search results for "OpenAI news" from the screenshots.
- Extract and display the computer actions performed
- Display the results from the computer tool execution
- Display the final response from the agent
- Clean up resources by deleting both agents