mirror of
https://github.com/microsoft/agent-framework.git
synced 2026-06-16 21:04:09 +08:00
Eduard van Valkenburg
977c3adfb2
* python: replace pre-commit with prek, add PEP 723 script deps, clean up dev dependencies - Replace pre-commit with prek (Rust-native, faster pre-commit alternative) - Move supported hooks to repo: builtin for zero-clone speed - Add new builtin hooks: trailing-whitespace, check-merge-conflict, detect-private-key, check-added-large-files - Update all hook versions to latest (pre-commit-hooks v6, pyupgrade v3.21.2, bandit 1.9.3, uv-pre-commit 0.10.0) - Add PEP 723 inline script metadata to 34 samples with external deps - Remove autogen-agentchat/autogen-ext from dev deps (now declared per-sample) - Remove unused dev deps: pytest-env, tomli-w - Add agent-framework-core>=1.0.0b260130 lower bound to all 21 packages - Update CI workflow to use j178/prek-action - Update docs: DEV_SETUP.md, AGENTS.md, CODING_STANDARD.md, SAMPLE_GUIDELINES.md * updated lock * python: fix prek config paths for local execution and CI workflow Remove global 'files: ^python/' filter and strip python/ prefix from all path patterns in .pre-commit-config.yaml so prek finds files when run from the python/ directory. Update CI workflow to use --cd python instead of --config path. Include trailing whitespace fixes and dev dependency cleanup. * python: move helper scripts to scripts/ folder and exclude from checks * python: exclude AGENTS.md from prek markdown code lint * python: exclude AGENTS.md and azure_ai_search sample from markdown lint * fix m365 sample * python: ignore CPY rule for samples with PEP 723 headers * fix in dev_setup * python: replace aiofiles with regular open in samples * python: suppress reportUnusedImport in markdown code block checker * python: use samples pyright config for markdown code block checker Write a temp pyrightconfig.json matching pyrightconfig.samples.json rules (typeCheckingMode=off, only reportMissingImports and reportAttributeAccessIssue). Filter output to only fail on these rules since syntax-level errors (top-level await, undefined vars) are expected in README documentation snippets. * python: use markdown-code-lint with fixed globs instead of prek file list The prek-markdown-code-lint task received all changed files including non-README markdown and files with pre-existing broken imports. Replace with the standard markdown-code-lint task which uses the correct glob patterns (README.md, packages/**/README.md, samples/**/*.md). * python: exclude READMEs with pre-existing broken imports from markdown lint * python: fix broken README code snippets instead of excluding them - ag-ui: replace TextContent (removed) with content.type == 'text' - durabletask: fix import path to durabletask.worker.TaskHubGrpcWorker - orchestrations: use constructor params instead of .participants() method - observability: mark deprecated code blocks as plain text, filter reportMissingImports to agent_framework modules only - remove README excludes from markdown-code-lint task * add revision to gaia download * feat(python): parallelize checks across packages Run (package × task) cross-product in parallel using ThreadPoolExecutor and subprocesses. Key changes: - Add scripts/task_runner.py with shared parallel execution engine - Update run_tasks_in_packages_if_exists.py to accept multiple tasks - Update run_tasks_in_changed_packages.py with --files flag and parallel support - Add check-packages poe task (fmt+lint+pyright+mypy in parallel) - Add prek-markdown-code-lint and prek-samples-check with change detection - Split CI code quality workflow into parallel prek and mypy jobs - Update DEV_SETUP.md to document new parallel behavior Core package changes still trigger checks on all packages. * feat(ci): split code quality into 4 parallel jobs Split the single prek job into parallel jobs: - pre-commit-hooks: lightweight hooks (SKIP=poe-check) - package-checks: fmt/lint/pyright/mypy via check-packages - samples-markdown: samples-lint, samples-syntax, markdown-code-lint - mypy: change-detected mypy checks All 4 jobs run concurrently (×2 Python versions = 8 runners). * feat(ci): use only Python 3.10 for code quality checks * refactor(python): add future annotations and remove quoted types Add `from __future__ import annotations` to 93 package files that used quoted string annotations, then run pyupgrade --py310-plus to remove the now-unnecessary quotes. Fixes https://github.com/microsoft/agent-framework/issues/3578
341 lines
11 KiB
Markdown
341 lines
11 KiB
Markdown
# Dev Setup
|
||
|
||
This document describes how to setup your environment with Python and uv,
|
||
if you're working on new features or a bug fix for Agent Framework, or simply
|
||
want to run the tests included.
|
||
|
||
For coding standards and conventions, see [CODING_STANDARD.md](CODING_STANDARD.md).
|
||
|
||
## System setup
|
||
|
||
We are using a tool called [poethepoet](https://github.com/nat-n/poethepoet) for task management and [uv](https://github.com/astral-sh/uv) for dependency management. At the [end of this document](#available-poe-tasks), you will find the available Poe tasks.
|
||
|
||
## If you're on WSL
|
||
|
||
Check that you've cloned the repository to `~/workspace` or a similar folder.
|
||
Avoid `/mnt/c/` and prefer using your WSL user's home directory.
|
||
|
||
Ensure you have the WSL extension for VSCode installed.
|
||
|
||
## Using uv
|
||
|
||
uv allows us to use AF from the local files, without worrying about paths, as
|
||
if you had AF pip package installed.
|
||
|
||
To install AF and all the required tools in your system, first, navigate to the directory containing
|
||
this DEV_SETUP using your chosen shell.
|
||
|
||
### For windows (non-WSL)
|
||
|
||
Check the [uv documentation](https://docs.astral.sh/uv/getting-started/installation/) for the installation instructions. At the time of writing this is the command to install uv:
|
||
|
||
```powershell
|
||
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||
```
|
||
|
||
### For WSL, Linux or MacOS
|
||
|
||
Check the [uv documentation](https://docs.astral.sh/uv/getting-started/installation/) for the installation instructions. At the time of writing this is the command to install uv:
|
||
|
||
```bash
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||
```
|
||
|
||
### Alternative for MacOS
|
||
|
||
For MacOS users, Homebrew provides an easy installation of uv with the [uv Formulae](https://formulae.brew.sh/formula/uv)
|
||
|
||
```bash
|
||
brew install uv
|
||
```
|
||
|
||
|
||
### After installing uv
|
||
|
||
You can then run the following commands manually:
|
||
|
||
```bash
|
||
# Install Python 3.10, 3.11, 3.12, and 3.13
|
||
uv python install 3.10 3.11 3.12 3.13
|
||
# Create a virtual environment with Python 3.10 (you can change this to 3.11, 3.12 or 3.13)
|
||
$PYTHON_VERSION = "3.10"
|
||
uv venv --python $PYTHON_VERSION
|
||
# Install AF and all dependencies
|
||
uv sync --dev
|
||
# Install all the tools and dependencies
|
||
uv run poe install
|
||
# Install prek hooks
|
||
uv run poe prek-install
|
||
```
|
||
|
||
Alternatively, you can reinstall the venv, pacakges, dependencies and prek hooks with a single command (but this requires poe in the current env), this is especially useful if you want to switch python versions:
|
||
|
||
```bash
|
||
uv run poe setup -p 3.13
|
||
```
|
||
|
||
You can then run different commands through Poe the Poet, use `uv run poe` to discover which ones.
|
||
|
||
## VSCode Setup
|
||
|
||
Install the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for VSCode.
|
||
|
||
Open the `python` folder in [VSCode](https://code.visualstudio.com/docs/editor/workspaces).
|
||
> The workspace for python should be rooted in the `./python` folder.
|
||
|
||
Open any of the `.py` files in the project and run the `Python: Select Interpreter`
|
||
command from the command palette. Make sure the virtual env (default path is `.venv`) created by `uv` is selected.
|
||
|
||
## LLM setup
|
||
|
||
Make sure you have an
|
||
[OpenAI API Key](https://platform.openai.com) or
|
||
[Azure OpenAI service key](https://learn.microsoft.com/azure/cognitive-services/openai/quickstart?pivots=rest-api)
|
||
|
||
There are two methods to manage keys, secrets, and endpoints:
|
||
|
||
1. Store them in environment variables. AF Python leverages pydantic settings to load keys, secrets, and endpoints from the environment.
|
||
> When you are using VSCode and have the python extension setup, it automatically loads environment variables from a `.env` file, so you don't have to manually set them in the terminal.
|
||
> During runtime on different platforms, environment settings set as part of the deployments should be used.
|
||
|
||
2. Store them in a separate `.env` file, like `dev.env`, you can then pass that name into the constructor for most services, to the `env_file_path` parameter, see below.
|
||
> Make sure to add `*.env` to your `.gitignore` file.
|
||
|
||
### Example for file-based setup with OpenAI Chat Completions
|
||
To configure a `.env` file with just the keys needed for OpenAI Chat Completions, you can create a `openai.env` (this name is just as an example, a single `.env` with all required keys is more common) file in the root of the `python` folder with the following content:
|
||
|
||
Content of `.env` or `openai.env`:
|
||
|
||
```env
|
||
OPENAI_API_KEY=""
|
||
OPENAI_CHAT_MODEL_ID="gpt-4o-mini"
|
||
```
|
||
|
||
You will then configure the ChatClient class with the keyword argument `env_file_path`:
|
||
|
||
```python
|
||
from agent_framework.openai import OpenAIChatClient
|
||
|
||
chat_client = OpenAIChatClient(env_file_path="openai.env")
|
||
```
|
||
|
||
## Tests
|
||
|
||
All the tests are located in the `tests` folder of each package. There are tests that are marked with a `@skip_if_..._integration_tests_disabled` decorator, these are integration tests that require an external service to be running, like OpenAI or Azure OpenAI.
|
||
|
||
If you want to run these tests, you need to set the environment variable `RUN_INTEGRATION_TESTS` to `true` and have the appropriate key per services set in your environment or in a `.env` file.
|
||
|
||
Alternatively, you can run them using VSCode Tasks. Open the command palette
|
||
(`Ctrl+Shift+P`) and type `Tasks: Run Task`. Select `Test` from the list.
|
||
|
||
If you want to run the tests for a single package, you can use the `uv run poe test` command with the package name as an argument. For example, to run the tests for the `agent_framework` package, you can use:
|
||
|
||
```bash
|
||
uv run poe --directory packages/core test
|
||
```
|
||
|
||
These commands also output the coverage report.
|
||
|
||
## Code quality checks
|
||
|
||
To run the same checks that run during a commit and the GitHub Action `Python Code Quality`, you can use this command, from the [python](../python) folder:
|
||
|
||
```bash
|
||
uv run poe check
|
||
```
|
||
|
||
Ideally you should run these checks before committing any changes, when you install using the instructions above the prek hooks should be installed already.
|
||
|
||
## Code Coverage
|
||
|
||
We try to maintain a high code coverage for the project. To run the code coverage on the unit tests, you can use the following command:
|
||
|
||
```bash
|
||
uv run poe test
|
||
```
|
||
|
||
This will show you which files are not covered by the tests, including the specific lines not covered. Make sure to consider the untested lines from the code you are working on, but feel free to add other tests as well, that is always welcome!
|
||
|
||
## Catching up with the latest changes
|
||
|
||
There are many people committing to Semantic Kernel, so it is important to keep your local repository up to date. To do this, you can run the following commands:
|
||
|
||
```bash
|
||
git fetch upstream main
|
||
git rebase upstream/main
|
||
git push --force-with-lease
|
||
```
|
||
|
||
or:
|
||
|
||
```bash
|
||
git fetch upstream main
|
||
git merge upstream/main
|
||
git push
|
||
```
|
||
|
||
This is assuming the upstream branch refers to the main repository. If you have a different name for the upstream branch, you can replace `upstream` with the name of your upstream branch.
|
||
|
||
After running the rebase command, you may need to resolve any conflicts that arise. If you are unsure how to resolve a conflict, please refer to the [GitHub's documentation on resolving conflicts](https://docs.github.com/en/get-started/using-git/resolving-merge-conflicts-after-a-git-rebase), or for [VSCode](https://code.visualstudio.com/docs/sourcecontrol/overview#_merge-conflicts).
|
||
|
||
# Task automation
|
||
|
||
## Available Poe Tasks
|
||
This project uses [poethepoet](https://github.com/nat-n/poethepoet) for task management and [uv](https://github.com/astral-sh/uv) for dependency management.
|
||
|
||
### Setup and Installation
|
||
|
||
Once uv is installed, and you do not yet have a virtual environment setup:
|
||
|
||
```bash
|
||
uv venv
|
||
```
|
||
|
||
and then you can run the following tasks:
|
||
```bash
|
||
uv sync --all-extras --dev
|
||
```
|
||
|
||
After this initial setup, you can use the following tasks to manage your development environment. It is advised to use the following setup command since that also installs the prek hooks.
|
||
|
||
#### `setup`
|
||
Set up the development environment with a virtual environment, install dependencies and prek hooks:
|
||
```bash
|
||
uv run poe setup
|
||
# or with specific Python version
|
||
uv run poe setup --python 3.12
|
||
```
|
||
|
||
#### `install`
|
||
Install all dependencies including extras and dev dependencies, including updates:
|
||
```bash
|
||
uv run poe install
|
||
```
|
||
|
||
#### `venv`
|
||
Create a virtual environment with specified Python version or switch python version:
|
||
```bash
|
||
uv run poe venv
|
||
# or with specific Python version
|
||
uv run poe venv --python 3.12
|
||
```
|
||
|
||
#### `prek-install`
|
||
Install prek hooks:
|
||
```bash
|
||
uv run poe prek-install
|
||
```
|
||
|
||
### Code Quality and Formatting
|
||
|
||
Each of the following tasks run against both the main `agent-framework` package and the extension packages in parallel, ensuring consistent code quality across the project.
|
||
|
||
#### `fmt` (format)
|
||
Format code using ruff (runs in parallel across all packages):
|
||
```bash
|
||
uv run poe fmt
|
||
```
|
||
|
||
#### `lint`
|
||
Run linting checks and fix issues (runs in parallel across all packages):
|
||
```bash
|
||
uv run poe lint
|
||
```
|
||
|
||
#### `pyright`
|
||
Run Pyright type checking (runs in parallel across all packages):
|
||
```bash
|
||
uv run poe pyright
|
||
```
|
||
|
||
#### `mypy`
|
||
Run MyPy type checking (runs in parallel across all packages):
|
||
```bash
|
||
uv run poe mypy
|
||
```
|
||
|
||
#### `typing`
|
||
Run both Pyright and MyPy type checking:
|
||
```bash
|
||
uv run poe typing
|
||
```
|
||
|
||
### Code Validation
|
||
|
||
#### `markdown-code-lint`
|
||
Lint markdown code blocks:
|
||
```bash
|
||
uv run poe markdown-code-lint
|
||
```
|
||
|
||
### Comprehensive Checks
|
||
|
||
#### `check-packages`
|
||
Run all package-level quality checks (format, lint, pyright, mypy) in parallel across all packages. This runs the full cross-product of (package × check) concurrently:
|
||
```bash
|
||
uv run poe check-packages
|
||
```
|
||
|
||
#### `check`
|
||
Run all quality checks including package checks, samples, tests and markdown lint:
|
||
```bash
|
||
uv run poe check
|
||
```
|
||
|
||
### Testing
|
||
|
||
#### `test`
|
||
Run unit tests with coverage by invoking the `test` task in each package in parallel:
|
||
```bash
|
||
uv run poe test
|
||
```
|
||
|
||
To run tests for a specific package only, use the `--directory` flag:
|
||
```bash
|
||
# Run tests for the core package
|
||
uv run --directory packages/core poe test
|
||
|
||
# Run tests for the azure-ai package
|
||
uv run --directory packages/azure-ai poe test
|
||
```
|
||
|
||
#### `all-tests`
|
||
Run all tests in a single pytest invocation across all packages in parallel (excluding lab and devui). This is faster than `test` as it uses pytest's parallel execution:
|
||
```bash
|
||
uv run poe all-tests
|
||
```
|
||
|
||
#### `all-tests-cov`
|
||
Same as `all-tests` but with coverage reporting enabled:
|
||
```bash
|
||
uv run poe all-tests-cov
|
||
```
|
||
|
||
### Building and Publishing
|
||
|
||
#### `build`
|
||
Build all packages:
|
||
```bash
|
||
uv run poe build
|
||
```
|
||
|
||
#### `clean-dist`
|
||
Clean the dist directories:
|
||
```bash
|
||
uv run poe clean-dist
|
||
```
|
||
|
||
#### `publish`
|
||
Publish packages to PyPI:
|
||
```bash
|
||
uv run poe publish
|
||
```
|
||
|
||
## Prek Hooks
|
||
|
||
Prek hooks run automatically on commit and execute a subset of the checks on changed files only. Package-level checks (fmt, lint, pyright) run in parallel but only for packages with changed files. Markdown and sample checks are skipped when no relevant files were changed. If the `core` package is changed, all packages are checked. You can also run all checks using prek directly:
|
||
|
||
```bash
|
||
uv run prek run -a
|
||
```
|