Files
agent-framework/python/scripts/run_tasks_in_changed_packages.py
T
Eduard van Valkenburg 977c3adfb2 Python: replace pre-commit with prek, add PEP 723 script deps, clean up dev dependencies (#3748)
* python: replace pre-commit with prek, add PEP 723 script deps, clean up dev dependencies

- Replace pre-commit with prek (Rust-native, faster pre-commit alternative)
- Move supported hooks to repo: builtin for zero-clone speed
- Add new builtin hooks: trailing-whitespace, check-merge-conflict, detect-private-key, check-added-large-files
- Update all hook versions to latest (pre-commit-hooks v6, pyupgrade v3.21.2, bandit 1.9.3, uv-pre-commit 0.10.0)
- Add PEP 723 inline script metadata to 34 samples with external deps
- Remove autogen-agentchat/autogen-ext from dev deps (now declared per-sample)
- Remove unused dev deps: pytest-env, tomli-w
- Add agent-framework-core>=1.0.0b260130 lower bound to all 21 packages
- Update CI workflow to use j178/prek-action
- Update docs: DEV_SETUP.md, AGENTS.md, CODING_STANDARD.md, SAMPLE_GUIDELINES.md

* updated lock

* python: fix prek config paths for local execution and CI workflow

Remove global 'files: ^python/' filter and strip python/ prefix from all path patterns in .pre-commit-config.yaml so prek finds files when run from the python/ directory. Update CI workflow to use --cd python instead of --config path. Include trailing whitespace fixes and dev dependency cleanup.

* python: move helper scripts to scripts/ folder and exclude from checks

* python: exclude AGENTS.md from prek markdown code lint

* python: exclude AGENTS.md and azure_ai_search sample from markdown lint

* fix m365 sample

* python: ignore CPY rule for samples with PEP 723 headers

* fix in dev_setup

* python: replace aiofiles with regular open in samples

* python: suppress reportUnusedImport in markdown code block checker

* python: use samples pyright config for markdown code block checker

Write a temp pyrightconfig.json matching pyrightconfig.samples.json rules (typeCheckingMode=off, only reportMissingImports and reportAttributeAccessIssue). Filter output to only fail on these rules since syntax-level errors (top-level await, undefined vars) are expected in README documentation snippets.

* python: use markdown-code-lint with fixed globs instead of prek file list

The prek-markdown-code-lint task received all changed files including non-README markdown and files with pre-existing broken imports. Replace with the standard markdown-code-lint task which uses the correct glob patterns (README.md, packages/**/README.md, samples/**/*.md).

* python: exclude READMEs with pre-existing broken imports from markdown lint

* python: fix broken README code snippets instead of excluding them

- ag-ui: replace TextContent (removed) with content.type == 'text'
- durabletask: fix import path to durabletask.worker.TaskHubGrpcWorker
- orchestrations: use constructor params instead of .participants() method
- observability: mark deprecated code blocks as plain text, filter
  reportMissingImports to agent_framework modules only
- remove README excludes from markdown-code-lint task

* add revision to gaia download

* feat(python): parallelize checks across packages

Run (package × task) cross-product in parallel using ThreadPoolExecutor
and subprocesses. Key changes:

- Add scripts/task_runner.py with shared parallel execution engine
- Update run_tasks_in_packages_if_exists.py to accept multiple tasks
- Update run_tasks_in_changed_packages.py with --files flag and parallel support
- Add check-packages poe task (fmt+lint+pyright+mypy in parallel)
- Add prek-markdown-code-lint and prek-samples-check with change detection
- Split CI code quality workflow into parallel prek and mypy jobs
- Update DEV_SETUP.md to document new parallel behavior

Core package changes still trigger checks on all packages.

* feat(ci): split code quality into 4 parallel jobs

Split the single prek job into parallel jobs:
- pre-commit-hooks: lightweight hooks (SKIP=poe-check)
- package-checks: fmt/lint/pyright/mypy via check-packages
- samples-markdown: samples-lint, samples-syntax, markdown-code-lint
- mypy: change-detected mypy checks

All 4 jobs run concurrently (×2 Python versions = 8 runners).

* feat(ci): use only Python 3.10 for code quality checks

* refactor(python): add future annotations and remove quoted types

Add `from __future__ import annotations` to 93 package files that
used quoted string annotations, then run pyupgrade --py310-plus to
remove the now-unnecessary quotes.

Fixes https://github.com/microsoft/agent-framework/issues/3578
2026-02-09 17:51:01 +00:00

82 lines
3.2 KiB
Python

# Copyright (c) Microsoft. All rights reserved.
"""Run task(s) only in packages that have changed files, in parallel by default."""
import argparse
from pathlib import Path
from rich import print
from task_runner import build_work_items, discover_projects, run_tasks
def get_changed_packages(projects: list[Path], changed_files: list[str], workspace_root: Path) -> set[Path]:
"""Determine which packages have changed files."""
changed_packages: set[Path] = set()
core_package_changed = False
for file_path in changed_files:
# Strip 'python/' prefix if present (when git diff is run from repo root)
file_path_str = str(file_path)
if file_path_str.startswith("python/"):
file_path_str = file_path_str[7:] # Remove 'python/' prefix
# Convert to absolute path if relative
abs_path = Path(file_path_str)
if not abs_path.is_absolute():
abs_path = workspace_root / file_path_str
# Check which package this file belongs to
for project in projects:
project_abs = workspace_root / project
try:
# Check if the file is within this project directory
abs_path.relative_to(project_abs)
changed_packages.add(project)
# Check if the core package was changed
if project == Path("packages/core"):
core_package_changed = True
break
except ValueError:
# File is not in this project
continue
# If core package changed, check all packages
if core_package_changed:
print("[yellow]Core package changed - checking all packages[/yellow]")
return set(projects)
return changed_packages
def main() -> None:
parser = argparse.ArgumentParser(description="Run task(s) in changed packages, in parallel by default.")
parser.add_argument("tasks", nargs="+", help="Task name(s) to run")
parser.add_argument("--files", nargs="*", default=None, help="Changed files to determine which packages to run")
parser.add_argument("--seq", action="store_true", help="Run sequentially instead of in parallel")
args = parser.parse_args()
pyproject_file = Path(__file__).parent.parent / "pyproject.toml"
workspace_root = pyproject_file.parent
projects = discover_projects(pyproject_file)
# Determine which packages to check
if not args.files or args.files == ["."]:
task_list = ", ".join(args.tasks)
print(f"[yellow]No specific files provided, running {task_list} in all packages[/yellow]")
target_packages = sorted(set(projects))
else:
changed_packages = get_changed_packages(projects, args.files, workspace_root)
if changed_packages:
print(f"[cyan]Detected changes in packages: {', '.join(str(p) for p in sorted(changed_packages))}[/cyan]")
else:
print(f"[yellow]No changes detected in any package, skipping[/yellow]")
return
target_packages = sorted(changed_packages)
work_items = build_work_items(target_packages, args.tasks)
run_tasks(work_items, workspace_root, sequential=args.seq)
if __name__ == "__main__":
main()