Python: replace pre-commit with prek, add PEP 723 script deps, clean up dev dependencies (#3748)

* python: replace pre-commit with prek, add PEP 723 script deps, clean up dev dependencies

- Replace pre-commit with prek (Rust-native, faster pre-commit alternative)
- Move supported hooks to repo: builtin for zero-clone speed
- Add new builtin hooks: trailing-whitespace, check-merge-conflict, detect-private-key, check-added-large-files
- Update all hook versions to latest (pre-commit-hooks v6, pyupgrade v3.21.2, bandit 1.9.3, uv-pre-commit 0.10.0)
- Add PEP 723 inline script metadata to 34 samples with external deps
- Remove autogen-agentchat/autogen-ext from dev deps (now declared per-sample)
- Remove unused dev deps: pytest-env, tomli-w
- Add agent-framework-core>=1.0.0b260130 lower bound to all 21 packages
- Update CI workflow to use j178/prek-action
- Update docs: DEV_SETUP.md, AGENTS.md, CODING_STANDARD.md, SAMPLE_GUIDELINES.md

* updated lock

* python: fix prek config paths for local execution and CI workflow

Remove global 'files: ^python/' filter and strip python/ prefix from all path patterns in .pre-commit-config.yaml so prek finds files when run from the python/ directory. Update CI workflow to use --cd python instead of --config path. Include trailing whitespace fixes and dev dependency cleanup.

* python: move helper scripts to scripts/ folder and exclude from checks

* python: exclude AGENTS.md from prek markdown code lint

* python: exclude AGENTS.md and azure_ai_search sample from markdown lint

* fix m365 sample

* python: ignore CPY rule for samples with PEP 723 headers

* fix in dev_setup

* python: replace aiofiles with regular open in samples

* python: suppress reportUnusedImport in markdown code block checker

* python: use samples pyright config for markdown code block checker

Write a temp pyrightconfig.json matching pyrightconfig.samples.json rules (typeCheckingMode=off, only reportMissingImports and reportAttributeAccessIssue). Filter output to only fail on these rules since syntax-level errors (top-level await, undefined vars) are expected in README documentation snippets.

* python: use markdown-code-lint with fixed globs instead of prek file list

The prek-markdown-code-lint task received all changed files including non-README markdown and files with pre-existing broken imports. Replace with the standard markdown-code-lint task which uses the correct glob patterns (README.md, packages/**/README.md, samples/**/*.md).

* python: exclude READMEs with pre-existing broken imports from markdown lint

* python: fix broken README code snippets instead of excluding them

- ag-ui: replace TextContent (removed) with content.type == 'text'
- durabletask: fix import path to durabletask.worker.TaskHubGrpcWorker
- orchestrations: use constructor params instead of .participants() method
- observability: mark deprecated code blocks as plain text, filter
  reportMissingImports to agent_framework modules only
- remove README excludes from markdown-code-lint task

* add revision to gaia download

* feat(python): parallelize checks across packages

Run (package × task) cross-product in parallel using ThreadPoolExecutor
and subprocesses. Key changes:

- Add scripts/task_runner.py with shared parallel execution engine
- Update run_tasks_in_packages_if_exists.py to accept multiple tasks
- Update run_tasks_in_changed_packages.py with --files flag and parallel support
- Add check-packages poe task (fmt+lint+pyright+mypy in parallel)
- Add prek-markdown-code-lint and prek-samples-check with change detection
- Split CI code quality workflow into parallel prek and mypy jobs
- Update DEV_SETUP.md to document new parallel behavior

Core package changes still trigger checks on all packages.

* feat(ci): split code quality into 4 parallel jobs

Split the single prek job into parallel jobs:
- pre-commit-hooks: lightweight hooks (SKIP=poe-check)
- package-checks: fmt/lint/pyright/mypy via check-packages
- samples-markdown: samples-lint, samples-syntax, markdown-code-lint
- mypy: change-detected mypy checks

All 4 jobs run concurrently (×2 Python versions = 8 runners).

* feat(ci): use only Python 3.10 for code quality checks

* refactor(python): add future annotations and remove quoted types

Add `from __future__ import annotations` to 93 package files that
used quoted string annotations, then run pyupgrade --py310-plus to
remove the now-unnecessary quotes.

Fixes https://github.com/microsoft/agent-framework/issues/3578

python/scripts/task_runner.py

@@ -0,0 +1,150 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Shared utilities for running poe tasks across workspace packages in parallel."""
+
+import concurrent.futures
+import glob
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+import tomli
+from rich import print
+
+
+def discover_projects(workspace_pyproject_file: Path) -> list[Path]:
+    """Discover all workspace projects from pyproject.toml."""
+    with workspace_pyproject_file.open("rb") as f:
+        data = tomli.load(f)
+
+    projects = data["tool"]["uv"]["workspace"]["members"]
+    exclude = data["tool"]["uv"]["workspace"].get("exclude", [])
+
+    all_projects: list[Path] = []
+    for project in projects:
+        if "*" in project:
+            globbed = glob.glob(str(project), root_dir=workspace_pyproject_file.parent)
+            globbed_paths = [Path(p) for p in globbed]
+            all_projects.extend(globbed_paths)
+        else:
+            all_projects.append(Path(project))
+
+    for project in exclude:
+        if "*" in project:
+            globbed = glob.glob(str(project), root_dir=workspace_pyproject_file.parent)
+            globbed_paths = [Path(p) for p in globbed]
+            all_projects = [p for p in all_projects if p not in globbed_paths]
+        else:
+            all_projects = [p for p in all_projects if p != Path(project)]
+
+    return all_projects
+
+
+def extract_poe_tasks(file: Path) -> set[str]:
+    """Extract poe task names from a pyproject.toml file."""
+    with file.open("rb") as f:
+        data = tomli.load(f)
+
+    tasks = set(data.get("tool", {}).get("poe", {}).get("tasks", {}).keys())
+
+    # Check if there is an include too
+    include: str | None = data.get("tool", {}).get("poe", {}).get("include", None)
+    if include:
+        include_file = file.parent / include
+        if include_file.exists():
+            tasks = tasks.union(extract_poe_tasks(include_file))
+
+    return tasks
+
+
+def build_work_items(projects: list[Path], task_names: list[str]) -> list[tuple[Path, str]]:
+    """Build cross-product of (package, task) for packages that define the task."""
+    work_items: list[tuple[Path, str]] = []
+    for project in projects:
+        available_tasks = extract_poe_tasks(project / "pyproject.toml")
+        for task in task_names:
+            if task in available_tasks:
+                work_items.append((project, task))
+    return work_items
+
+
+def _run_task_subprocess(project: Path, task: str, workspace_root: Path) -> tuple[Path, str, int, str, str, float]:
+    """Run a single poe task in a project directory via subprocess."""
+    start = time.monotonic()
+    cwd = workspace_root / project
+    result = subprocess.run(
+        ["uv", "run", "poe", task],
+        cwd=cwd,
+        capture_output=True,
+        text=True,
+    )
+    elapsed = time.monotonic() - start
+    return (project, task, result.returncode, result.stdout, result.stderr, elapsed)
+
+
+def _run_sequential(work_items: list[tuple[Path, str]]) -> None:
+    """Run tasks sequentially using in-process PoeThePoet (streaming output)."""
+    from poethepoet.app import PoeThePoet
+
+    for project, task in work_items:
+        print(f"Running task {task} in {project}")
+        app = PoeThePoet(cwd=project)
+        result = app(cli_args=[task])
+        if result:
+            sys.exit(result)
+
+
+def _run_parallel(work_items: list[tuple[Path, str]], workspace_root: Path) -> None:
+    """Run all (package × task) combinations in parallel via subprocesses."""
+    max_workers = min(len(work_items), os.cpu_count() or 4)
+    failures: list[tuple[Path, str, str, str]] = []
+    completed = 0
+    total = len(work_items)
+
+    print(f"[cyan]Running {total} task(s) in parallel (max {max_workers} workers)...[/cyan]")
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+        futures = {
+            executor.submit(_run_task_subprocess, project, task, workspace_root): (project, task)
+            for project, task in work_items
+        }
+        for future in concurrent.futures.as_completed(futures):
+            project, task, returncode, stdout, stderr, elapsed = future.result()
+            completed += 1
+            progress = f"[{completed}/{total}]"
+            if returncode == 0:
+                print(f"  [green]✓[/green] {progress} {task} in {project} ({elapsed:.1f}s)")
+            else:
+                print(f"  [red]✗[/red] {progress} {task} in {project} ({elapsed:.1f}s)")
+                failures.append((project, task, stdout, stderr))
+
+    if failures:
+        print(f"\n[red]{len(failures)} task(s) failed:[/red]")
+        for project, task, stdout, stderr in failures:
+            print(f"\n[red]{'='*60}[/red]")
+            print(f"[red]FAILED: {task} in {project}[/red]")
+            if stdout.strip():
+                print(stdout)
+            if stderr.strip():
+                sys.stderr.write(stderr)
+        sys.exit(1)
+
+    print(f"\n[green]All {total} task(s) passed ✓[/green]")
+
+
+def run_tasks(work_items: list[tuple[Path, str]], workspace_root: Path, *, sequential: bool = False) -> None:
+    """Run work items either in parallel or sequentially.
+
+    Single items use in-process PoeThePoet for streaming output.
+    Multiple items use parallel subprocesses by default.
+    """
+    if not work_items:
+        print("[yellow]No matching tasks found in any package[/yellow]")
+        return
+
+    if sequential or len(work_items) == 1:
+        _run_sequential(work_items)
+    else:
+        _run_parallel(work_items, workspace_root)