Python: updated integration tests and guidance (#4181)

* updated integration tests and guidance

* fixed merge test

* updated integration tests

* fix: remove duplicate --dist loadfile flag from pytest-xdist config

Only one --dist mode can be active at a time; the second value silently
overrides the first. Keep --dist worksteal (dynamic load balancing) and
remove the redundant --dist loadfile from all workflow files and
pyproject.toml configs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add keep-in-sync notes for merge and integration test workflows

Both python-merge-tests.yml and python-integration-tests.yml share the
same parallel job structure. Added sync reminders in workflow file
comments, the python-testing SKILL.md, and CODING_STANDARD.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor: remove RUN_INTEGRATION_TESTS flag

Integration test gating now uses two mechanisms:
- `@pytest.mark.integration` for test selection via `-m` filtering
- `skip_if_*_disabled` for credential/service availability checks

The RUN_INTEGRATION_TESTS env var was redundant since the marker handles
selection and the skip decorators already check for actual credentials.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: sync missing env vars from merge-tests to integration-tests

Add OPENAI_EMBEDDINGS_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME
to python-integration-tests.yml to match python-merge-tests.yml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: remove remaining RUN_INTEGRATION_TESTS from embedding tests and docs

Missed test_openai_embedding_client.py and vector-stores README in the
earlier cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* set functions tests to 3.10

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

python/packages/durabletask/tests/integration_tests/.env.example

@@ -11,7 +11,3 @@ TASKHUB=default
 # Redis Configuration (for streaming tests)
 REDIS_CONNECTION_STRING=redis://localhost:6379
 REDIS_STREAM_TTL_MINUTES=10
-
-# Integration Test Control
-# Set to 'true' to enable integration tests
-RUN_INTEGRATION_TESTS=true

python/packages/durabletask/tests/integration_tests/README.md

@@ -16,7 +16,6 @@ Required variables:
 - `AZURE_OPENAI_ENDPOINT`
 - `AZURE_OPENAI_CHAT_DEPLOYMENT_NAME`
 - `AZURE_OPENAI_API_KEY` (optional if using Azure CLI authentication)
-- `RUN_INTEGRATION_TESTS` (set to `true`)
 - `ENDPOINT` (default: http://localhost:8080)
 - `TASKHUB` (default: default)
 
@@ -75,7 +74,7 @@ pytestmark = [
 ## Troubleshooting
 
 **Tests are skipped:**
-Ensure `RUN_INTEGRATION_TESTS=true` is set in your `.env` file.
+Ensure the required environment variables (e.g., `AZURE_OPENAI_ENDPOINT`) are set in your `.env` file.
 
 **DTS connection failed:**
 Check that the DTS emulator container is running: `docker ps | grep dts-emulator`

python/packages/durabletask/tests/integration_tests/conftest.py

@@ -289,9 +289,6 @@ def pytest_configure(config: pytest.Config) -> None:
 
 def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
     """Skip tests based on markers and environment availability."""
-    run_integration = os.getenv("RUN_INTEGRATION_TESTS", "false").lower() == "true"
-    skip_integration = pytest.mark.skip(reason="RUN_INTEGRATION_TESTS not set to 'true'")
-
     # Check Azure OpenAI environment variables
     azure_openai_vars = ["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
     azure_openai_available = all(os.getenv(var) for var in azure_openai_vars)
@@ -308,8 +305,6 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item
     skip_redis = pytest.mark.skip(reason="Redis is not available at redis://localhost:6379")
 
     for item in items:
-        if "integration_test" in item.keywords and not run_integration:
-            item.add_marker(skip_integration)
         if "requires_azure_openai" in item.keywords and not azure_openai_available:
             item.add_marker(skip_azure_openai)
         if "requires_dts" in item.keywords and not dts_available:

python/packages/durabletask/tests/integration_tests/test_01_dt_single_agent.py

@@ -14,6 +14,8 @@ import pytest
 
 # Module-level markers - applied to all tests in this module
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("01_single_agent"),
     pytest.mark.integration_test,
     pytest.mark.requires_azure_openai,

python/packages/durabletask/tests/integration_tests/test_02_dt_multi_agent.py

@@ -18,6 +18,8 @@ MATH_AGENT_NAME: str = "MathAgent"
 
 # Module-level markers - applied to all tests in this module
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("02_multi_agent"),
     pytest.mark.integration_test,
     pytest.mark.requires_azure_openai,

python/packages/durabletask/tests/integration_tests/test_03_dt_single_agent_streaming.py

@@ -34,6 +34,8 @@ from redis_stream_response_handler import RedisStreamResponseHandler  # type: ig
 
 # Module-level markers - applied to all tests in this file
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("03_single_agent_streaming"),
     pytest.mark.integration_test,
     pytest.mark.requires_azure_openai,

python/packages/durabletask/tests/integration_tests/test_04_dt_single_agent_orchestration_chaining.py

@@ -23,6 +23,8 @@ logging.basicConfig(level=logging.WARNING)
 
 # Module-level markers - applied to all tests in this module
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("04_single_agent_orchestration_chaining"),
     pytest.mark.integration_test,
     pytest.mark.requires_azure_openai,

python/packages/durabletask/tests/integration_tests/test_05_dt_multi_agent_orchestration_concurrency.py

@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
 
 # Module-level markers
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("05_multi_agent_orchestration_concurrency"),
     pytest.mark.integration_test,
     pytest.mark.requires_dts,

python/packages/durabletask/tests/integration_tests/test_06_dt_multi_agent_orchestration_conditionals.py

@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
 
 # Module-level markers
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("06_multi_agent_orchestration_conditionals"),
     pytest.mark.integration_test,
     pytest.mark.requires_dts,

python/packages/durabletask/tests/integration_tests/test_07_dt_single_agent_orchestration_hitl.py

@@ -24,6 +24,8 @@ logging.basicConfig(level=logging.WARNING)
 
 # Module-level markers
 pytestmark = [
+    pytest.mark.flaky,
+    pytest.mark.integration,
     pytest.mark.sample("07_single_agent_orchestration_hitl"),
     pytest.mark.integration_test,
     pytest.mark.requires_dts,