Pytest Configuration and Testing Guide

This document describes the pytest configuration for the socialseed-e2e framework and how to run different types of tests.

Configuration Overview

The pytest configuration is defined in pyproject.toml under the [tool.pytest.ini_options] section.

Test Discovery

Test paths: tests/
Test files: test_*.py
Test classes: Test*
Test functions: test_*

Test Markers

We use markers to categorize tests for selective execution:

Marker	Description	Example Usage
`@pytest.mark.unit`	Fast, isolated unit tests	Testing individual functions
`@pytest.mark.integration`	Integration tests	Testing component interactions
`@pytest.mark.slow`	Tests that take significant time	Long-running operations
`@pytest.mark.cli`	CLI command tests	Testing CLI commands
`@pytest.mark.core`	Core framework tests	Testing framework internals
`@pytest.mark.e2e`	End-to-end tests	Full workflow tests
`@pytest.mark.mock_api`	Tests using mock API	Tests with Flask mock server

Running Tests by Marker

# Run only unit tests
pytest -m unit

# Run only integration tests
pytest -m integration

# Run only slow tests
pytest -m slow

# Run unit tests excluding slow ones
pytest -m "unit and not slow"

# Run multiple marker types
pytest -m "unit or cli"

# Run integration tests using mock API
pytest -m "integration and mock_api"

Default Options

The following options are applied by default:

-v: Verbose output
--tb=short: Short traceback format
--strict-markers: Fail on unknown markers

Coverage Configuration

Coverage is configured in pyproject.toml with the following settings:

Coverage Settings

Source: src/socialseed_e2e
Omit: Tests, cache, templates
Branch coverage: Enabled
Minimum coverage: 80%
HTML report: htmlcov/

Running Tests with Coverage

# Run all tests with coverage
pytest --cov=socialseed_e2e

# Run with coverage and generate HTML report
pytest --cov=socialseed_e2e --cov-report=html

# Run with coverage and generate XML report (for CI)
pytest --cov=socialseed_e2e --cov-report=xml

# Run with coverage and show missing lines
pytest --cov=socialseed_e2e --cov-report=term-missing

# Fail if coverage is below 80%
pytest --cov=socialseed_e2e --cov-fail-under=80

Common Test Commands

Basic Test Execution

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/unit/test_validators.py

# Run specific test class
pytest tests/unit/test_validators.py::TestValidateUrl

# Run specific test method
pytest tests/unit/test_validators.py::TestValidateUrl::test_valid_https_url

Running Test Suites

# Run all unit tests
pytest tests/unit/

# Run all integration tests
pytest tests/integration/

# Run CLI integration tests
pytest tests/integration/cli/

# Run mock API tests
pytest tests/integration/test_mock_api_integration.py

Debugging Tests

# Stop on first failure
pytest -x

# Stop after N failures
pytest --maxfail=3

# Run with debugger on failure
pytest --pdb

# Show local variables in tracebacks
pytest -l

# Show full traceback
pytest --tb=long

Parallel Test Execution

# Run tests in parallel (requires pytest-xdist)
pip install pytest-xdist
pytest -n auto

# Run with 4 workers
pytest -n 4

CI/CD Integration

GitHub Actions

The project includes GitHub Actions workflow configured in .github/workflows/ci.yml:

- name: Test with pytest
  run: pytest --cov=socialseed_e2e --cov-report=xml

- name: Upload coverage
  uses: codecov/codecov-action@v3
  with:
    file: ./coverage.xml
    fail_ci_if_error: true

Coverage Reporting (codecov.io)

Coverage reports are automatically uploaded to codecov.io when tests run in CI.

To view coverage reports:

Visit: https://codecov.io/gh/daironpf/socialseed-e2e
View trends, file coverage, and diff coverage on PRs

Test Organization

Unit Tests (`tests/unit/`)

Tests for individual components in isolation:

test_base_page.py - HTTP client tests
test_config_loader.py - Configuration management tests
test_loaders.py - Module loader tests
test_orchestrator.py - Test orchestration tests
test_validators.py - Validation helper tests
test_template_engine.py - Template system tests
test_imports_compatibility.py - Import system tests

All unit tests are marked with @pytest.mark.unit.

Integration Tests (`tests/integration/`)

Tests for component interactions:

cli/ - CLI command integration tests
test_mock_api_integration.py - Mock API integration tests

All integration tests are marked with @pytest.mark.integration.

Fixtures

Common fixtures are defined in tests/conftest.py:

Mock API Fixtures

mock_api_server - Session-scoped running Flask server
mock_api_url - Base URL string for mock API
mock_api_reset - Function-scoped data reset
sample_user_data - Sample user data for testing
admin_credentials - Admin user credentials
user_credentials - Regular user credentials

CLI Fixtures

cli_runner - Click test runner
temp_dir - Temporary directory for CLI tests

Writing Tests

Unit Test Example

import pytest

pytestmark = pytest.mark.unit

def test_validate_url():
    """Test URL validation."""
    from socialseed_e2e.utils.validators import validate_url

    result = validate_url("https://api.example.com")
    assert result == "https://api.example.com"

Integration Test Example

import pytest

pytestmark = pytest.mark.integration

def test_cli_init(cli_runner, temp_dir):
    """Test CLI init command."""
    from socialseed_e2e.cli import cli

    result = cli_runner.invoke(cli, ['init', str(temp_dir)])
    assert result.exit_code == 0
    assert (temp_dir / "services").exists()

Slow Test Example

import pytest

@pytest.mark.slow
def test_long_running_operation():
    """Test that takes significant time."""
    import time
    time.sleep(5)
    assert True

Best Practices

Use markers: Always mark tests with appropriate markers
Keep unit tests fast: Unit tests should run in milliseconds
Use fixtures: Leverage fixtures for setup/teardown
Test docstrings: Write clear docstrings explaining what the test verifies
Coverage: Aim for >80% coverage on new code
CI first: Ensure tests pass in CI before merging

Troubleshooting

Common Issues

Issue: Tests fail with “marker not registered” Solution: Add the marker to pyproject.toml in [tool.pytest.ini_options].markers

Issue: Coverage report shows 0% Solution: Ensure --cov=src/socialseed_e2e points to the correct source directory

Issue: Mock API tests timeout Solution: The mock API server may not have started. Check port availability (default: 8765)

Issue: Import errors in tests Solution: Install package in editable mode: pip install -e ".[dev]"