# Testing with pytest: Fixtures, Fake Models, and Properties

Here's a feeling you'll know soon enough. You change one small thing in your code, and you're not sure what else you just broke. So you run the program by hand, try a couple of inputs, and hope.

Tests replace the hoping. A test is a small piece of code that checks another piece of code, and you can run hundreds of them in a second. This post is core, and it matters even more for AI work than for ordinary programming, because AI apps have a part that's unpredictable (the model) surrounded by a lot of code that isn't. That surrounding code is what we'll test.

## Tests are not evals

Before we start, one distinction that will save you confusion later.

-   **Tests** check _your code_. Given this input, does the function return that output? They're fast, free, and always give the same answer.
    
-   **Evals** measure _how good the model's answers are_. They call a real model, the results vary, and they take real time and money. That's Post 60.
    

You need both. This post is about the first kind. And the trick for testing code that talks to a model is to replace the model with a fake, which is the heart of this post.

## Your first test

We'll use the `textkit` package from Post 12. Install pytest in your project's environment:

```bash
pip install pytest
```

If you set up the `dev` extras in Post 12, `pip install -e ".[dev]"` does it for you. With uv, it's `uv add --dev pytest`.

Create a file called `tests/test_counting.py`:

```python
from textkit.counting import count_words, top_words

def test_counts_repeated_words():
    assert count_words("the cat the dog") == {"the": 2, "cat": 1, "dog": 1}

def test_ignores_case_and_punctuation():
    assert count_words("The cat. the CAT,") == {"the": 2, "cat": 2}

def test_top_words_returns_at_most_n():
    result = top_words("a b c d e f", n=3)
    assert len(result) == 3

def test_top_words_orders_by_count():
    result = top_words("x y y z z z", n=2)
    assert result == [("z", 3), ("y", 2)]
```

Now run:

```bash
pytest
```

You'll see something like this (your version numbers and timing will differ):

```bash
============================= test session starts ==============================
collected 4 items

tests/test_counting.py ....                                              [100%]

============================== 4 passed in 0.02s ===============================
```

Each dot is a passing test. That's the whole system. Two rules make pytest find your tests:

-   Files are named `test_something.py`.
    
-   Test functions are named `test_something`.
    

And a test is just a function with a plain `assert` in it. If the assert is true, the test passes. If it's false, the test fails.

### When a test fails

Say we break a test on purpose by changing the expected length to 4. Here's the kind of message pytest gives you:

```python
def test_top_words_returns_at_most_n():
        result = top_words("a b c d e f", n=3)
>       assert len(result) == 4
E       assert 3 == 4
E        +  where 3 = len([('a', 1), ('b', 1), ('c', 1)])

tests/test_counting.py:14: AssertionError
```

Notice it doesn't just say "failed." It shows the line, the values on both sides, and where the number came from. That's a big part of why people like pytest. You rarely need extra print statements to understand a failure.

One practical note. The tests import `textkit`, which works because you installed the package in editable mode in Post 12. If you'd rather not, you can add this to `pyproject.toml` so pytest finds your code in `src/`:

```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
```

## What makes a good test

A few habits, learned the hard way by lots of programmers:

-   **One idea per test.** If a test fails, the name should tell you what broke. `test_ignores_case_and_punctuation` is much better than `test_stuff`.
    
-   **Arrange, act, assert.** Set things up, do the thing, check the result. Keeping those three steps visible makes tests easy to read.
    
-   **Test behavior, not insides.** Check what a function returns or does, not how it does it. Then you can rewrite the inside and the tests still pass.
    
-   **Cover the edges.** Empty input, a single item, very long input, odd characters, zero, negative numbers. Bugs hide at the edges.
    
-   **Keep them fast and independent.** A test shouldn't rely on another test running first.
    

Every time you find a bug, write a test that fails because of it, _then_ fix the bug. It's the "lock it in" step from Post 11, and it means the same bug can never sneak back unnoticed.

## Testing that an error is raised

Sometimes the correct behavior is to fail. Remember `set_temperature` from Post 11? Here's how you check it raises an error:

```python
import pytest

def set_temperature(value):
    if not 0 <= value <= 2:
        raise ValueError("temperature must be between 0 and 2")
    return value

def test_temperature_rejects_out_of_range():
    with pytest.raises(ValueError, match="temperature"):
        set_temperature(5)
```

`pytest.raises` is a context manager (Post 8). The test passes only if the code inside the `with` block raises that error. The optional `match` checks the message too. If no error happens, the test fails.

## Many cases, one test: parametrize

You often want to run the same check on lots of inputs. Instead of copying the test, use a decorator with arguments (Post 8 again):

```python
import pytest

from textkit.counting import count_words

@pytest.mark.parametrize(
    "text, expected",
    [
        ("", {}),
        ("hello", {"hello": 1}),
        ("A a", {"a": 2}),
    ],
)
def test_count_words_edge_cases(text, expected):
    assert count_words(text) == expected
```

Pytest runs this function three times, once per row, and reports each one separately. Adding a new case is one new line.

## Fixtures: reusable setup

A lot of tests need the same starting point: a file to read, a configured object, a fake service. A **fixture** is a function that builds that thing, and pytest hands it to any test that asks for it by name.

Remember the text pipeline from Post 7? To test it, we need a file. Pytest has a built-in fixture called `tmp_path` that gives you a fresh temporary folder, and we can build our own fixture on top of it:

```python
import pytest

from pipeline import keep_real_text, read_lines

@pytest.fixture
def temp_notes(tmp_path):
    path = tmp_path / "notes.txt"
    path.write_text("line one is long enough to keep\nshort\n")
    return path

def test_pipeline_keeps_only_long_lines(temp_notes):
    lines = list(keep_real_text(read_lines(temp_notes)))
    assert lines == ["line one is long enough to keep"]
```

See how it works. The test has a parameter called `temp_notes`. Pytest notices that name, finds the fixture with the same name, runs it, and passes in the result. Each test gets a fresh one, and the temporary folder is cleaned up afterward.

(One heads-up. If your `pipeline.py` from Post 7 has code at the bottom that runs the pipeline, importing it in a test will run that code too. That's exactly what the `if __name__ == "__main__":` guard from Post 12 is for. Wrap the bottom part in it.)

If several test files need the same fixture, put it in a file called `conftest.py` in the `tests/` folder. Pytest picks it up automatically, with no import needed.

### monkeypatch: changing things temporarily

Another built-in fixture, `monkeypatch`, lets a test change something and automatically puts it back afterward. It's perfect for environment variables, like the API key from Post 12:

```python
import os

import pytest

def get_api_key():
    key = os.environ.get("MODEL_API_KEY")
    if key is None:
        raise RuntimeError("Set the MODEL_API_KEY environment variable first")
    return key

def test_missing_key_raises(monkeypatch):
    monkeypatch.delenv("MODEL_API_KEY", raising=False)
    with pytest.raises(RuntimeError):
        get_api_key()

def test_key_is_read(monkeypatch):
    monkeypatch.setenv("MODEL_API_KEY", "test-key")
    assert get_api_key() == "test-key"
```

Both tests work no matter what's set on your computer, and neither leaves a mess behind.

## Testing code that calls a model

Now the part you came for. Suppose you have a function that asks a model for JSON and retries if the reply isn't valid. Testing it against a real model would be slow, cost money, need an API key, and give a different answer each time. You can't test logic that way.

The fix is to make the model something you can swap out, which is exactly what we set up in Post 9 with composition and protocols. Here's the function (add it as `src/textkit/replies.py`):

```python
import json

def get_json_reply(model, prompt, max_attempts=3):
    """Ask the model for JSON. Retry if the reply isn't valid JSON."""
    for _ in range(max_attempts):
        reply = model.generate(prompt)
        try:
            return json.loads(reply)
        except json.JSONDecodeError:
            prompt = prompt + "\nYour last reply was not valid JSON. Reply with JSON only."
    raise ValueError("Model never returned valid JSON")
```

It takes `model` as a parameter, and all it needs is something with a `generate` method. So in tests, we hand it a **fake**: a tiny class that returns replies we chose in advance.

```python
class FakeModel:
    """A stand-in for a real model that returns canned replies."""

    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = []

    def generate(self, prompt):
        self.calls.append(prompt)
        return self.replies.pop(0)
```

It also records every prompt it received in `self.calls`, so tests can check what we sent. Now the tests:

```python
import pytest

from textkit.replies import get_json_reply

def test_returns_parsed_json():
    model = FakeModel(['{"answer": 42}'])
    assert get_json_reply(model, "Give me JSON") == {"answer": 42}

def test_retries_after_bad_json():
    model = FakeModel(["not json", '{"answer": 42}'])

    result = get_json_reply(model, "Give me JSON")

    assert result == {"answer": 42}
    assert len(model.calls) == 2
    assert "not valid JSON" in model.calls[1]

def test_gives_up_after_max_attempts():
    model = FakeModel(["bad", "bad", "bad"])
    with pytest.raises(ValueError, match="valid JSON"):
        get_json_reply(model, "Give me JSON")
```

These tests run instantly, cost nothing, and are the same every time. And they check real things: that good JSON is parsed, that a bad reply triggers a retry, that the retry prompt includes the correction message, and that we give up eventually instead of looping forever.

### When you can't pass a fake in: mock and patch

Sometimes the model call is buried inside code you can't easily change. Python's built-in `unittest.mock` library has tools for that, and they work fine inside pytest.

`Mock` makes an object that accepts any call and remembers it:

```python
from unittest.mock import Mock

def test_works_with_a_mock_object():
    model = Mock()
    model.generate.return_value = '{"answer": 1}'

    assert get_json_reply(model, "x") == {"answer": 1}
    model.generate.assert_called_once_with("x")
```

`side_effect` lets a mock fail or change its answer from call to call. This is great for testing retries without waiting around:

```python
def call_with_retries(func, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            return func()
        except TimeoutError:
            if attempt == attempts:
                raise

def test_retries_then_succeeds():
    flaky = Mock(side_effect=[TimeoutError("slow"), "ok"])
    assert call_with_retries(flaky) == "ok"
    assert flaky.call_count == 2
```

`patch` temporarily replaces a real function by name. Imagine `my_app.py` has a function `call_api` that hits the network, and `answer` that uses it:

```python
# my_app.py
def call_api(prompt):
    ...  # the real network call

def answer(question):
    return call_api(question).strip()
```

```python
from unittest.mock import patch

import my_app

def test_answer_strips_whitespace():
    with patch("my_app.call_api", return_value="  hi  "):
        assert my_app.answer("q") == "hi"
```

The one rule people forget: **patch the name where it's used, not where it was defined.** Here `answer` looks up `call_api` inside `my_app`, so that's the name we replace. Pytest's `monkeypatch.setattr(my_app, "call_api", lambda prompt: " hi ")` does the same job if you prefer it.

### Don't overdo it

Mocks are powerful, and it's easy to go too far. A few cautions:

-   **Fake only at the edges**: the model, the network, the clock. Don't mock your own internal functions, or you'll end up testing the mocks instead of your code.
    
-   **Fakes can drift.** If the real model API changes and your fake doesn't, your tests pass while your app breaks. Keep a small number of tests that call the real thing, mark them, and run them only occasionally:
    

```toml
[tool.pytest.ini_options]
markers = ["integration: tests that call a real model API (slow, cost money)"]
```

```python
@pytest.mark.integration
def test_real_model_returns_json():
    ...
```

Then `pytest -m "not integration"` runs everything except those, and `pytest -m integration` runs only them.

-   **Tools that record and replay** real API responses also exist, if you want realistic data without hitting the network each time.
    

## Property-based testing: let the computer find the weird inputs

Ordinary tests check examples you thought of. The trouble is that bugs live in the examples you _didn't_ think of. Property-based testing flips it around. You describe a rule that should be true for _any_ input, and a tool generates hundreds of inputs to try to break it.

The standard library for this in Python is **Hypothesis**. Install it with `pip install hypothesis`. Here are two properties of our word counter:

```python
from hypothesis import given
from hypothesis import strategies as st

from textkit.counting import count_words, top_words

@given(st.text(), st.integers(min_value=0, max_value=10))
def test_top_words_is_limited_and_sorted(text, n):
    result = top_words(text, n)
    assert len(result) <= n
    counts = [count for _, count in result]
    assert counts == sorted(counts, reverse=True)

@given(st.text())
def test_adding_a_word_adds_one_to_its_count(text):
    before = count_words(text).get("zzz", 0)
    after = count_words(text + " zzz")["zzz"]
    assert after == before + 1
```

`@given(...)` tells Hypothesis what kinds of inputs to create: any text, any whole number between 0 and 10. It then runs your test many times with different values, including nasty ones like empty strings and odd Unicode characters.

Notice we never said what the output should be for any particular input. We said what must _always_ hold: the result is never longer than `n`, the counts are sorted from biggest to smallest, and adding a word once raises its count by exactly one.

When Hypothesis finds a failing input, it doesn't just show you the mess it started with. It shrinks it to the smallest failing example, which is usually much easier to understand. A function that divides by `len(numbers)`, for example, would be caught almost instantly with the empty list `[]`.

Properties are harder to write than examples, so don't use them for everything. They shine for code with clear rules: parsers, formatters, sorting, batching, and anything where you can say "convert it and convert it back, and you should get the original."

## Running tests well

A few options you'll use constantly:

| Command | What it does |
| --- | --- |
| pytest | Run everything |
| pytest -v | Show each test name |
| pytest -x | Stop at the first failure |
| pytest -k counts | Run only tests whose name contains "counts" |
| pytest tests/test_counting.py::test_top_words_orders_by_count | Run one specific test |
| pytest --lf | Re-run only the tests that failed last time |
| pytest -s | Show print output, which pytest hides by default |
| pytest --pdb | Drop into the debugger from Post 11 when a test fails |

Later, in Post 28, we'll set up automatic runs so your tests execute every time you push code. A test suite that only runs when you remember is a test suite that stops running.

One last warning. You may hear about "test coverage," a percentage showing how many lines your tests ran. It's a handy way to spot code that has no tests at all. But a high number doesn't mean good tests. You can run every line and check nothing. Aim for tests that would actually catch a mistake.

## Mini project: a test suite for textkit

Let's add a real test suite to the `textkit` package from Post 12.

**Step 1.** Add the `replies.py` module shown above to `src/textkit/`.

**Step 2.** Update the project settings in `pyproject.toml`:

```toml
[project.optional-dependencies]
dev = ["pytest>=8.0", "hypothesis>=6.0"]

[tool.pytest.ini_options]
testpaths = ["tests"]
markers = ["integration: tests that call a real model API (slow, cost money)"]
```

**Step 3.** Install everything:

```bash
pip install -e ".[dev]"
```

**Step 4.** Create three test files in `tests/`:

-   `test_counting.py`: the four example tests from earlier, plus the parametrized edge-case test.
    
-   `test_replies.py`: the `FakeModel` class, the three tests that use it, and the `Mock` test.
    
-   `test_properties.py`: the two Hypothesis tests.
    

**Step 5.** Run `pytest`. You should see all 13 tests pass:

```bash
tests/test_counting.py .......                                           [ 53%]
tests/test_properties.py ..                                              [ 69%]
tests/test_replies.py ....                                               [100%]

============================== 13 passed in 0.31s ==============================
```

(The seven in the first file are four examples plus three parametrized cases.)

Now the most important step: **try to break it.** Open `counting.py` and make a deliberate mistake, such as removing `.lower()`, flipping `reverse=True` to `reverse=False`, or changing `ranked[:n]` to `ranked[:n + 1]`. Run the tests and watch which ones catch it. A test that never fails isn't protecting you. Then put the code back.

### Make it yours

1.  Test the command line tool. Using `monkeypatch.setattr("sys.argv", ["textkit", "the", "cat", "the"])` and the `capsys` fixture, call `main()` and check that the printed output contains `the: 2`.
    
2.  Add an integration test, marked with `@pytest.mark.integration`, that would call a real model. Confirm `pytest -m "not integration"` skips it.
    
3.  Install `pytest-cov` and run `pytest --cov=textkit`. Look at which lines have no tests.
    
4.  Write a fake model that raises an error on the first call, and test how `get_json_reply` should behave.
    

## Practice

About 20 minutes of real effort on each before asking for help.

1.  Write five tests for `safe_int` from Post 11, using `parametrize` for the different inputs.
    
2.  Write a test for `set_temperature` that uses `pytest.raises` with a `match`, and add a second test for a value on the boundary (exactly 0 and exactly 2).
    
3.  Write a fixture that creates a temporary file, and use it to test `read_lines` from Post 7.
    
4.  Take an earlier project that retries an API call. Replace the real call with a `Mock(side_effect=[...])` and test the retry logic without waiting.
    
5.  Write a property-based test for sorting: the result is in order, and it has the same items as the input.
    
6.  Break a function on purpose, confirm a test fails with a readable message, then fix it.
    

Then ask an AI tutor to review your tests, and don't let it write them for you. Tests you didn't think through are the ones that give false comfort.

## What to take away

-   Tests check your code and give the same answer every time. Evals check the model's quality, and come later.
    
-   Name files `test_*.py` and functions `test_*`, and use plain `assert`.
    
-   `pytest.raises` checks for errors, and `parametrize` runs one test on many inputs.
    
-   Fixtures provide reusable setup, and `tmp_path` and `monkeypatch` are two built-in ones you'll use a lot.
    
-   To test code that uses a model, swap in a fake. Pass it in as a parameter, or use `Mock` and `patch` when you can't.
    
-   Patch where a name is used, fake only at the edges, and keep a few real integration tests.
    
-   Property-based tests describe rules that must always hold, and let Hypothesis hunt for the inputs that break them.
    
-   A test is only useful if it can fail, so try breaking your code on purpose.
    

* * *

**Previous:** Post 12, _Packaging & Environments_

**Next in the series:** Post 14, _Concurrency_, where we cover threads, processes, `asyncio`, and why LLM apps are so often async.

---

*Published via [ZyVOP](https://zyvop.com/testing-with-pytest-fixtures-fake-models-and-properties-edvab?utm_source=hashnode&utm_medium=crosspost&utm_campaign=syndication) — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.*