Notes Tutorials

Testing basics

A test is a small program that checks another program. What a good unit test looks like, which cases to pick, how a failing test points at the bug, regression tests, the test pyramid, and the same idea in pytest, Go, Node's test runner, and PHPUnit.

7 min readAlso available in Bahasa Indonesia →

Every programmer tests. The question is whether you test by hand, by clicking around and eyeballing the result, or whether you write the checks down as code so they can run again in a second, every time anything changes. This part is about the second kind.

We’ve already met tests twice. Part 12 used characterization tests to pin down what existing code does. Part 13 talked about expected failures, which are exactly the cases tests should cover. Now let’s look at testing on its own.

A test is a small program that checks another program

Unit testing is “a form of software testing by which isolated source code is tested to validate expected behavior.” In plain words: call a function with known inputs, and check that the output is what you expect.

Here’s the pricing function from part 12, slightly tidied:

def final_price(price, qty):
    """Price for `qty` items: 5% off from 10 items, never below zero."""
    total = price * qty
    if qty >= 10:
        total = total * 95 // 100
    return max(total, 0)

And a test for it, using pytest. pytest “will run all files of the form test_*.py or *_test.py”, and “You can use the assert statement to verify test expectations.” No classes or special methods needed:

from pricing import final_price

def test_single_item_has_no_discount():
    # arrange
    price, qty = 100, 1
    # act
    result = final_price(price, qty)
    # assert
    assert result == 100

The three comments show the usual shape of a test, often called arrange, act, assert: set up the inputs, call the thing, check the result. Most tests are short enough that the comments aren’t needed, but the shape is always there.

Which cases to test

You can’t test every input. You pick a few that matter. A good habit is to think in groups:

  • The normal case. One item, a few items.
  • The boundaries. Where behavior changes: 9 items (no discount) and 10 items (discount). Off-by-one bugs live here, as in the loops of part 6.
  • Empty and zero. Zero items, zero price, an empty list.
  • Invalid or odd input. The refund row with qty = -1 from part 12.
  • Known tricky values. 333 × 10 with a 5% discount is 3,163.5, so it checks the rounding.

pytest’s parametrize lets you list examples as a table, which keeps them easy to read and easy to extend:

@pytest.mark.parametrize("price, qty, expected", [
    (0, 5, 0),
    (100, 0, 0),
    (333, 10, 3163),   # rounds down
    (85000, 12, 969000),
])
def test_final_price_examples(price, qty, expected):
    assert final_price(price, qty) == expected

Run the whole file:

test_pricing.py::test_single_item_has_no_discount PASSED                 [ 12%]
test_pricing.py::test_ten_items_get_five_percent_off PASSED              [ 25%]
test_pricing.py::test_nine_items_get_no_discount PASSED                  [ 37%]
test_pricing.py::test_refund_rows_never_go_negative PASSED               [ 50%]
test_pricing.py::test_final_price_examples[0-5-0] PASSED                 [ 62%]
test_pricing.py::test_final_price_examples[100-0-0] PASSED               [ 75%]
test_pricing.py::test_final_price_examples[333-10-3163] PASSED           [ 87%]
test_pricing.py::test_final_price_examples[85000-12-969000] PASSED       [100%]

============================== 8 passed in 0.02s ===============================

Eight checks in 0.02 seconds. Clicking through the same cases by hand would take minutes, and nobody does it on every change.

A failing test is a pointer, not a punishment

Tests earn their keep when something breaks. Suppose someone “simplifies” the condition from qty >= 10 to qty > 10. It looks harmless. The tests say otherwise:

>       assert final_price(100, 10) == 950
E       assert 1000 == 950
E        +  where 1000 = final_price(100, 10)

test_pricing.py:13: AssertionError
...
FAILED test_pricing.py::test_ten_items_get_five_percent_off - assert 1000 == 950
FAILED test_pricing.py::test_final_price_examples[333-10-3163] - assert 3330 ...
2 failed, 6 passed in 0.01s

The output says which test, which line, what was expected, and what came back. With 10 items the customer is no longer getting the discount. That’s found in a second, on the developer’s machine, instead of by a customer.

This is also why test names should describe behavior. test_ten_items_get_five_percent_off failing tells you the problem before you read anything else. test_price_2 doesn’t.

The same idea in other languages

Every mainstream language has a standard way to write tests. The shape is the same; only the syntax changes.

Go has testing built in. “Package testing provides support for automated testing of Go packages”, and you put tests in a file whose name ends in _test.go, in functions of the form func TestXxx(*testing.T). Go code often uses table-driven tests, the same idea as parametrize:

func TestParseQty(t *testing.T) {
	tests := []struct {
		name    string
		raw     string
		want    int
		wantErr bool
	}{
		{"plain number", "3", 3, false},
		{"spaces around", " 12 ", 12, false},
		{"letters", "3x", 0, true},
		{"empty", "", 0, true},
		{"zero", "0", 0, true},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			got, err := ParseQty(tt.raw)
			if (err != nil) != tt.wantErr {
				t.Fatalf("ParseQty(%q) error = %v, wantErr %v", tt.raw, err, tt.wantErr)
			}
			if got != tt.want {
				t.Errorf("ParseQty(%q) = %d, want %d", tt.raw, got, tt.want)
			}
		})
	}
}
--- PASS: TestParseQty (0.00s)
    --- PASS: TestParseQty/plain_number (0.00s)
    --- PASS: TestParseQty/spaces_around (0.00s)
    --- PASS: TestParseQty/letters (0.00s)
    --- PASS: TestParseQty/empty (0.00s)
    --- PASS: TestParseQty/zero (0.00s)
PASS

Note that three of the five cases test errors. Checking that bad input is rejected is as important as checking that good input works.

JavaScript: Node has a built-in runner too; “The node:test module facilitates the creation of JavaScript tests.”

import { test } from "node:test";
import assert from "node:assert/strict";
import { formatRupiah } from "./rupiah.js";

test("adds thousands separators", () => {
  assert.equal(formatRupiah(1500000), "Rp1.500.000");
});

If someone changes the separator to a comma, node --test shows exactly what differs:

✖ adds thousands separators
  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  + actual - expected

  + 'Rp1,500,000'
  - 'Rp1.500.000'

PHP: the standard tool is PHPUnit. A test class extends TestCase, and each public test... method is one test:

use PHPUnit\Framework\TestCase;

final class PricingTest extends TestCase
{
    public function testTenItemsGetFivePercentOff(): void
    {
        $this->assertSame(950, finalPrice(100, 10));
    }

    public function testRefundRowsNeverGoNegative(): void
    {
        $this->assertSame(0, finalPrice(100, -1));
    }
}

assertSame checks value and type, so "950" would not pass for 950. That matters in PHP, for the reasons in parts 5 and 13.

Regression tests: every bug fixed once

A regression is a bug that comes back. Regression testing is “re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change.”

The habit that makes this work is simple: when you fix a bug, first write a test that fails because of it. Then fix the code and watch the test pass. The refund bug from part 12 would get:

def test_refund_rows_never_go_negative():
    assert final_price(100, -1) == 0

Now that bug can’t quietly come back. If someone removes the max(total, 0) while “cleaning up”, the test catches it, and Chesterton’s fence from part 12 has a guard on it.

Writing the test first can be extended to all new code. Test-driven development (TDD) is “writing an automated unit-level test case that fails, then writing just enough code to make the test pass, then refactoring both the test code and the production code, then repeating with another new test case.” You don’t have to work that way all the time, but trying it on a small function is a good exercise: it forces you to decide what the function should do before deciding how.

Kinds of tests, and the pyramid

Not all tests are unit tests:

  • Unit tests check one function or class in isolation. Fast, precise, many of them.
  • Integration tests check that pieces work together: your code and a real database, two services talking.
  • End-to-end tests drive the whole application, often through a browser, like a user would.

Martin Fowler’s test pyramid sums up the balance: “you should have many more low-level UnitTests than high level BroadStackTests running through a GUI.” The reason is cost. Tests that run end-to-end through the UI “are: brittle, expensive to write, and time consuming to run.” A few of them are valuable, to check that the whole thing is wired together. Most checks belong lower down, where they’re fast and point straight at the problem.

What makes code easy to test

You’ll notice quickly that some code is easy to test and some is painful. The difference is mostly what part 7 talked about:

  • Pure functions are trivial to test. Same input, same output, nothing else to set up. final_price needs one line per test.
  • Side effects make tests hard. A function that reads the clock, calls an API, and writes to the database needs all of that faked or prepared before it can be tested.
  • So keep the logic separate from the I/O. The report in part 9 had one function that read the file and four pure ones that did the work. Four of five were easy to test.

When something is hard to test, that’s often a design hint, not a testing problem.

What tests can’t do

Tests check the cases you thought of. They can’t prove there are no bugs, only that these particular checks pass. Dijkstra made this point decades ago: testing can show that bugs are present, but never that they’re absent.

So tests are not a substitute for thinking. They’re a way to write your thinking down so the computer can check it again, every time, without getting tired.

Why this matters day to day

  • Changing code without fear. With tests, refactoring (part 17) and upgrades become routine instead of risky.
  • Faster feedback. A failing test in a second beats a bug report in a week.
  • Documentation that can’t go stale. Test names and examples show how code is meant to be used, and they fail if they stop being true.
  • Checking AI-generated code. Writing a few tests for code you didn’t write (by hand or by an AI tool) is one of the quickest ways to find out whether it actually does what it claims. Part 24 comes back to this.

Try it yourself

  1. Write tests for final_price covering 9, 10, and 11 items. Then break the condition on purpose and watch them fail.
  2. Pick a function in your own code and list its normal case, boundaries, empty case, and invalid input. Turn the list into a parametrized or table-driven test.
  3. Next time you fix a bug, write the failing test first.
  4. Find a function that’s hard to test. What would you need to move out of it to make the core logic pure?

Summary

  • A unit test calls code with known inputs and checks the output; the shape is arrange, act, assert.
  • Pick cases by group: normal, boundaries, empty/zero, invalid, tricky values.
  • A failing test shows what, where, expected and actual; give tests names that describe behavior.
  • Every language has a standard tool: pytest, Go’s testing package, node:test, PHPUnit. Tables of examples work in all of them.
  • Regression tests: when fixing a bug, write the failing test first.
  • Follow the test pyramid: many unit tests, fewer integration tests, a few end-to-end tests.
  • Pure functions are easy to test; keep logic separate from I/O.
  • Tests show bugs are present, never that they’re absent.

References