Notes Tutorials

Reviewing code written by AI

AI tools write plausible code in seconds. Plausible isn't the same as correct. How to review generated code with the same standards as any other code: read it against the requirement, run it, pin expectations with tests, check what it hides, verify that packages and APIs exist, and own what you merge.

7 min readAlso available in Bahasa Indonesia →

Many earlier parts ended with a note about AI tools: check what’s caught (part 13), read the diff (part 16), ask what happens when two requests run at once (part 21). This part pulls those notes together.

AI coding tools are genuinely useful. They write boilerplate, suggest approaches, explain unfamiliar code, and produce a first draft faster than anyone can type. The catch is that the draft always looks finished. It compiles, it’s formatted, the names are reasonable, there’s a docstring. Whether it’s correct is a separate question, and answering it is now a bigger part of the job.

Same standard as any other code

Reviewing generated code isn’t a new skill. It’s code review, applied to an author who is fast, confident, and has never seen your production data.

Google’s public code review guide is a good baseline. “The most important thing to cover in a review is the overall design of the CL.” On complexity: “Is the CL more complex than it should be?”, where ““Too complex” usually means “can’t be understood quickly by code readers.”” On tests: “Will the tests actually fail when the code is broken?” And its overall standard: reviewers “should favor approving a CL once it is in a state where it definitely improves the overall code health” of the system.

That last line matters: the goal isn’t perfect code. It’s code you understand, that does what was asked, that won’t hurt users, and that leaves the codebase better rather than worse.

A plausible first draft

Here’s a task: “Import orders from a CSV file into the database and return the total.” Below is the kind of first draft that’s easy to get from any assistant, or from a hurried human. (I wrote it to show common patterns; it isn’t the output of a specific tool.)

def import_orders(csv_path, db):
    """Import orders from CSV into the database."""
    total = 0.0
    with open(csv_path) as f:
        for row in csv.DictReader(f):
            try:
                amount = float(row["qty"]) * float(row["price"])
                db.execute(f"INSERT INTO orders (customer, amount) VALUES ('{row['customer']}', {amount})")
                total += amount
            except Exception:
                pass
    db.commit()
    return total

It reads well. It even “handles errors”. Run it on four rows:

customer,qty,price
Ana,2,85000
Budi,10,4000
Siti O'Neil,1,35000
Dewi,3x,12000
returned total: 210000.0
rows saved: [('Ana', 170000.0), ('Budi', 40000.0)]

Two of four rows are gone, and nothing said so. Dewi’s 3x is genuinely invalid, which is fine to reject, but it should be reported. Siti’s row is valid: the apostrophe in O'Neil broke the SQL string, the except swallowed the error, and a real customer’s order disappeared. The same hole would let a crafted name rewrite the query (part 19).

Reviewing it, line by line

Going through the draft with the questions from this series:

What to check In the draft Part
Does it do what was asked, for every input? Drops valid rows silently 9
Are errors visible? except Exception: pass hides everything 13
Is data kept apart from code? SQL built with an f-string 19, 22
Right types? float for money 2
All or nothing? Partial import if it crashes midway 19
Encoding and file handling? No encoding, no newline="" 2
Can the caller tell what happened? Returns one float, no errors, no count 7

None of these are exotic. They’re the fundamentals. That’s the point of this whole series: the review is only as good as what the reviewer knows to look for.

Pin the expectations with tests

Before changing anything, write down what “correct” means as tests (part 14). They’re short, and they turn a vague feeling into a failing check:

def test_name_with_apostrophe_is_saved(impl):
    db = fresh_db()
    impl.import_orders("orders.csv", db)
    names = [r[0] for r in db.execute("SELECT customer FROM orders")]
    assert "Siti O'Neil" in names

def test_bad_rows_are_reported_not_hidden(impl):
    result = impl.import_orders("orders.csv", fresh_db())
    assert hasattr(result, "errors") and len(result.errors) == 1

Against the draft:

E       assert "Siti O'Neil" in ['Ana', 'Budi']
E       AssertionError: assert (False)
FAILED test_import.py::test_name_with_apostrophe_is_saved[draft]
FAILED test_import.py::test_bad_rows_are_reported_not_hidden[draft]

Asking an AI tool to “write tests” for its own code is useful, but check that the tests describe what you need, not just what the code already does. Google’s guide again: “Tests do not test themselves, and we rarely write tests for our tests—a human must ensure that tests are valid.”

The reviewed version

def parse_row(line_no, row):
    try:
        qty, price = int(row["qty"]), int(row["price"])
    except (KeyError, ValueError):
        raise ValueError(f"line {line_no}: qty and price must be whole numbers, got {row!r}")
    if qty < 1 or price < 0:
        raise ValueError(f"line {line_no}: qty must be >= 1 and price >= 0")
    return row["customer"].strip(), qty * price


def import_orders(csv_path, db):
    """Import orders from a CSV file. Valid rows are saved in one transaction;
    invalid rows are reported, not silently dropped."""
    result = ImportResult()
    rows = []
    with open(csv_path, newline="", encoding="utf-8") as f:
        for line_no, row in enumerate(csv.DictReader(f), start=2):
            try:
                rows.append(parse_row(line_no, row))
            except ValueError as e:
                result.errors.append(str(e))
    with db:
        db.executemany("INSERT INTO orders (customer, amount) VALUES (?, ?)", rows)
    result.saved = len(rows)
    result.total = sum(amount for _, amount in rows)
    return result

Placeholders, whole rupiah, a specific exception with line number and value, one transaction, and a result that says what was saved and what wasn’t. The same tests:

2 failed, 3 passed in 0.02s

The two failures are the draft’s; all three checks against the reviewed version pass, including saved == 3, total == 245000, and the error message line 5: qty and price must be whole numbers, got {...'qty': '3x'...}. It’s longer than the draft, and that’s fine (part 17).

You could get to this version by prompting too: “use placeholders, report invalid rows with line numbers, use integers for money, one transaction”. But you can only ask for those things if you know they matter.

Check that things actually exist

Generated code sometimes uses functions, options, or whole packages that don’t exist, or existed in a different version. A 2024 study, “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs”, tested 16 models and found “the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models”, “including a staggering 205,474 unique examples of hallucinated package names”.

That’s a security issue, not just an inconvenience: if someone registers a package under a commonly hallucinated name, pip install or npm install will happily download it (part 22). Before installing a suggested package, check that it exists, is the one you think it is, and is maintained:

$ pip index versions requests
requests (2.34.2)

$ pip index versions csv-order-importer-pro
ERROR: No matching distribution found for csv-order-importer-pro

The same goes for APIs and options: if a suggested function or flag looks unfamiliar, look it up in the official docs for the version you actually use. Five seconds in the docs beats an hour of debugging a method that was never there.

A review checklist for generated code

  1. Restate the requirement. What exactly should this do, including edge cases? Does the code do that, or something nearby?
  2. Read every line. If you can’t explain a line, don’t merge it (part 12). Ask the tool to explain, then verify the explanation.
  3. Run it with real and awkward input: empty, huge, Unicode, apostrophes, negative numbers, duplicates.
  4. Write or check tests that would fail if it were wrong.
  5. Look for hidden failure: broad try/except, catch (e) {}, default values that mask errors, missing status checks (parts 13, 20).
  6. Security pass: input validation, placeholders, escaping, authorization, secrets, randomness (part 22).
  7. Concurrency and scale: check-then-act, queries in loops, O(n²) hiding in small test data (parts 10, 21).
  8. Dependencies and APIs exist, at the version you use.
  9. Fits the codebase: naming, structure, conventions (part 11). Generated code tends to bring its own style.
  10. Read the final diff before committing (part 16), and commit before letting a tool make large changes, so you can undo them.

Working with AI tools, not just after them

Review gets easier when the request is good:

  • Ask for small pieces, not whole features. “Write parse_row(line_no, row) that returns (customer, amount) or raises ValueError with the line number” is reviewable. “Build the import feature” isn’t (part 9).
  • Give context: language version, framework, existing conventions, constraints like “money is integer rupiah”.
  • Ask for tests and edge cases, then check them.
  • Ask it to review your code or its own against a checklist like the one above. It’s often good at spotting issues when asked directly; it’s still your call.
  • Don’t accept what you don’t understand. “It works” is not the same as “I know why it works”.

In the end, the rule is simple: if you merge it, you own it. The tool doesn’t get paged at 3 a.m.; you do.

Why this matters day to day

  • More code is generated than ever, so more code needs reviewing. Reviewing is becoming as important as writing.
  • Plausible bugs are the expensive ones. Obvious nonsense gets caught; a clean-looking except: pass ships.
  • Fundamentals are the checklist. Every item above maps to a part of this series.
  • Speed is only real if the result is right. Ten minutes of review is cheaper than a lost order or a leaked database.

Try it yourself

  1. Ask an AI tool for the CSV import in this part. Review its answer with the checklist. How many items does it pass?
  2. Write the two tests from this part first, then ask the tool to make them pass. Did it change the tests instead of the code?
  3. Take a package name an AI tool suggested recently and check that it exists, who maintains it, and when it was last released.
  4. Ask a tool to review a function you wrote, then decide which suggestions you agree with and why.

Summary

  • AI-generated code looks finished; whether it’s correct is your job to decide.
  • Review it like any code: design, correctness, complexity, tests, and whether it improves the codebase.
  • Run it with awkward input and pin expectations with tests that fail when it’s wrong.
  • Look for hidden failures, security holes, wrong types, and missing transactions.
  • Verify packages and APIs exist at your version; hallucinated packages are a real supply-chain risk.
  • Ask for small, well-specified pieces, and don’t merge what you can’t explain.
  • If you merge it, you own it.

References