← Lectures
Back

1 / 8

Next

Slide 1 of 8: The thing that had to be represented

Arrow keys move through the slides while these controls have focus.

The theatre is not shown, because your system asks for reduced motion. The lecture itself is below, in full.

Lecture · 8 slides

Null, and the mistake its inventor apologised for

In 2009, Tony Hoare stood up at a conference and apologised for null references. He had put them into ALGOL W in 1965, he said, simply because they were easy to implement, and he estimated the consequences at a billion dollars in errors and damage over the following forty years.

It is a good line and it has been repeated ever since, usually as a way of dismissing null rather than understanding it. That is a shame, because the interesting part is not that null is bad. It is that null is a perfectly reasonable idea implemented in a way that defeats the one mechanism that could have caught the resulting errors — and that the fix, which is not complicated and was known at the time, took four decades to reach mainstream languages.

This lecture is about what precisely went wrong, about the two families of solution and what each costs, and about the fact that the problem null was solving has not gone away. Something has to represent absence. The question was never whether to have it, but whether the type system is allowed to know.

AssumesYou have been woken up by a null pointer exception, or its equivalent in whatever language you use.

  1. Slide 1

    The thing that had to be represented

    Start with the requirement, because null was an answer to a real question and the question has not changed.

    A program constantly needs to say that something is not there. The user has no middle name. The search found nothing. This node has no parent, this list has no next element, this configuration key was never set. There is no way to write software that manipulates the world without a way of saying "absent", and any language that pretends otherwise has simply named it something else.

    So null is not a mistake in the sense of being unnecessary. Hoare's regret was more specific than that, and the specificity is the whole lesson.

  2. Slide 2

    What actually went wrong

    The mistake was not having a value for absence. It was making that value a member of every type, so no type could exclude it.

    String name = null;
    java

    That compiles. So does the same assignment to a User, a List, a Connection — every reference type in the language includes null in its set of possible values, whether or not that makes any sense for the thing being described. A function that returns String might return no string. A function that takes a User might get nothing. Nothing in the signature distinguishes the cases where absence is meaningful from the overwhelming majority where it is a bug.

    This is why it is a type system failure rather than a programming failure. The one tool whose job is to rule out impossible states had a hole in it large enough to admit the commonest runtime error in the industry, and it had that hole by design.

  3. Slide 3

    The failure is delayed and displaced

    Because the type says nothing, the error cannot happen where the mistake was made, and this is what makes the resulting bugs expensive.

    A null is created in one place, stored in another, passed through four more, and dereferenced somewhere that has no idea where it came from. The stack trace names the site of the dereference, which is usually innocent. The bug is at the site of the assignment, which is not in the trace at all.

    Compare this with the type errors the same compiler catches happily. Pass a String where an int is expected and you are told immediately, at the line where you did it, before the program has run. Pass a null and you are told hours later, in production, in a different file. Same category of mistake, utterly different cost, and the difference is entirely whether the type system was permitted to look.

  4. Slide 4

    The first fix: make it a value

    The oldest solution predates the problem and was sitting in ML in 1973: put absence in the type, as an ordinary value that must be unwrapped.

    data Maybe a = Nothing | Just a

name :: Maybe String
    haskell

    A Maybe String is not a string. You cannot call string functions on it, and the compiler will refuse if you try. To get at the value you must handle both cases, and the compiler checks that you have — forgetting one is not a runtime crash but a compile error listing the case you missed.

    What this costs is real and worth stating plainly. Every access needs unwrapping, which is ceremony where absence is rare. Nesting is awkward: a Maybe inside a Maybe is a thing you can construct and rarely want. Languages that adopt this need decent syntax for chaining, or the code fills up with pattern matches over values that are almost never absent.

  5. Slide 5

    The second fix: make it a flag on the type

    The other approach keeps null but writes it into the type, and this is what most mainstream languages eventually did.

    let a: string;
let b: string | null;
    typescript

    a cannot be null and the compiler enforces it. b can, and every use is checked until you have proved otherwise. Kotlin did this in 2011 with ?, TypeScript with strict null checks in 2016, C# in 2019, and Swift from the start. The result is the same guarantee as the option type with far less ceremony, because absence is a property of the declaration rather than a wrapper around the value.

    The catch is that it only works if the whole program plays. TypeScript's guarantee stops at any value that came from untyped JavaScript; Kotlin's stops at the boundary with Java, which is why it has a distinct notation for types it cannot vouch for. A retrofitted non-null guarantee is a claim about the code the compiler can see, and the bugs move to the edges.

  6. Slide 6

    Absence is not the same as failure

    A distinction worth drawing, because conflating the two is how codebases end up with three ways of saying nothing.

    Absence is ordinary and expected. The user has no middle name. Nothing has gone wrong and there is nothing to report. Failure is different: the database was unreachable, the input was malformed, something happened that a caller may need to know the reason for. An option type says "there is no value". It does not say why, and when the why matters you want a result type carrying an error instead.

    Using the same construct for both loses information at exactly the moment somebody needs it. A lookup returning nothing is fine; a lookup returning nothing because the connection dropped is an outage being silently reported as an empty result.

  7. Slide 7

    What is left, honestly

    The languages that removed null did not remove the possibility of getting it wrong. They moved it somewhere better lit.

    You can still write a partial function that crashes on the case it does not handle. Rust has unwrap, Swift has the exclamation mark, Kotlin has !!, and all three exist because sometimes you genuinely know better than the compiler. Each of them is a place where the guarantee is being deliberately suspended, and unlike null they are visible in the source: a reviewer can search for them, and their presence is a decision somebody made rather than a hole in the language.

    That is the real improvement, and it is more modest than the "billion-dollar mistake" framing suggests. The bug did not become impossible. It became local, visible and named, which is the most any type system offers about anything.

  8. Slide 8

    What to take from it

    The story is usually told as an argument for particular languages, and it is more useful as an argument about a principle.

    The principle is that a type system's value comes from what it can rule out, and any value that inhabits every type rules out nothing. Null was not catastrophic because absence is a bad idea. It was catastrophic because it was invisible to the mechanism that existed to catch it, and it stayed that way for forty years because backwards compatibility is the strongest force in language design.

    Which is the practical lesson for code you write today. When you invent a sentinel — an empty string meaning missing, a zero meaning unset, a magic identifier meaning none — you are recreating exactly this mistake at a smaller scale, in a place your compiler cannot see. The fix is the same one Hoare wished he had made: say it in the type, where something is checking.

Works in the collection

The arguments above are hanging on the walls of the museum, in one form or another. These are the ones worth looking at next.

The rest of the programme

  1. Compiled, interpreted, and the space betweenA distinction that stopped describing anything decades ago, why it persists, and what is actually different about the machinery underneath.
  2. What a compiler actually doesFour jobs in a row, each mechanical, none magic: text to tokens, tokens to a tree, a tree to a judgement, and a judgement to instructions.
  3. Functional programming and object orientationTwo ways of arranging a program, what each one genuinely makes easy, and the trade-off underneath the argument that neither side can escape.
  4. The principles of object orientationEncapsulation, inheritance, polymorphism and SOLID, one at a time: what each actually claims, which held up, and which its own community abandoned.
  5. Types: what they can and cannot proveA type checker proves one thing about every possible run of your program. Knowing which proposition explains both the enthusiasm and the disappointment.
  6. What abstraction costsAbstraction is sold as free and is not. What you buy, what you pay, and how to tell before writing it which of the two is larger.
  7. Immutability, and what it is not free ofValues that never change buy sharing, comparison and time travel. They are not free, and it is worth knowing where they are expensive before you commit.
  8. Recursion, and why it feels like cheatingA function that calls itself looks like an unpaid debt. What makes it terminate, what it costs on the stack, and why some problems resist any other shape.
  9. Big-O, and what it deliberately ignoresComplexity notation throws away constants, hardware and every input you will actually see. Knowing what it discards is what makes the number useful.
  10. What regular expressions cannot matchThere is a precise boundary around what a regex can recognise. It explains the famous refusal to parse HTML, and why some patterns run forever.
  11. Concurrency is not parallelismOne is a way of structuring a program, the other a way of executing it. Keeping them apart explains why async helps a web server and threads often do not.
  12. Errors: exceptions, values, and what each hidesThrowing makes the happy path readable and the failure paths invisible. Returning errors as values does the opposite. Neither side has won.
  13. Why programs are hard to changeSoftware is called soft because it can be edited. Why editing gets harder every year, what the mechanism is, and which of the usual remedies work.

← All lecturesAtrium