Can AI summarise a document reliably?

What a summariser is actually doing

A summarisation model does not extract sentences and stitch them together. It reads the document and then writes new sentences, one word at a time, choosing each word by what tends to follow.

That is why the output reads naturally, and it is also why it can be wrong in ways that a copy-paste approach never could: nothing anchors the generated text to the source except the model's training.

Failure one: it copies instead of summarising

This was the biggest surprise in our testing. Two models with clean, permissive licences — one 600M parameters, one 350M — did not summarise narrative prose. They reproduced the opening of the document verbatim.

Both handled encyclopedia articles fine. It was only when we added a piece of narrative writing that the behaviour showed up. If we had tested with reference material alone, we would have shipped a model that copies.

Forcing it not to copy made things worse rather than better: wrapped in delimiters, the same model produced a fluent sentence that said something the document never said.

Failure two: it invents when there is nothing to work with

Given an empty input, one model produced a summary of a public health campaign. Given the single word "hello", it wrote three lines of a diary entry.

A model always produces something — there is no state in which it returns nothing. If the input is too short to summarise, what comes out is not a bad summary; it is fiction. This is why our tool has a minimum length and refuses below it rather than obliging.

Failure three: it gets facts wrong, fluently

In one run the model expanded the abbreviation NADPH into a chemical name that was not what NADPH stands for. The sentence was well-formed and confident.

Fluency and accuracy are separate things, and they fail separately. This is the failure mode you cannot spot by reading the summary alone, which is precisely why a summary is a way to decide whether to read something — not a replacement for reading it.

What it is genuinely good at

Working out whether a long document is relevant to you. Getting the gist of a report in a language you read slowly. Producing a first draft of an abstract that you then correct.

One more limit worth knowing: asking for a summary in a different language than the source is where small models fall apart — inventing words that do not exist, or ignoring the instruction and answering in the source language. Summarising and translating are two jobs; ask for them one at a time.

If the document is a scan, nothing above applies yet

A scanned PDF has no text in it, only pictures of text. A summariser given such a file receives nothing at all — and, per the second failure above, a model given nothing will still write something.

Run text recognition first, check the result, then summarise. Recognition on a poor scan produces its own errors, and summarising those errors compounds them quietly.