Skip to content
Happen Softwarewww.happen.software · sales@happen.software
All case studies

Case study · Media · AI & automation

A quarter of a million pages, searchable by who's in them

Zondo Explorer let anyone search the State Capture Commission's reports, transcripts, and statements by person or organisation, see every page they appear on, and follow the relationships between them. We processed more than 250,000 pages using OCR and traditional NLP for News24, with the work beginning in 2021, before ChatGPT and today's LLM tools.

  • 2021–2024

  • 250,000+

    Pages of commission reports, transcripts, and statements processed

  • 10,000+

    People and organisations identified, tens of thousands in all, each linked to every page that mentions them

  • 1,000s

    Of entities connected by relationships the models found, each tied to the page it came from

  • 100%

    Of mentions traceable to the original document page, which readers can open or download

At a glance

Client
News24
Scale
250,000+ pages, tens of thousands of entities, and relationships between thousands of them
Documents
The commission's final report volumes, hearing transcripts, and statements submitted as evidence
Engagement
From 2021 to 2024: extraction, matching, portal, and the public site
Users
The public, researchers, and journalists, with News24's editors curating in the portal
Platforms
A public explorer, an editors' portal, and tools for loading new source material

01The situation

The Judicial Commission of Inquiry into State Capture, chaired by now Chief Justice Raymond Zondo, sat from 2018 to 2022. It published its final report in volumes, alongside hearing transcripts and the statements and evidence submitted to it: more than a quarter of a million pages, much of it scanned.

News24 wanted South Africans to be able to find every person and organisation that surfaced in the proceedings, and how they connect. We ran OCR and natural language models over every page to find the people, organisations, and relationships in them, built the matching that turned raw mentions into clean entities, and built the portal News24's editors curated it in and the Explorer the public used.

The extraction work predates ChatGPT and today's widely used large language model tools. We used OCR and traditional NLP models for entity and relationship extraction, with matching and editorial review turning the results into a searchable archive.

  1. 01

    Scanned pages

    Much of the material was scanned, so names arrived with OCR errors: letters swapped, split, or misread.

  2. 02

    One name, many spellings

    The same company or person appears with and without titles, in different word orders, as acronyms, and in possessive forms.

  3. 03

    Evidence, not opinion

    Every connection the site shows had to lead back to the exact page it came from, so readers could check it themselves.

02What changes

  • Finding a nameWithout it: Searching volume by volume, PDF by PDF.With Zondo Explorer: One search across every report, transcript, and statement, by person or organisation.
  • Spellings and OCR errorsWithout it: Each variant searched separately, and misread names missed.With Zondo Explorer: Variants and OCR errors merged into one entity, with its aliases listed.
  • Where they appearWithout it: No overview of how much a name features, or where.With Zondo Explorer: A breakdown of every document and page an entity is mentioned in, with charts.
  • ConnectionsWithout it: Relationships pieced together by reading.With Zondo Explorer: Relationships between people and organisations drawn as a diagram, grouped by type.
  • Checking the sourceWithout it: Finding the page again in the original.With Zondo Explorer: Every mention links to its page, and each document can be viewed and downloaded.

03How it works

From scanned page to explorer, end to end

How Zondo Explorer turns scanned commission documents into searchable people and organisationsStarting before ChatGPT and today's LLM tools, OCR and traditional NLP made commission reports, transcripts, and statements searchable by person and organisation. Different spellings and reading errors are brought into consistent records, editors review the results, and readers explore relationships with links back to the original pages. The example names are illustrative.DocumentsReports, transcriptsand statementsReadOCR page by pagePage images keptExtractPeople, organisations,places and their linksRelationships foundPrepare recordsReading errors fixedConsistent records250,000+pages processedDIFFERENT NAMES, ONE RECORDAcme HoldingsACME Holdings (Pty) LtdAH (an acronym)Acrne Holdings (OCR error)Acme HoldingsOne entity, 4 aliasesEvery page it's onCurateEditors merge strays,add roles & photosExploreSearch → entity → links→ the original page

Swipe to see the whole diagram.

  1. 1

    Read

    Each document is split into pages and run through OCR. Page images and thumbnails are stored so readers can see the original later.

  2. 2

    Extract

    Traditional NLP identifies people, organisations, places, and relationships on each page, ready for matching and editorial review.

  3. 3

    Clean

    Irrelevant fragments and reading errors are cleaned up so people and organisations have useful, consistent records.

  4. 4

    Match

    Alternative spellings and OCR errors are brought together into one record, with its aliases and links to every page where it appears.

  5. 5

    Curate

    Editors review people, organisations, and relationships in the portal, merge duplicate records, and add roles, photos, and items of interest.

  6. 6

    Explore

    Readers search the Explorer, open an entity, see where it appears and what it's connected to, and click through to the page itself.

04What we built

  • OCR & entity extraction

    OCR and traditional NLP identify people, organisations, and places across the archive, each linked to the page that mentions it.

  • Entity matching

    Different spellings, acronyms, and OCR errors are brought into one record so readers can find the same person or organisation consistently.

  • Relationships

    Readers can explore relationships between people and organisations and follow each connection back to its source page.

  • Search

    Search across every person and organisation, with people and organisations of interest featured on the home page alongside the documents and key relationships.

  • Entity pages

    For each person or organisation: role, photo, a breakdown of every document they're mentioned in with charts, and their relationships grouped by type and drawn as a diagram.

  • Document pages

    Each report volume, transcript, and statement with the entities mentioned in it and the most-mentioned charted, page images to view, and the document to download.

  • Editors' portal

    Entities, aliases, alias page references, relationships, and documents managed by News24's editors, who can link stray entities together, add roles and photos, and flag entities of interest.

  • Audited changes

    Every edit in the portal audited with who made it, with portal users and permissions per section, and imports run from the portal as new material came in.

05What each team gets

  • Readers

    • Every person and organisation in the commission's documents, in one search
    • Where each appears, and how often
    • The original page, a click away

  • Journalists & researchers

    • Connections between people and organisations, drawn as a diagram
    • Spellings and OCR errors already merged
    • Pages and documents to cite

  • News24 editors

    • A portal to review and correct entities
    • People and organisations of interest featured on the site
    • Roles, photos, and links added where the models fell short

06How we delivered

  1. 2021–2022

    OCR & extraction

    OCR and traditional NLP made the commission's growing collection of documents searchable by the people and organisations they mentioned, with work beginning before ChatGPT and today's LLM tools.

  2. Late 2022

    Matching & the Explorer

    Consistent entity records, source-document imports, the editors' portal, and the public Explorer brought the archive into a searchable service.

  3. 2023

    Relationships & refinement

    Improved document handling, entity roles and photos, clear page references, and refined relationships made the archive easier to explore and verify.

  4. 2024

    Support & maintenance

    Supported the Explorer for readers and researchers.

07Built to last

  • Matching brings spelling variations and OCR errors together so readers can find all references to the same person or organisation.
  • People and organisations have consistent records with their alternative names visible.
  • Quality checks and editorial review help keep irrelevant or incorrect mentions out of the public explorer.
  • Every mention remains linked to its original document and page so readers can verify the evidence.
  • Relationships are reviewed and linked to their source material, supporting research without losing context.
  • Search and entity pages remain responsive across more than 250,000 pages of source material.
  • .NET
  • ASP.NET Core
  • SQL Server
  • Angular
  • NLP
  • OCR

Sitting on a mountain of documents?

We'll talk through how OCR and natural language processing could pull the people, organisations, and connections out of your archive, and how to put them in front of readers or investigators with every claim traceable to its page. Book a free 30-minute call with one of our engineers.

sales@happen.software · www.happen.software/contact

Keep reading

Related case studies

All case studies