We tested our memory against a folder of docs. It tied.

Last week we ran a test that could have gone badly for us. We wanted to know, honestly, whether a structured memory answers questions better than the obvious alternative. Not a vector database. The real alternative: a folder of files and an AI agent that can search through them.

So we built exactly that, and put the two head to head.

The setup

• Two rounds. Round one used a single document. Round two used a corpus of 35 documents.
• The same content went into both systems: loaded into an Ontonym memory on one side, kept as plain files on the other.
• Both sides got the same questions. A judge scored the answers blind — it never knew which system wrote which answer.

The result

A tie. 90 points to 90, in both rounds. And zero hallucination on either side — neither system made anything up.

We could have quietly shelved this. We’re publishing it because the tie taught us more than a win would have.

What the tie actually means

On a small, clean corpus, a good model with file search is genuinely strong. If your knowledge is 35 tidy markdown files, an agent reading them will answer well. Anyone selling you a memory product should be able to say that out loud.

But answer quality on a frozen snapshot was never the point. The test couldn’t measure the things that break file-based knowledge in real life:

• Files don’t know who is allowed to read them. A memory checks permissions on every fact.
• Files don’t tell you they’re stale. When two documents disagree, the agent picks one and moves on. A memory holds one current version of each fact.
• Files have no write path. When the agent learns something new, where does it put it? Appending to a text file is not an answer a team can share.

The tie also paid for itself in a very direct way: it flushed out a real bug in how our event objects inherited their date and summary fields. We fixed it before any customer hit it.

We’ll rerun the test as the corpus grows and gets messier — more files, more contradictions, more time passing. That’s where we expect the two lines to separate. When we have those numbers, we’ll publish them too, whichever way they point.