Benchmark Report · v2 (industrial)

AIveilix vs AnythingLLM
22 documents · 80 questions · 6 formats

An industrial-grade head-to-head: 22 real-world documents (financial filings, scientific papers, government reports, whitepapers, RFCs, spreadsheets, images), 80 questions across 11 categories and 7 question types, scored by an independent LLM judge with a fixed rubric.

Section 01

Summary scoreboard

All metrics on the same 80-question eval set with the same 22 documents on both sides.

Section 02

Visual breakdown

Section 03

Where each system wins

Average correctness sliced by document category, question type, and file format.

Section 04

How the benchmark works

The corpus (22 documents, 6 formats)

• Scientific / arXiv (5): Attention Is All You Need, BERT, GPT-3, Llama 2, DALL-E
• Financial (3): Apple Q4 2023 results, Apple 2023 Environmental Report, Berkshire 2022 letter
• Crypto whitepapers (2): Bitcoin (Nakamoto), Ethereum (Buterin)
• Government / global (3): NASA Artemis Plan 2020, IMF WEO Oct 2023, UN SDG Report 2023
• Legal / standards (3): US Constitution, GPLv3, CDC MMWR Vol 72/4
• Technical RFCs (2): RFC 9110 (HTTP), RFC 8259 (JSON)
• Tabular data (2): iris.csv, titanic.csv
• Markdown / plain text (1): python_tutorial.md
• Images (1): chart_us_gdp.png (US GDP line chart)

Question types (80 total)

• Factual — single-fact retrieval
• Conceptual — definitions and explanations
• Multi-hop — needs ≥2 chunks combined
• Numeric — find a number in a doc/table
• Summarization — gist of a section
• Cross-doc — compare facts across multiple docs
• Visual — read a chart / image
• Hallucination-trap — the answer isn't in the docs. Should refuse, not invent.

Judge

an independent LLM judge (temperature=0 for deterministic scoring) scored each answer with a fixed rubric prompt. Same judge for both systems → bias cancels.

Section 05

Per-question deep dive

Click any question to expand both answers + the judge's reasoning.

Section 06

Takeaways