Perceptron DataWeekly check: Research Agent
Every Sunday an automated watchdog runs the live actors on Apify with fixed inputs whose correct answers are known, and checks the content of the results — not only whether the run succeeded. The results are published here as they are, failures included.
Last check on 12 Oct 2026: 10 of 10 cases passed, 1 of them with a note. Checked weekly since 8 Oct 2026.
| Case | Result | Time | What is checked |
|---|---|---|---|
| EU AI Act obligations – cited from the official text | ✅ passed | 59 s | 5 sources, strongest evidence: official |
| GDPR article lookup – official text with sources | ✅ passed | 30 s | 5 sources, strongest evidence: official |
| German law § lookup (BGB, HGB, GG) – official text | ✅ passed | 34 s | 7 sources, strongest evidence: official |
| Fact-check prices and figures across independent sites | ⚠️ passed with a note | 87 s | |
| Scientific evidence from open-access papers | ✅ passed | 3 s | 6 sources, strongest evidence: academic |
| Developer answers from Stack Overflow, GitHub and HN | ✅ passed | 45 s | 4 sources, strongest evidence: community |
| This week's news as cited context for LLMs | ✅ passed | 32 s | 7 sources, strongest evidence: official |
| RAG chunks for Pinecone or Qdrant with stable IDs | ✅ passed | 100 s | 6 sources, strongest evidence: academic |
| Web search API for AI agents – cited context in one call | ✅ passed | 5 s | 7 sources, strongest evidence: reference |
| Deep research: second search round with learned terms | ✅ passed | 49 s | 9 sources, strongest evidence: official |
Failures and notes from the last check
- ⚠️ passed with a note — Fact-check prices and figures across independent sites: Google did not answer (status partial, not charged) — usually temporary
Earlier checks
| Date | Passed |
|---|---|
| 12 Oct 2026 | 10 / 10 |
| 8 Oct 2026 | 10 / 10 |
Sample against the original source
On 12 Oct 2026 we took 20 result rows and compared 116 field values one by one with the original source: 108 matched (93.1%).
20 passages from the watchdog's runs of 12 Oct 2026 (9 questions, 20 different sources; per question the highest-ranked passages from different sources). Each was compared with the cited page fetched the same day with independent tools (lxml for HTML, pdftotext for PDF; EUR-Lex via the EU Publications Office, Stack Overflow via its API). Text: every word sequence of the passage must occur on the page (case, punctuation and whitespace ignored; indentation counts in code). Title and heading must match the page; the URL must answer; the date must be one the page states for itself; the evidence level must fit the actor's published definitions. After repairs released on 12 Oct 2026 (build 1.0.14), the same 20 rows give 115 of 116 matching fields. These are the rows the repairs were aimed at, so this is not an independent estimate; a fresh sample with new rows follows.
| Field | Matched | Rate |
|---|---|---|
| text | 18 / 20 | 90.0% |
| title | 19 / 20 | 95.0% |
| url | 20 / 20 | 100.0% |
| evidenceLevel | 19 / 20 | 95.0% |
| date | 17 / 20 | 85.0% |
| heading | 15 / 16 | 93.8% |
Every difference and its cause
| Row | Field | Actor output | Original source | Cause |
|---|---|---|---|---|
| german-law-bgb-paragraph-official · rank 1 · https://www.gesetze-im-internet.de/bgb/__438.html | date | 2022-01-01 | no date on the page | The page states no date; the date heuristic read “2022” from an HTML element id (blaettercontainer_2022). |
| deep-research-regulation-second-round · rank 1 · https://www.enisa.europa.eu/topics/vulnerability-services | date | 2023-02-16 | topic page without own date; 16 Feb 2023 is the first listed publication | Topic page without its own date; the date belongs to the first publication listed on it. |
| deep-research-regulation-second-round · rank 2 · https://www.enisa.europa.eu/topics/product-security/vulnerability-services/eu-incident-response-and-cyber-crisis-management/single-reporting-platform-srp/frequently-asked-questions | date | 2026-09-11 | FAQ page without own date; 11 Sep 2026 is a linked press release | FAQ page without its own date; the date belongs to a press release linked on it. |
| developer-answers-stack-overflow-github · rank 1 · https://stackoverflow.com/questions/79150620/closing-a-httpx-client-results-in-a-runtimeerror-event-loop-is-closed | text | code without the indentation of 'def __new__' (column 0) | def __new__(cls): (indented 4 spaces) | Words match, but in code blocks the first line after a blank line lost its indentation — Python code changes meaning. |
| news-this-week-cited-llm-context · rank 1 · https://www.taipeitimes.com/News/feat/archives/2026/10/08/2003865586 | evidenceLevel | commercial | news (Taipei Times is a newspaper) | Taipei Times is a news outlet; its domain is missing from the actor's news list, so the rule fell back to commercial. |
| scientific-evidence-open-access-papers · rank 2 · https://exonpublications.com/index.php/exon/article/download/intermittent-fasting-methods-health-benefits-risks/1604/ | text | OpenAlex abstract of the book 'Intermittent Fasting: Education for the Public' | PDF of the chapter 'Intermittent Fasting: Methods, Health Benefits, and Risks' with a different abstract | The passage is the book abstract from OpenAlex (it is on the DOI page); the cited URL is OpenAlex's PDF link, a chapter with a different abstract. |
| scientific-evidence-open-access-papers · rank 2 · https://exonpublications.com/index.php/exon/article/download/intermittent-fasting-methods-health-benefits-risks/1604/ | title | Intermittent Fasting: Education for the Public | Intermittent Fasting: Methods, Health Benefits, and Risks | Title from the OpenAlex record, the cited PDF carries another title (same cause as the text). |
| scientific-evidence-open-access-papers · rank 2 · https://exonpublications.com/index.php/exon/article/download/intermittent-fasting-methods-health-benefits-risks/1604/ | heading | Intermittent Fasting: Education for the Public | not in the cited PDF | Heading is the OpenAlex title (same cause as the text). |
What this shows: on the day of the check, the live actor gave the known answers for these inputs. It does not prove that every result is correct. The sample against the original source measures accuracy on real result rows.