Perceptron DataWeekly check: PDF to Markdown
Every Sunday an automated watchdog runs the live actors on Apify with fixed inputs whose correct answers are known, and checks the content of the results — not only whether the run succeeded. The results are published here as they are, failures included.
First public check on 25 Oct 2026. Until then this page lists what will be checked.
| Case | Result | Time | What is checked |
|---|---|---|---|
| A 15-page research paper (arXiv 1706.03762v7) is converted in full: status ok, 15 of 15 pages charged as text pages, the paper title as a Markdown heading and the section text | – not checked yet | – |
|
| Table 2 of the same paper comes out as a table row: “Transformer (big)” with BLEU 28.4 (English–German) and 41.8 (English–French) | – not checked yet | – |
|
| An Excel file (pandas test file, fixed version) becomes a table with the exact cell values — whole numbers, decimals, TRUE/FALSE and dates | – not checked yet | – |
|
| A missing file (HTTP 404) is reported as failed with the reason and not charged — no empty document is delivered | – not checked yet | – |
|
Sample against the original source
On 12 Oct 2026 we took 20 result rows and compared 105 field values one by one with the original source: 73 matched (69.5%).
20 table rows from 4 public PDFs with tables (two arXiv papers, a US Census Bureau release, an IRS tax table), 5 rows per document chosen at random, converted on 12 Oct 2026 with the code of the live version (build 0.1.6). Every cell of these rows was compared with the original PDF — an independent text extraction (poppler pdftotext) and the rendered page. A cell counts when the source or our output has a value; it matches only if the text is identical apart from surrounding whitespace. After repairs released on 12 Oct 2026 (build 0.1.8), the same 20 rows match in 105 of 105 cells. These are the rows the repairs were aimed at, so this is not an independent estimate; 20 further rows from other tables of the same four files match in 112 of 112 cells (before: 88). A fresh sample from new documents follows.
| Document | Matched | Rate |
|---|---|---|
| arXiv 1706.03762v7 (Attention Is All You Need), page 8, Table 2 | 9 / 20 | 45.0% |
| arXiv 1512.03385v1 (Deep Residual Learning), page 6, Table 3 | 14 / 15 | 93.3% |
| US Census Bureau, Quarterly Retail E-Commerce Sales, 2nd quarter 2024, page 2, Table 1 | 40 / 40 | 100.0% |
| IRS, 2023 Instructions for Form 1040 — Tax Table (prior-year archive), page 4 | 10 / 30 | 33.3% |
Every difference and its cause
| Row | Field | Actor output | Original source | Cause |
|---|---|---|---|---|
| arXiv 1706.03762v7, p. 8, Table 2, data row 1 | Model | ByteNet[18] | ByteNet [18] | Word spaces inside the cell were lost; the characters are otherwise identical. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 7 | Model | GNMT+RLEnsemble[38] | GNMT + RL Ensemble [38] | Word spaces inside the cell were lost; the characters are otherwise identical. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 7 | Training cost EN-DE | 1.8·1020 1.1·1021 | 1.8 · 10^20 | The two training-cost columns were merged into one cell, and the superscript exponent was flattened (10^20 became “1020”). |
| arXiv 1706.03762v7, p. 8, Table 2, data row 7 | Training cost EN-FR | – | 1.1 · 10^21 | The value sits in the neighbouring cell, which holds both training-cost columns. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 8 | Model | ConvS2SEnsemble[9] | ConvS2S Ensemble [9] | Word spaces inside the cell were lost; the characters are otherwise identical. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 8 | Training cost EN-DE | 7.7·1019 1.2·1021 | 7.7 · 10^19 | The two training-cost columns were merged into one cell, and the superscript exponent was flattened (10^20 became “1020”). |
| arXiv 1706.03762v7, p. 8, Table 2, data row 8 | Training cost EN-FR | – | 1.2 · 10^21 | The value sits in the neighbouring cell, which holds both training-cost columns. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 9 | Model | Transformer(basemodel) | Transformer (base model) | Word spaces inside the cell were lost; the characters are otherwise identical. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 9 | Training cost EN-DE | 3.3·1018 | 3.3 · 10^18 | The superscript exponent was flattened (10^18 became “1018”) — the number reads wrong. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 10 | Model | Transformer(big) | Transformer (big) | Word spaces inside the cell were lost; the characters are otherwise identical. |
| arXiv 1706.03762v7, p. 8, Table 2, data row 10 | Training cost EN-DE | 2.3·1019 | 2.3 · 10^19 | The superscript exponent was flattened (10^18 became “1018”) — the number reads wrong. |
| arXiv 1512.03385v1, p. 6, Table 3, data row 2 | model | GoogLeNet[44] | GoogLeNet [44] | Word spaces inside the cell were lost; the characters are otherwise identical. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 3 | At least | – | 3,100 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 3 | But less than | – | 3,150 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 3 | Single | – | 313 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 3 | Married filing separately | – | 313 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 8 | At least | – | 3,350 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 8 | But less than | – | 3,400 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 8 | Single | – | 338 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 8 | Married filing separately | – | 338 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 11 | At least | – | 3,500 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 11 | But less than | – | 3,550 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 11 | Single | – | 353 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 11 | Married filing separately | – | 353 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 14 | At least | – | 3,650 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 14 | But less than | – | 3,700 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 14 | Single | – | 368 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 14 | Married filing separately | – | 368 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 18 | At least | – | 3,850 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 18 | But less than | – | 3,900 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 18 | Single | – | 388 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
| IRS 2023 Instructions 1040 Tax Table (prior-year archive), p. 4, first block, data row 18 | Married filing separately | – | 388 | Dense table without row lines: for this column, all 20 rows of the block were packed into one cell of the block's first row — the value is in the output, but not in its row. |
What this shows: on the day of the check, the live actor gave the known answers for these inputs. It does not prove that every result is correct. The sample against the original source measures accuracy on real result rows.