{ "cells": [ { "cell_type": "markdown", "id": "workshop-00", "metadata": {}, "source": [ "# Your RAG Is Only as Good as Your PDFs\n", "**LlamaParse + LiteParse · Sacha Bron · OpenTech Connect, 12 November 2026**\n", "\n", "Build a small document assistant, change only its parser, and inspect the evidence behind its answers.\n", "\n", "**90 minutes:** setup 10 · parsing 25 · RAG 20 · evaluation 20 · experiments 15.\n", "\n", "You need basic Python. The PDF and the questions are included. No external vector database.\n", "Start this notebook with `uv run --group notebook jupyter lab workshop.ipynb` from the project folder.\n", "\n", "**Live mode:** LlamaParse sends the PDF to its API; embeddings and answers use OpenAI with either parser.\n", "LiteParse itself runs locally. API calls may consume credits. Only fictional data is included.\n", "\n", "**Replay mode:** set `REPLAY = True` below to inspect saved, genuine outputs without API calls.\n", "Replay does not run a new parser, retrieval, or LLM inference.\n", "Credentials are loaded from the project `.env`: `LLAMA_CLOUD_API_KEY` and `OPENAI_API_KEY`. Exported environment variables take precedence.\n", "After editing `.env`, restart the kernel. After changing the shell environment, restart Jupyter too.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "workshop-01", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mode: LIVE\n" ] } ], "source": [ "import hashlib\n", "import json\n", "import os\n", "from pathlib import Path\n", "from IPython.display import IFrame, Markdown, display\n", "from workshop import DATA, PARSERS, ROOT, parse_pdf, build_engine, ask, show\n", "\n", "REPLAY = False # True = inspect recorded results, no API calls\n", "PDF = DATA / \"alpine_solar.pdf\"\n", "QUESTIONS = json.loads((DATA / \"questions.json\").read_text())\n", "\n", "os.environ.setdefault(\"OPENAI_MODEL\", \"gpt-6-luna\")\n", "os.environ.setdefault(\"OPENAI_EMBEDDING_MODEL\", \"text-embedding-3-small\")\n", "os.environ.setdefault(\"LLAMA_PARSE_TIER\", \"agentic\")\n", "os.environ.setdefault(\"LLAMA_PARSE_VERSION\", \"latest\")\n", "\n", "def recorded_pages(parser):\n", " record = json.loads((ROOT / \"recordings\" / f\"{parser}.json\").read_text())\n", " assert record[\"sha256\"] == hashlib.sha256(PDF.read_bytes()).hexdigest(), \"PDF changed: use live mode.\"\n", " return record[\"pages\"]\n", "\n", "print(\"Mode:\", \"REPLAY — saved results\" if REPLAY else \"LIVE\")" ] }, { "cell_type": "markdown", "id": "workshop-02", "metadata": {}, "source": [ "## 1. Read the PDF before asking the model\n", "Three pages: operating notes, a table with grouped headers, and a chart.\n", "This is an intentionally small, fictional teaching corpus, not a representative benchmark.\n", "\n", "Open the PDF and find the answer yourself:\n", "- When is maintenance scheduled?\n", "- What is Sion's Q2 margin **in CHF**, not thousands of CHF?\n", "- What was April's energy production?\n", "- Is there a revenue forecast for 2027?\n", "\n", "Notice that the chart has no numeric labels on its bars." ] }, { "cell_type": "code", "execution_count": 2, "id": "workshop-03", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(IFrame(\"data/alpine_solar.pdf\", width=\"100%\", height=550))" ] }, { "cell_type": "markdown", "id": "workshop-04", "metadata": {}, "source": [ "## 2. Parse locally with LiteParse\n", "The core API is just:\n", "```python\n", "from liteparse import LiteParse\n", "result = LiteParse(output_format=\"markdown\", ocr_enabled=True).parse(PDF)\n", "print(result.pages[0].markdown)\n", "```\n", "`parse_pdf` in `workshop.py` makes that call, keeps physical page numbers, and saves its actual output.\n", "The cache avoids repeating work. Open the function if you want to inspect it.\n", "\n", "No API key is needed. OCR language data may download on first use; run once before an offline workshop." ] }, { "cell_type": "code", "execution_count": 3, "id": "workshop-05", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "liteparse: cache / alpine_solar.pdf\n", "\n", "--- Page 1 ---\n", "# Operating overview\n", "\n", "Reporting period: January-June 2026\n", "\n", "## The company\n", "\n", "Alpine Solar operates rooftop solar installations in western Switzerland. The portfolio has three sites: Lausanne, Sion and Neuchatel. All three sites remained operational throughout the reporting period.\n", "\n", "## Maintenance\n", "\n", "The Lausanne site is scheduled for maintenance on 18 November 2026. The planned outage is four hours, from 08:00 to 12:00 local time. Sion and Neuchatel have no scheduled outage in November.\n", "\n", "## Reporting scope\n", "\n", "The reporting perimeter covers all three operating sites. Revenue reflects electricity sales during the reporting period. Operating costs include maintenance, site leases and insurance.\n", "\n", "--- Page 2 ---\n", "### Quarterly operating results\n", "\n", "CHF thousands / Revenue and operating costs by site\n", "\n", "**Q1 2026 Q2 2026 Site**\n", "\n", "**Revenue Costs Revenue Costs**\n", "\n", "---\n", "\n", "| Lausanne | 120 | 80 | 150 | 90 |\n", "|---|---|---|---|---|\n", "| Sion | 95 | 60 | 140 | 70 |\n", "| Neuchatel | 80 | 55 | 100 | 65 |\n", "| Total | 295 | 195 | 390 | 225 |\n", "\n", "Operating costs exclude financing costs and corporate income tax.\n", "\n", "--- Page 3 ---\n", "# Monthly energy production\n", "\n", "Portfolio total / January-June 2026 / MWh\n", "\n", "300\n", "\n", "---\n", "\n", "250\n", "\n", "---\n", "\n", "200\n", "\n", "---\n", "\n", "150\n", "\n", "---\n", "\n", "100\n", "\n", "i\n", "\n", "---\n", "\n", "50\n", "\n", "---\n", "\n", "0\n", "\n", "---\n", "\n", "Jan Feb Mar Apr May Jun\n", "\n", "Source: consolidated site meter readings.\n" ] } ], "source": [ "lite_pages = recorded_pages(\"liteparse\") if REPLAY else parse_pdf(PDF, \"liteparse\")\n", "for page in lite_pages:\n", " print(f\"\\n--- Page {page['page']} ---\\n{page['text']}\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "workshop-05-rendered", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "### LiteParse — rendered output" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 1" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "# Operating overview\n", "\n", "Reporting period: January-June 2026\n", "\n", "## The company\n", "\n", "Alpine Solar operates rooftop solar installations in western Switzerland. The portfolio has three sites: Lausanne, Sion and Neuchatel. All three sites remained operational throughout the reporting period.\n", "\n", "## Maintenance\n", "\n", "The Lausanne site is scheduled for maintenance on 18 November 2026. The planned outage is four hours, from 08:00 to 12:00 local time. Sion and Neuchatel have no scheduled outage in November.\n", "\n", "## Reporting scope\n", "\n", "The reporting perimeter covers all three operating sites. Revenue reflects electricity sales during the reporting period. Operating costs include maintenance, site leases and insurance." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 2" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "### Quarterly operating results\n", "\n", "CHF thousands / Revenue and operating costs by site\n", "\n", "**Q1 2026 Q2 2026 Site**\n", "\n", "**Revenue Costs Revenue Costs**\n", "\n", "---\n", "\n", "| Lausanne | 120 | 80 | 150 | 90 |\n", "|---|---|---|---|---|\n", "| Sion | 95 | 60 | 140 | 70 |\n", "| Neuchatel | 80 | 55 | 100 | 65 |\n", "| Total | 295 | 195 | 390 | 225 |\n", "\n", "Operating costs exclude financing costs and corporate income tax." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 3" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "# Monthly energy production\n", "\n", "Portfolio total / January-June 2026 / MWh\n", "\n", "300\n", "\n", "---\n", "\n", "250\n", "\n", "---\n", "\n", "200\n", "\n", "---\n", "\n", "150\n", "\n", "---\n", "\n", "100\n", "\n", "i\n", "\n", "---\n", "\n", "50\n", "\n", "---\n", "\n", "0\n", "\n", "---\n", "\n", "Jan Feb Mar Apr May Jun\n", "\n", "Source: consolidated site meter readings." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Render Markdown and embedded HTML, including tables with merged headers.\n", "display(Markdown(\"### LiteParse — rendered output\"))\n", "for page in lite_pages:\n", " display(Markdown(f\"#### Page {page['page']}\"))\n", " display(Markdown(page[\"text\"]))\n" ] }, { "cell_type": "markdown", "id": "workshop-06", "metadata": {}, "source": [ "**Pause:** Are the table headers still attached to the right columns? Does the chart output contain\n", "month-to-value pairs, or only axis labels? Text extraction and visual interpretation are different tasks." ] }, { "cell_type": "markdown", "id": "workshop-07", "metadata": {}, "source": [ "## 3. Parse the same PDF with LlamaParse\n", "The commercial API is exposed by the `llama-cloud` SDK:\n", "```python\n", "from llama_cloud import LlamaCloud\n", "with LlamaCloud() as client:\n", " file = client.files.create(file=PDF, purpose=\"parse\")\n", " result = client.parsing.parse(\n", " file_id=file.id, tier=\"agentic\", version=\"latest\", expand=[\"markdown\"],\n", " )\n", "```\n", "The SDK handles polling. We keep the returned per-page Markdown/HTML unchanged.\n", "The same `parse_pdf` helper adds caching and rejects failed pages.\n", "\n", "The SDK reads `LLAMA_CLOUD_API_KEY` from the environment, including the project `.env` loaded at startup.\n", "Copy `.env.example` to `.env` and fill in the keys, or export them before starting Jupyter.\n", "Do not enter credentials in notebook cells. The `.env` file is ignored by Git." ] }, { "cell_type": "code", "execution_count": 5, "id": "workshop-08", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "llamaparse: cache / alpine_solar.pdf\n", "\n", "--- Page 1 ---\n", "## Operating overview\n", "\n", "Reporting period: January-June 2026\n", "\n", "### The company\n", "\n", "Alpine Solar operates rooftop solar installations in western Switzerland.\n", "\n", "The portfolio has three sites: Lausanne, Sion and Neuchatel.\n", "\n", "All three sites remained operational throughout the reporting period.\n", "\n", "### Maintenance\n", "\n", "The Lausanne site is scheduled for maintenance on 18 November 2026.\n", "\n", "The planned outage is four hours, from 08:00 to 12:00 local time.\n", "\n", "Sion and Neuchatel have no scheduled outage in November.\n", "\n", "### Reporting scope\n", "\n", "The reporting perimeter covers all three operating sites.\n", "\n", "Revenue reflects electricity sales during the reporting period.\n", "\n", "Operating costs include maintenance, site leases and insurance.\n", "\n", "--- Page 2 ---\n", "## Quarterly operating results\n", "\n", "CHF thousands / Revenue and operating costs by site\n", "\n", "
SiteQ1 2026Q2 2026
RevenueCostsRevenueCosts
Lausanne1208015090
Sion956014070
Neuchatel805510065
Total295195390225
\n", "\n", "Operating costs exclude financing costs and corporate income tax.\n", "\n", "--- Page 3 ---\n", "## **Monthly energy production**\n", "\n", "Portfolio total / January-June 2026 / MWh\n", "\n", "
MonthEnergy production (MWh)
Jan100
Feb150
Mar200
Apr250
May200
Jun300
\n", "\n", "Source: consolidated site meter readings.\n" ] } ], "source": [ "llama_pages = recorded_pages(\"llamaparse\") if REPLAY else parse_pdf(PDF, \"llamaparse\")\n", "for page in llama_pages:\n", " print(f\"\\n--- Page {page['page']} ---\\n{page['text']}\")" ] }, { "cell_type": "code", "execution_count": 6, "id": "workshop-08-rendered", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "### LlamaParse — rendered output" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 1" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "## Operating overview\n", "\n", "Reporting period: January-June 2026\n", "\n", "### The company\n", "\n", "Alpine Solar operates rooftop solar installations in western Switzerland.\n", "\n", "The portfolio has three sites: Lausanne, Sion and Neuchatel.\n", "\n", "All three sites remained operational throughout the reporting period.\n", "\n", "### Maintenance\n", "\n", "The Lausanne site is scheduled for maintenance on 18 November 2026.\n", "\n", "The planned outage is four hours, from 08:00 to 12:00 local time.\n", "\n", "Sion and Neuchatel have no scheduled outage in November.\n", "\n", "### Reporting scope\n", "\n", "The reporting perimeter covers all three operating sites.\n", "\n", "Revenue reflects electricity sales during the reporting period.\n", "\n", "Operating costs include maintenance, site leases and insurance." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 2" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "## Quarterly operating results\n", "\n", "CHF thousands / Revenue and operating costs by site\n", "\n", "
SiteQ1 2026Q2 2026
RevenueCostsRevenueCosts
Lausanne1208015090
Sion956014070
Neuchatel805510065
Total295195390225
\n", "\n", "Operating costs exclude financing costs and corporate income tax." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "#### Page 3" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "## **Monthly energy production**\n", "\n", "Portfolio total / January-June 2026 / MWh\n", "\n", "
MonthEnergy production (MWh)
Jan100
Feb150
Mar200
Apr250
May200
Jun300
\n", "\n", "Source: consolidated site meter readings." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Render Markdown and embedded HTML, including tables with merged headers.\n", "display(Markdown(\"### LlamaParse — rendered output\"))\n", "for page in llama_pages:\n", " display(Markdown(f\"#### Page {page['page']}\"))\n", " display(Markdown(page[\"text\"]))\n" ] }, { "cell_type": "markdown", "id": "workshop-09", "metadata": {}, "source": [ "**Exercise:** compare page 3 with the original PDF. Are all six values and month associations correct?\n", "Do the same for the merged headers on page 2. Do not assume the commercial parser always wins.\n", "\n", "The local cache is under `.cache//`, as JSON and readable Markdown.\n", "Use `parse_pdf(PDF, \"llamaparse\", refresh=True)` to request a new parse, which can incur API usage.\n", "The PDF contents, parser version and settings are part of the cache key. A cached `latest` result stays\n", "cached until refreshed; it does not silently update when the hosted model changes." ] }, { "cell_type": "markdown", "id": "workshop-10", "metadata": {}, "source": [ "## 4. Compare both parsers with the same RAG pipeline\n", "`Document` → `TokenTextSplitter` → `VectorStoreIndex` → `CitationQueryEngine`.\n", "\n", "Both assistants use the same embedding model, LLM, prompt, chunk sizes and retrieval settings.\n", "Only the parsed text changes. Each chunk retains its filename and physical PDF page number.\n", "The LLM receives text, **not the original PDF or its images**.\n", "\n", "The vector index lives in memory. It is rebuilt here, so this step calls the embedding API again.\n", "The source PDF is small enough for this to stay easy to understand." ] }, { "cell_type": "code", "execution_count": 7, "id": "workshop-11", "metadata": {}, "outputs": [], "source": [ "pages_by_parser = {\"liteparse\": lite_pages, \"llamaparse\": llama_pages}\n", "engines = {}\n", "if not REPLAY:\n", " engines = {name: build_engine(pages) for name, pages in pages_by_parser.items()}\n", "else:\n", " recorded_run = json.loads((ROOT / \"recordings\" / \"comparison.json\").read_text())\n", " recorded_answers = recorded_run[\"results\"]\n", " print(\"Recorded model:\", recorded_run[\"settings\"][\"llm\"])\n", " print(\"REPLAY: no index is built; answers below come from the recorded run.\")" ] }, { "cell_type": "markdown", "id": "workshop-12", "metadata": {}, "source": [ "## 5. Ask a first question and inspect the citation\n", "`show()` displays the filename and physical PDF page directly, e.g. `[alpine_solar.pdf, p. 2]`.\n", "The list below the answer distinguishes cited passages from retrieved passages that were not cited.\n", "\n", "Internally, the model uses source IDs such as `[1]`, which restart for each question.\n", "These IDs identify retrieved excerpts, not PDF pages. The raw answer and sources remain available\n", "in `answer[\"answer\"]` and `answer[\"sources\"]` for inspection.\n", "\n", "A citation is not proof that the claim is correct. Read the actual excerpt.\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "workshop-13", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "=== liteparse ===\n", "Maintenance at the Lausanne site is scheduled for 18 November 2026, from 08:00 to 12:00 local time—an outage of four hours. [alpine_solar.pdf, p. 1]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 1 (cited)\n", " alpine_solar.pdf, p. 2 (not cited)\n", "\n", "=== llamaparse ===\n", "Maintenance at the Lausanne site is scheduled for 18 November 2026, from 08:00 to 12:00 local time—an outage of four hours. [alpine_solar.pdf, p. 1]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 1 (cited)\n", " alpine_solar.pdf, p. 2 (not cited)\n", "\n", "Evidence actually given to the LLM:\n", "Source 1:\n", "file_name: alpine_solar.pdf\n", "page: 1\n", "\n", "## Operating overview\n", "\n", "Reporting period: January-June 2026\n", "\n", "### The company\n", "\n", "Alpine Solar operates rooftop solar installations in western Switzerland.\n", "\n", "The portfolio has three sites: Lausanne, Sion and Neuchatel.\n", "\n", "All three sites remained operational throughout the reporting period.\n", "\n", "### Maintenance\n", "\n", "The Lausanne site is scheduled for maintenance on 18 November 2026.\n", "\n", "The planned outage is four hours, from 08:00 to 12:00 local time.\n", "\n", "Sion and Neuchatel have no scheduled outage in November.\n", "\n", "### Reporting scope\n", "\n", "The reporting perimeter covers all three operating sites.\n", "\n", "Revenue reflects electricity sales during the reporting period.\n", "\n", "Operating costs include maintenance, site leases and insurance.\n", "\n" ] } ], "source": [ "first_answers = {}\n", "question = QUESTIONS[0]\n", "for name in PARSERS:\n", " print(f\"\\n=== {name} ===\")\n", " answer = (recorded_answers[name][0] if REPLAY else ask(engines[name], question[\"question\"]))\n", " first_answers[name] = answer\n", " show(answer)\n", "\n", "print(\"\\nEvidence actually given to the LLM:\")\n", "print(first_answers[\"llamaparse\"][\"sources\"][0][\"text\"])" ] }, { "cell_type": "markdown", "id": "workshop-14", "metadata": {}, "source": [ "## 6. Evaluate the same five questions\n", "We reuse the first answer. The other questions test a calculation, visual information, and abstention.\n", "**The answer key is never indexed or given to the model.**\n", "\n", "Before revealing it, label each answer: correct / incorrect / abstains.\n", "Check citation support separately. A correct-sounding answer can cite a page that lacks the required values." ] }, { "cell_type": "code", "execution_count": 9, "id": "workshop-15", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "QUESTION: What is Sion's operating margin in Q2 2026, in CHF? Show the calculation.\n", "\n", "--- liteparse ---\n", "Sion’s Q2 2026 operating margin is **CHF 70 thousand**:\n", "\n", "**Revenue − operating costs = CHF 140 thousand − CHF 70 thousand = CHF 70 thousand.** [alpine_solar.pdf, p. 2]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 2 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "--- llamaparse ---\n", "Sion’s Q2 2026 operating margin was **CHF 70,000**: revenue of CHF 140,000 minus operating costs of CHF 70,000 = **CHF 70,000**. [alpine_solar.pdf, p. 2]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 2 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "QUESTION: How much energy did the portfolio produce in April 2026, in MWh?\n", "\n", "--- liteparse ---\n", "I cannot determine the April 2026 energy production from the sources. The chart provides no readable monthly value for April, and I should not infer it from the axis labels. [alpine_solar.pdf, p. 3]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 3 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "--- llamaparse ---\n", "The portfolio produced **250 MWh** in April 2026. [alpine_solar.pdf, p. 3]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 3 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "QUESTION: Which month had the highest energy production, and how much more did it produce than January, in MWh?\n", "\n", "--- liteparse ---\n", "I cannot determine which month had the highest production or how much more it produced than January. The chart provides month labels and a vertical scale, but no readable monthly values; bar heights cannot be inferred from the axis labels. [alpine_solar.pdf, p. 3]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 3 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "--- llamaparse ---\n", "June had the highest energy production at 300 MWh. That was 200 MWh more than January: 300 − 100 = 200 MWh. [alpine_solar.pdf, p. 3]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 3 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "QUESTION: What is Alpine Solar's revenue forecast for the full year 2027, in CHF?\n", "\n", "--- liteparse ---\n", "I cannot determine Alpine Solar’s revenue forecast for full-year 2027 from the sources. They report revenue for Q1 and Q2 2026 only, not a 2027 forecast. [alpine_solar.pdf, p. 1][alpine_solar.pdf, p. 2]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 1 (cited)\n", " alpine_solar.pdf, p. 2 (cited)\n", "\n", "--- llamaparse ---\n", "The sources do not provide a revenue forecast for 2027, so it cannot be determined from the available information. [alpine_solar.pdf, p. 1][alpine_solar.pdf, p. 2]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 1 (cited)\n", " alpine_solar.pdf, p. 2 (cited)\n" ] } ], "source": [ "results = {name: [{\"id\": QUESTIONS[0][\"id\"], **first_answers[name]}] for name in PARSERS}\n", "for question in QUESTIONS[1:]:\n", " print(f\"\\nQUESTION: {question['question']}\")\n", " for name in PARSERS:\n", " answer = (next(a for a in recorded_answers[name] if a[\"id\"] == question[\"id\"])\n", " if REPLAY else ask(engines[name], question[\"question\"]))\n", " results[name].append({\"id\": question[\"id\"], **answer})\n", " print(f\"\\n--- {name} ---\")\n", " show(answer)" ] }, { "cell_type": "markdown", "id": "workshop-16", "metadata": {}, "source": [ "## 7. Reveal the answer key\n", "For the chart, compare with the original image, not just with another parser's output.\n", "For the unknown forecast, abstention is the correct behavior.\n", "\n", "On a question with an answer, distinguish a justified abstention (information lost by the parser)\n", "from a wrong, confident answer. A prompt requesting abstention cannot guarantee it." ] }, { "cell_type": "code", "execution_count": 10, "id": "workshop-17", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "text: 18 November 2026, four hours (08:00-12:00 local time). | PDF pages: [1]\n", "table: CHF 70,000: (140 - 70) x 1,000. | PDF pages: [2]\n", "chart: 250 MWh. | PDF pages: [3]\n", "chart_comparison: June, 300 MWh; 200 MWh more than January (100 MWh). | PDF pages: [3]\n", "missing: Not provided. The assistant should abstain. | PDF pages: []\n" ] } ], "source": [ "answer_key = json.loads((DATA / \"answer_key.json\").read_text())\n", "for question in QUESTIONS:\n", " key = answer_key[question[\"id\"]]\n", " print(f\"{question['id']}: {key['answer']} | PDF pages: {key['pages']}\")" ] }, { "cell_type": "markdown", "id": "workshop-18", "metadata": {}, "source": [ "## 8. Try your own question\n", "Change the question and rerun this cell in live mode. Ask in French if you prefer.\n", "Investigate retrieval failures by printing `answer[\"sources\"]`.\n", "\n", "In replay mode, only the five saved questions are available." ] }, { "cell_type": "code", "execution_count": 11, "id": "workshop-19", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "=== liteparse ===\n", "La marge totale au deuxième trimestre 2026 est de **165 000 CHF** : revenus de 390 000 CHF moins coûts d’exploitation de 225 000 CHF [alpine_solar.pdf, p. 2].\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 2 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n", "\n", "=== llamaparse ===\n", "La marge totale au deuxième trimestre 2026 est de **165 000 CHF** : revenus de 390 000 CHF moins coûts d’exploitation de 225 000 CHF (390 − 225 = 165, en milliers de CHF). [alpine_solar.pdf, p. 2]\n", "\n", "Retrieved passages:\n", " alpine_solar.pdf, p. 2 (cited)\n", " alpine_solar.pdf, p. 1 (not cited)\n" ] } ], "source": [ "my_question = \"Quelle est la marge totale du portefeuille au deuxième trimestre 2026, en CHF ?\"\n", "if not REPLAY:\n", " for name in PARSERS:\n", " print(f\"\\n=== {name} ===\")\n", " show(ask(engines[name], my_question))\n", "else:\n", " print(\"Switch REPLAY to False and rerun the notebook to ask a new question.\")" ] }, { "cell_type": "markdown", "id": "workshop-20", "metadata": {}, "source": [ "## 9. Experiments and discussion\n", "Choose one:\n", "1. Replace the PDF with a public document of your own; write a ground truth before running it.\n", "2. Change LlamaParse from `agentic` to `agentic_plus`. Reparse and rebuild its index.\n", "3. Change `similarity_top_k` in `build_engine` for **both** assistants. Did retrieval cause the error?\n", "4. Change chunk size for **both** assistants. Did it split a table or lose its header?\n", "\n", "Keep one variable at a time. These few examples demonstrate failure modes; they do not establish a\n", "universal ranking of parsers. LiteParse is a strong fit for fast local extraction. LlamaParse adds\n", "visual understanding for documents where the layout itself carries information.\n", "\n", "The assistant is a demo: no chat history, no deployed service, no LLM judge, no external vector database.\n", "\n", "References: [LiteParse](https://developers.llamaindex.ai/liteparse/),\n", "[LlamaParse](https://developers.llamaindex.ai/llamaparse/parse/getting_started/),\n", "[CitationQueryEngine](https://developers.llamaindex.ai/python/examples/query_engine/citation_query_engine/)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.5" } }, "nbformat": 4, "nbformat_minor": 5 }