Wrestling PDFs with PyMuPDF & Tesseract
In the previous article we had a look at the some common tools available for PDF processing. In this article, we're going to go deeper into how to create a pdf parsing pipeline so we can ingest large volumes of PDFs into a knowledge base for any number of downstream use-cases. The pipeline we discuss is one we built for internal processing and sharing of PDFs for our knowledge base. We consume a significant amount of knowledge weekly keeping up with the latest (or sometimes the evergreen) topics in the Data and AI space. Consolidating all our reading into one shared knowledge hub means we open up each persons' learnings. This pipeline extracts all highlighted text from PDFs and condenses it into a summary Markdown which we then upload to our knowledge base.
Local Processing
One of our key drivers for this project is to avoid using the Foundation AI models for this work. Now admittedly both Claude and Codex played a role in building out the code, but one of the aims of the project was to avoid using these models for the operational work. Why the distinction? A few reasons:
- For cost efficiency and certainty. LLM pricing is on a consistent upward trend and we wanted to be in control of this
- To make it secure. We wanted to create a secure pipeline where no data leaves our cloud infrastructure. This isn't so necessary for the current project, but in the future we may want to use it for confidential or private information.
- To support open source. We are huge fans of open source code and models and we are getting to the stage (some say we are already there) where open source models can and should handle a lot of work.
One of the key learnings of this project was that actually for the vast majority of the PDFs we process, we don't even have to use LLMs tuned for PDF extraction - the native Python libraries are good enough to do the work.
Graceful Cascades
There are so many libraries, packages and tools to extract data from PDFs and they all have their strengths and weaknesses. Some are excellent at extracting text from images, others at parsing tables while quite a few are focussed on academic papers. Good engineering design means understanding the trade-offs and choosing the best fit tool for the job. In this case we relied on a mix of libraries and tools. Starting with the simplest, we'd then only use more complex tooling as required by the complexity or quality of the pdf. Our escalation path was this:
- Naive PDF processing: PyMuPDF (& PyMuPDF4LLM)
- Optical Character Recognition: PyMuPDF, PyMuPDF4LLm & Tesseract
- LLM Fallback: Qwen2.5:14b
PyMuPDF (& PyMuPDF4LLM)
PyMuPDF is a fast Python library designed for data
extraction and parsing. The PyMuPDF4LLM
extension makes it easy to process entire PDFs in one go and outputs data into a clean
structured format (such as JSON, Markdown or TXT file formats).
Getting up and running is simple and it has the ability to convert from PDF, EPUB, SVG, TXT and MD files into PDF, SVG Image, Markdown and JSON formats. It's an effective tool and because it's so fast it results in very performant pipelines. For simple, computer generated PDFs this will do the job well.
from pathlib import Path
import pymupdf4llm
pdf_path = Path("input/some.pdf")
markdown: str = pymupdf4llm.to_markdown(pdf_path, use_ocr=False)
pdf_path.with_suffix(".md").write_text(markdown, encoding="utf-8")
That's all the code you need to convert an entire document into text. This will process the entire document (assuming it's not too big to fit into memory). If you wanted to process each page then you can drop back to PyMuPDF (which gives you the option to operate at a per-page level).
import time
from collections.abc import Iterator
from pathlib import Path
import pymupdf
import pymupdf4llm
@dataclass
class PageErrors():
page_number: int
message: str
def get_page_count(path: Path) -> int:
"""Retrieve size of pdf"""
with pymupdf.open(path) as doc:
return doc.page_count
def iterate_pages(
path: Path,
errors: list[PageErrors]
) -> Iterator[tuple[int, str]]:
"""Process each pdf page"""
total_pages = get_page_count(path)
for page_index in range(total_pages):
try:
markdown = pymupdf4llm.to_markdown(
str(path),
pages=[page_index],
use_ocr=False,
)
except Exception as error:
errors.append(
PageErrors(
page_index,
string(error)
)
)
yield page_index + 1, markdown
def process_page(page_number: int, markdown: str) -> None:
"""Whatever post-processing you want to do"""
print(f"Processing page {page_number}")
# downstream work here
def process_pdf(path: Path) -> list[PageErrors]:
"""Run the pipeline page by page. This returns any pages that failed"""
errors = []
for page_number, markdown in iterate_pages(path, errors):
process_page(page_number, markdown)
return errors
errors = process_pdf(Path("input/some.pdf"))
for error in errors:
print(f"Page: {error.page_numer} had error: {error.message}")In practice, processing PDFs page by page means you can isolate failures and identify what caused a particular page to fail. This also means if you are processing a large PDF that you don't need to rerun the process again from the very start.
To reprocess just the failed pages - you can then specify which pages should be used fairly easily.
import pymupdf4llm
error_pages = [error.page_number for error in errors]
markdown = pymupdf4llm.to_markdown(
"input/small-talk.pdf",
pages=error_pages,
)OCR with Tesseract
One thing you may have noticed in the previous examples is that we have forced the parsing engine to not use OCR. This is because the internal threshold for using OCR is fairly low, and PyMuPDF and PyMuPDF4LLM very quickly default to use OCR for all PDFs (even ones that can be parsed very easily). We prefer controlling this threshold ourselves and deciding when to use OCR. We've found that using OCR is about 20-30% slower than using naive pdf parsing (where the text layer is read directly).
If you do need to use OCR (for example if the PDF is encoded or a scan) then there are many OCR options available, the default for PyMuPDF is Tesseract, a C/C++ library shipped across all major operating systems and with many client libraries. It's development started at Hewlett-Packard in 1985 and was sponsored by Google between 2006-2018. It is completely free and open-source and powers many applications and use cases. In the latest versions it's internal engine has moved to using Long Short-Term Memory (LSTM) recurrent neural networks to achieve high accuracy.
Exactly how it works is beyond the scope of this article, but essentially it turns page images into machine-readable text. In a PDF pipeline, this usually means:
- Render each PDF page to an image.
- Send the image pixels to Tesseract.
- Tesseract recognises text from the image.
- Your app stores the result as plain text, Markdown, hOCR, TSV, or a searchable PDF text layer.
Important distinction: Tesseract itself is mainly an image OCR engine. It supports image inputs such as PNG, JPEG and TIFF, and can output formats including plain text, hOCR, TSV and searchable PDF. In a PDF workflow, a tool like PyMuPDF often renders PDF pages to images first and then calls Tesseract to perform OCR on these images.
Setup
First thing you need to do is ensure that tesseract language data files are installed in your environment and then specify the path to the directory as an environment variable. Here's the Dockerfile we use to containerise our project:
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim
ENV UV_COMPILE_BYTECODE=1 \
PYTHONPATH=/app/src
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
tesseract \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-install-project --no-dev
COPY src/ ./src/
COPY static/ ./static/
COPY README.md ./
EXPOSE 8080
CMD ["sh", "-c", "uvicorn project.app:app --host 0.0.0.0 --port ${PORT:-8080}"]
This should now allow PyMuPDF to find Tesseract, in some cases you may need to set the path to the language datasets. Do this by setting the following environment variable (with the default location for Unix systems):
# Unix
TESSDATA_PREFIX=/usr/share/tesseract-ocr/4.00/tessdata
# Mac (if installed via HomeBrew)
TESSDATA_PREFIX=/opt/homebrew/share/tessdata/
Now you can use Tesseract OCR within PyMuPDF and using it is as easy as setting the
use_ocr=True:
pymupdf4llm.to_markdown(
str(pdf_path),
force_ocr=True,
use_ocr=True,
show_progress=True
)
An interesting caveat is that on some PDFs the encoding can be broken on certain pages so OCR using the PyMuPDF4LLM may not work or you may want to process each page individually to create a pipeline. In these cases, you'll need to drop down to the lower level processing provided by PyMuPDF (which allows processing page by page). We've found on large PDFs, processing each page in a pipeline can improve performance by 10-20%. A pipeline that processes each page at a time then looks more like this:
TESSDATA = os.getenv("TESSDATA_PREFIX")
page_texts: list[str] = []
with pymupdf.open("input/some.pdf") as doc:
page_numbers = pages if pages is not None else list(range(doc.page_count))
for page_number in page_numbers:
page = doc[page_number]
textpage = page.get_textpage_ocr(
language="eng", dpi=300, full=True, tessdata=TESSDATA
)
text = page.get_text("text", textpage=textpage).strip()
page_texts.append(text)
markdown = "\n\n---\n\n".join(page_texts) + "\n"
pdf_path.with_suffix(".md").write_text(markdown, encoding="utf-8")
Highlights
Let's get back to the main reason we were doing all this processing in the first place. We wanted to extract highlights from PDF files. A highlight and the text it covers are two separate objects in the PDF that happen to overlap on screen.
Most annotation software will encode this information in one of a few ways. Highlight
annotations are stored in PDF_ANNOT_HIGHLIGHT, this is the proper way to markup a pdf
and is what most modern apps do. Each highlight is tightly bound to each line of text. Ink
annotations on the other hand are encoded in PDF_ANNOT_INK, these are free-hand
highlights with a loose bounding box. Finally there are pdf with baked-in colour fills - a
flattened/printed PDF where the highlight is a yellow rectangle in the page content - not
captured in the annotation layer at all.
To deal with each of these situations properly we inspect the file (selecting a number of pages) and running through the following methods to see if they reveal any information:
1. page.annots()
2. annot.type()
3. page.get_drawings()
We progressively walk down the highlight options:
- Collect the Highlight Rectangles: Walk the page's annotation list and extract the
coordinates of the highlight. If this is stored in
PDF_ANNOT_HIGHLIGHTyou get specific coordinates for the edge points of each highlight, if the more free-handPDF_ANNOT_INKthen you may have a wobbly line, we then just take the min/max of each edge point and pad it out.
def _highlight_rects(page) -> list[pymupdf.Rect]:
"""Return rectangles for supported highlight annotations on a page."""
rects: list[pymupdf.Rect] = []
annot = page.first_annot
while annot:
annot_type = annot.type[0]
if annot_type == pymupdf.PDF_ANNOT_HIGHLIGHT:
vertices = annot.vertices or []
if vertices and len(vertices) % 4 == 0:
for i in range(0, len(vertices), 4):
rects.append(pymupdf.Quad(vertices[i : i + 4]).rect)
else:
rects.append(annot.rect)
elif annot_type == pymupdf.PDF_ANNOT_INK and _is_highlighter_yellow(
annot.colors.get("stroke")
):
rects.extend(_ink_stroke_rects(annot))
annot = annot.next
return rects
def _ink_stroke_rects(annot, padding: float = INK_STROKE_PADDING) -> list[pymupdf.Rect]:
"""Return padded rectangles for ink annotation strokes."""
vertices = getattr(annot, "vertices", None) or []
rects: list[pymupdf.Rect] = []
if vertices and isinstance(vertices[0], list):
try:
for stroke in vertices:
if not stroke:
continue
xs = [float(point[0]) for point in stroke]
ys = [float(point[1]) for point in stroke]
rects.append(
pymupdf.Rect(
min(xs) - padding,
min(ys) - padding,
max(xs) + padding,
max(ys) + padding,
)
)
except (IndexError, TypeError, ValueError):
return [annot.rect]
return rects or [annot.rect]
- Get the Text Coordinates: Extract content from each page as nested blocks -> lines -> spans. Each line also has start-finish coordinates. Now we have both sets of coordinates with the same origin.
def convert_highlighted(
pdf_path: Path, *, page_range: PageRange | None = None
) -> list[str]:
"""Extract only text lines covered by supported highlight annotations."""
logger.info("Converting PDF with highlighted-line extraction")
doc = pymupdf.open(pdf_path)
pages: list[str] = []
highlighted_lines = 0
page_indexes = _selected_page_indexes(_page_count(doc), page_range)
for page_index in page_indexes:
page = doc[page_index]
rects = _highlight_rects(page)
lines: list[str] = []
for block in page.get_text("dict")["blocks"]:
for line in block.get("lines", []):
line_rect = pymupdf.Rect(line["bbox"])
if _line_coverage(line_rect, rects) >= HIGHLIGHT_LINE_COVERAGE:
text = "".join(span["text"] for span in line["spans"]).strip()
if text:
lines.append(text)
highlighted_lines += len(lines)
pages.append("\n".join(lines).strip())
if highlighted_lines == 0:
raise NoHighlightsFound("No highlights were found.")
return pages
- Measure Overlap: For each line measure the intersection between the highlights and text and this gives your the highlighted text. A line is deemed highlighted if more than 50% of it is covered by a highlight box.
def _line_coverage(line_rect: pymupdf.Rect, highlight_rects: list[pymupdf.Rect]) -> float:
"""Return the fraction of a text line width covered by highlight regions."""
if line_rect.width <= 0:
return 0
intervals: list[tuple[float, float]] = []
for highlight_rect in highlight_rects:
intersection = line_rect & highlight_rect
if intersection.is_empty or intersection.width <= 0 or intersection.height <= 0:
continue
intervals.append((intersection.x0, intersection.x1))
covered_width = sum(end - start for start, end in _merge_intervals(intervals))
return covered_width / line_rect.width
Cost and Security
Now one of the best things about this highlights extraction pipeline is that it is small and quick to run and we can run it in our own Google Cloud Project. No LLMs, no subscriptions and scale down to zero when we need to. This is deployed to a Cloud Run instance and protected by authentication so only authorised (internal) users can use it. Because this runs on stateless infrastructure - there is a slight cold-start issue for the container to fire up, however this is more than acceptable for our use case, but once live this pipeline tears through PDFs. A 200 page pdf can be processed in 7 seconds, and the output is highly accurate and deterministic.
We do have another fallover processing stage where we can use an LLM, however we rarely have the need to fall back on it. We'll cover that in an upcoming piece.
Wrap Up
In this article we showed how we've implemented a PDF parsing pipeline using PyMuPDF, PyMuPDF4LLM and Tesseract to create a high accuracy, resilient, secure and cheap to run pipeline. This pipeline takes in a wide range of PDFs that have been marked up or highlighted and extracts the useful information from them, all this is done securely in our own Google Cloud Project and the resulting Markdown files are sent downstream to our knowledge base., resilient, secure and cheap to run pipeline. This pipeline takes in a wide range of PDFs that have been marked up or highlighted and extracts the useful information from them, all this is done securely in our own Google Cloud Project and the resulting Markdown files are sent downstream to our knowledge base where it can be accessed and read by Human and Agent alike.
If you are looking to unlock valuable business knowledge from PDFs and other documents and make them available to your AI, we can build you something similar that is cost efficient to run and secure. Get in touch and we'd be happy to help.