About LibTrails

Parse & Chunk

EPUBs are parsed with selectolax (HTML block tags converted to paragraph breaks) and recursively split into ~500-word chunks — first at paragraph boundaries, then sentences, then words as a last resort. The 100-book demo library produces 31,849 chunks.

Two-Pass LLM Extraction

Pass 1: A larger model (gemma3:27b) reads the first few chunks plus Calibre metadata (tags, description, series) to extract 5–10 book-level themes in a single call.

Pass 2: A smaller model (gemma3:4b or 12b) extracts chunk-level topics, contextualized with the book themes from Pass 1. This produces domain-specific noun phrases instead of generic single words — yielding 121,118 deduplicated topics across the demo library.

Embed & Deduplicate

All topics get 384-dimensional embeddings via BGE-small-en-v1.5. A two-tier deduplication then merges near-duplicates: cosine similarity >0.95 merges unconditionally, while 0.85–0.95 only merges if both topics share at least one book — preventing cross-domain conflation (e.g., "energy manipulation" in fantasy vs. psychology).

Co-occurrence & KNN Graph

Topic pairs that appear in the same chunk get Pointwise Mutual Information (PMI) scores. A KNN graph is built from two edge types: co-occurrence edges weighted by PMI with a book-count boost (PMI × (1 + log(1 + book_count))), and embedding-similarity edges from each topic's 10 nearest neighbors (cosine ≥ 0.65). Only genuinely surprising co-occurrences make the cut (PMI ≥ 1.0).

First Leiden Pass: Topic Clusters

High-degree "hub" topics (95th percentile) are removed before clustering — these generic terms would create artificially large, incoherent groups. The Leiden algorithm then runs with the CPM (Constant Potts Model) partition type at resolution 0.001, producing 2,468 fine-grained topic clusters. Each cluster groups topics that are tightly related — for example, "categorical imperative," "moral duty," and "Kantian ethics" all land in one cluster. The cluster's label is drawn from its most representative topic, so you see names like "Kantian Ethics & Moral Philosophy" rather than raw topic strings.

These are the labeled chips you see throughout the app — on topic detail pages, in universe popups, and in search results. When a topic page shows "27 clusters," that means 27 distinct sub-themes make up that broader topic community. Clusters also power the search engine: topic and chunk vectors are indexed in sqlite-vec for cosine similarity lookups, and FTS5 indexes cluster labels, topic labels, and chunk text for keyword matching.

Second Leiden Pass: Topic Communities

A second Leiden pass runs on the same topic graph with a different resolution, grouping clusters into 202 broader communities. These are the browsable entries on the Topics page and the spheres in the Universe view. Each community is a coherent theme like "Victorian Social Norms," "Nautical Adventure," or "Ancient Greek Philosophy" — large enough to span multiple books, small enough to be meaningful.

Themes via K-Means

K-means groups the cluster centroids into 26 high-level themes based on embedding similarity. These appear on the Themes page with names like "Adventure & Human Folly," "Literary Classics & Philosophy," or "Nature & Rural Life" — LLM-generated and human-refined.

3D Projection & Semantic Colors

Community centroids are projected into 3D with UMAP (cosine metric, n_neighbors=15). Theme embeddings are mapped onto a single PCA axis, then each theme's position becomes a hue value — so semantically similar themes share similar colors in the Universe. The result is rendered with React Three Fiber using instanced meshes for performance.

Hybrid Book Search

When you search on the Books page, your query is evaluated against six independent retrieval signals, each producing its own ranked list:

• 1. FTS5 book metadata — keyword search over title, author, and description
• 2. FTS5 topic labels — keyword search across 121K topic labels, mapped back to books
• 3. FTS5 chunk text — keyword search across 31,849 chunk passages, mapped back to books
• 4. Semantic book theme vectors — cosine similarity against book-level theme embeddings
• 5. Semantic book vectors — cosine similarity against whole-book embeddings (title + themes)
• 6. Semantic chunk vectors — cosine similarity against chunk embeddings, best per book

Each result shows a colored badge indicating which signal contributed most: keyword, topic, content, theme, book, or chunk.

Reciprocal Rank Fusion

Each signal produces its own ranked list. RRF combines them by scoring each result as 1/(k+rank) across all lists, where k=60. This means a book that ranks #1 in keyword search and #5 in semantic search scores higher than one that ranks #2 in both — rewarding results that excel in at least one signal without requiring score normalization across different retrieval methods.

Scope-Aware Search

Search adapts to each page. On Books, all 6 signals rank books directly. On Topics, 4 signals surface the most relevant topic communities — topic FTS, cluster label matching, chunk FTS, and chunk semantic. On Themes, topic scores roll up to highlight matching themes, with a boost for direct label matches.

Universe Search

The Universe search bar runs the same cluster search, then aggregates results by community. Matching communities glow amber in the 3D view — the brighter the glow, the stronger the match — with a ranked sidebar list showing the top hits. This lets you visually see where a concept lives in the semantic landscape and which neighboring communities are related.

Navigation

Drag to rotate, scroll to zoom, and right-drag to pan. Each sphere is a topic community — larger spheres span more books. Colors encode themes: semantically similar themes share similar hues, so you can see natural neighborhoods form in 3D space.

Interacting

Hover over a sphere to see its label and the books it spans. Click to open a detail panel with the community's cluster labels and all contributing books. Use the search bar to light up matching communities — try searching for a concept like "justice" or "natural selection" to see which corners of the library discuss it.

What to Look For

Tight clusters of similarly-colored spheres reveal thematic neighborhoods — philosophy spheres near political theory, gothic fiction near psychological horror. But the interesting discoveries are the unexpected bridges: a nature-writing community sitting near a philosophy community because Thoreau and Emerson link them both.