Solved problems get re-solved.
Intakes, elevators, climbers, and drivetrains already exist in countless proven forms, yet every January teams reinvent them.
Solo project / 2025
FRC RAG
A multimodal search engine that lets FIRST Robotics teams learn from a decade of real, competition-proven robots instead of starting every season from scratch.
The Problem
Every January, teams rebuild what already exists.
A search problem in disguise.
FRC pushes every team to build something new each season, but most of the hard mechanical problems were solved years ago by someone else. The knowledge exists. It just isn't searchable.
The result is a yearly tax: weeks spent reinventing intakes and climbers that a past robot already proved out, simply because nobody could find the right example in time.
The best examples are scattered.
Winning designs are buried across old Chief Delphi threads, match videos, and dead CAD links that are hard to search by mechanism.
Generic chatbots guess.
Ask a frontier model for an intake and it returns a plausible idea with no evidence that any real robot ever ran it.
Robotics is visual.
A CAD screenshot or a photo of a mechanism says more than a paragraph, but text-only search throws that signal away.
The Approach
Proven designs, not plausible ones.
Why retrieval, not just a bigger model.
A frontier model with web search can describe an intake, but it optimizes for what sounds right, not for what survived a regional. FRC RAG answers only from binders of robots that actually competed, and every claim carries a team and year you can check.
It also leans into the visual side that chat models tend to skip. Because the index understands CAD screenshots and mechanism photos, answers arrive with the real images attached, each credited to the team that built it.
Ingestion
Turning a decade of binders into a searchable corpus.
A seven-stage pipeline.
Each PDF runs offline on a GPU box through parsing, captioning, and embedding before anything reaches the live server. Heavy work happens once; serving stays cheap.
- 01
Parse + OCR
PyMuPDF, pdfplumber, and Tesseract pull layout-aware text, tables, and figures out of technical binders, with results cached per document.
PyMuPDFTesseract - 02
Extract + dedupe images
Figures are extracted and perceptual-hashed so duplicate CAD renders and reused photos collapse into one indexed asset.
imagehash - 03
Chunk
Text is split into roughly 500-token chunks with overlap, keeping section context attached to every fragment.
500 tok~150 overlap - 04
Structured visual facts
A vision model reads each figure and returns schema-constrained JSON: mechanism type, actuation, latching, materials.
Qwen2-VL-7B - 05
Embed
Text goes through BGE-large; images go through SigLIP. Captions and facts are folded back into the text vectors.
BGE-largeSigLIP - 06
Index
Vectors land in Qdrant across separate text, image, and page collections, keyed by team, year, and subsystem.
Qdrant - 07
Page-level vectors
Full pages are rendered at 150 DPI and encoded with ColQwen2 into bags of patch vectors for late-interaction search.
ColQwen2
Fig. 1 Offline ingestion pipeline, PDF to indexed vectors.
Retrieval
One question, five ways to answer it.
Intent decides the mix.
A rule-based classifier reads each query and a planner reweights five retrieval channels before fusing them with reciprocal rank fusion. A visual question leans on images and page search; a spec question leans on exact terms.
Pick an example query below to see how the routing shifts the weight behind each channel.
Query
Dense
0%
BGE-large-en-v1.5
Semantic meaning of the question.
Sparse
0%
BM25Okapi
Exact terms like NEO, Kraken, gear ratios.
Image
0%
SigLIP SO400M
Photos and CAD that look like the query.
Page
0%
ColQwen2 MaxSim
Layout-rich pages: wiring, labeled CAD.
Web
0%
DuckDuckGo fallback
Current-season gaps the corpus can't cover.
RRF fusion
then mechanism rerank, confidence filter
Grounded context + cited images
Fig. 2 Intent-routed hybrid retrieval. Weights are illustrative of the planner's bias per query type.
Visual Facts
Teaching the index what a mechanism is.
Beyond generic captions.
Most FRC photos look alike: aluminum plates, belts, and NEOs. A free-text caption barely separates them. Instead, a vision model returns schema-constrained JSON describing the mechanism, which gets embedded alongside the text so a search for "passive climber hook" actually lands.
{
"mechanism_type": "buddy_climber",
"actuation": [
"winch",
"rope"
],
"latching": [
"passive_hook"
],
"degrees_of_freedom": 1,
"materials": [
"aluminum"
],
"notable_features": [
"over_center_latch"
],
"confidence": 0.86
} 25
mechanism types
intake, swerve_module, buddy_climber
17
actuation kinds
neo, falcon, kraken, pneumatic
10
latching kinds
passive_hook, ratchet, servo
11
material kinds
aluminum, polycarbonate, steel
Fig. 3 A controlled vocabulary turns pixels into queryable structure.
Page Search
Searching pages the way an engineer scans them.
Late interaction with ColPali.
Squeezing a dense wiring diagram into a single vector loses the detail that matters. ColQwen2 encodes a page as a bag of patch vectors, and a query is scored by the best match against any patch. Fine-grained layout survives.
Single vector
One embedding per page. Local detail averages away.
ColPali MaxSim
Many patch vectors. The best patch match wins.
Fig. 4 Late-interaction retrieval keeps layout-level signal for diagrams and labeled CAD.
Performance
Built to hold up on a real server.
From a script to a service.
Moving embedding into a dedicated inference server, swapping in standalone Qdrant, and pushing user ingestion onto a Celery queue turned a single-user prototype into something that serves a room full of teams on the clock.
Encode 100 chunks
94.5% faster
Upload response time
non-blocking queue
Embedding throughput
~18x
Fig. 5 Before and after the move to a serving-grade architecture.
The Corpus
What's inside the index.
0
Technical binders indexed
0
Elite teams represented
0
Searchable text chunks
0
Indexed figures + CAD
0
Seasons of designs (2010-2025)
Retrieval backend
FastAPIQdrantText Embeddings InferenceCelery + RedisBGE-largeSigLIPColQwen2Qwen2-VL
Product frontend
Next.js 16React 19ConvexVercel AI SDKGroqOpenRouterTailwind CSS
Ingestion
PyMuPDFpdfplumberTesseractCadQueryOpenCascadeimagehash
The Product
A copilot teams can use in build season.
Retrieval is only half of it.
The frontend keeps the backend honest: it owns generation, citations, and image rendering while the retrieval service owns the truth. Teams can bring their own model and key, drop in private binders that fuse with the global corpus, and get answers with inline figures credited by team number.
Fig. 6 A grounded answer with structured specs, streamed from a chosen model.