Multimodal Research Assistant
Researchers work with heterogeneous inputs — PDFs, diagrams, data tables, images, and text notes. A useful research assistant must handle all of these without requiring format conversion. This project builds a retrieval-augmented QA system that ingests mixed-modality inputs, extracts and indexes their content, and responds to research queries with citations back to the source material.
Built a multimodal AI research assistant with PDF, image, and text ingestion
- ›Retrieval-augmented QA system handling PDFs, images, and text without format conversion
- ›Claude vision integration for image understanding and semantic indexing
- ›Citation-grounded responses tied back to specific source documents and pages
A retrieval-augmented research assistant that ingests PDFs, images, and text, processes them with a multimodal pipeline (PyMuPDF + Claude vision), and answers research queries with cited, grounded responses.
Research workflows involve heterogeneous inputs: PDFs with embedded tables and figures, images, and plain text. Most RAG pipelines handle only one modality. A research assistant that forces format conversion before ingestion loses context (e.g., captions and visual relationships in PDF figures).
- Must ingest PDFs, images, and plain text without requiring pre-processing by the user
- Responses must cite specific source documents and page numbers
- Must handle long documents that exceed LLM context windows
Ingestion: PDFs parsed with PyMuPDF to extract text and embedded images; images described via Claude vision API; all chunks embedded and stored in Pinecone with source metadata. At query time: retrieve top-k chunks, construct context with citations, generate response via Claude with explicit source attribution.