Home/Projects/Multimodal Research Assistant
Prototype
AIKnowledge SystemsResearch AutomationNLP

Multimodal Research Assistant

Researchers work with heterogeneous inputs — PDFs, diagrams, data tables, images, and text notes. A useful research assistant must handle all of these without requiring format conversion. This project builds a retrieval-augmented QA system that ingests mixed-modality inputs, extracts and indexes their content, and responds to research queries with citations back to the source material.

recruiter perspective

Built a multimodal AI research assistant with PDF, image, and text ingestion

  • ›Retrieval-augmented QA system handling PDFs, images, and text without format conversion
  • ›Claude vision integration for image understanding and semantic indexing
  • ›Citation-grounded responses tied back to specific source documents and pages
View repo
Hiring Summary

A retrieval-augmented research assistant that ingests PDFs, images, and text, processes them with a multimodal pipeline (PyMuPDF + Claude vision), and answers research queries with cited, grounded responses.

Research workflows involve heterogeneous inputs: PDFs with embedded tables and figures, images, and plain text. Most RAG pipelines handle only one modality. A research assistant that forces format conversion before ingestion loses context (e.g., captions and visual relationships in PDF figures).

Challenges Addressed
  • Must ingest PDFs, images, and plain text without requiring pre-processing by the user
  • Responses must cite specific source documents and page numbers
  • Must handle long documents that exceed LLM context windows
◆Multimodal Research AssistantAI/visual-placeholder

Ingestion: PDFs parsed with PyMuPDF to extract text and embedded images; images described via Claude vision API; all chunks embedded and stored in Pinecone with source metadata. At query time: retrieve top-k chunks, construct context with citations, generate response via Claude with explicit source attribution.

Stack
PythonClaude APIOpenAI EmbeddingsPineconeFastAPIPyMuPDF
Decision
Claude vision for image description over OCR-only approaches
OCR extracts text from images but loses visual structure (charts, diagrams, spatial relationships). Claude vision produces natural language descriptions that capture what the image shows, making it retrievable via semantic search.
Decision
Citation-first response format
Research responses without citations are unverifiable. Requiring source attribution in the response format forces the system to retrieve grounded information rather than generate from model knowledge.
Tradeoff
Ingestion cost vs. modality coverage
Chose: Vision API for images with content-hash caching
Running Claude vision on every image is expensive. Caching processed image descriptions by content hash ensures each image is processed once, amortizing cost over multiple queries.
01Table extraction and structured querying for PDF tables
02Multi-document comparison mode for identifying contradictions across sources
03Citation graph visualization showing how claims connect across documents