Video is the world's largest unstructured knowledge base, but it's completely unsearchable. VidCrawl converts YouTube channels into queryable datasets through a five-stage pipeline: transcript extraction (with Whisper fallback for uncaptioned videos), semantic chunking with boundary detection, OpenAI embedding, Pinecone vector indexing, and a FastAPI retrieval layer. The core insight: naive sentence splitting loses topic coherence in long-form content — semantic chunking with cosine-similarity boundary detection fixes this.
›Processed 200k+ semantic chunks across 50+ YouTube channels (prototype)
›Sub-3-minute end-to-end processing per hour of video content
See the pipeline
Hiring Summary
Hiring Summary
VidCrawl is a data engineering project I built to make YouTube video content semantically searchable. It's a five-stage pipeline: transcript extraction (with Whisper fallback for uncaptioned videos), semantic chunking with cosine-similarity boundary detection, OpenAI embedding, Pinecone vector indexing, and a FastAPI retrieval layer. At prototype scale: 200k+ chunks, 50+ channels, sub-3-minute processing per hour of video. The interesting engineering work was in the chunking strategy — semantic boundaries instead of fixed-size windows meaningfully improved retrieval quality.
Evidence & Proof
Each claim is paired with evidence that a recruiter can request or verify.
“The full pipeline — transcript fetch, semantic chunking, embedding, and vector upsert — processes a 60-minute video in under 3 minutes end-to-end.”
benchmarkPipeline Latency Benchmark● local / internal✓
engineerairecruiter
“Semantic chunking with cosine-similarity boundary detection outperforms fixed-size sentence splitting on retrieval accuracy for long-form video content.”
writeupChunking Strategy Comparison● coming soon✓
aiengineer
“Hybrid transcription — YouTube captions first, Whisper fallback — achieves near-100% coverage while cutting Whisper compute cost by ~70%.”
benchmarkPipeline Latency Benchmark● local / internal✓
engineerai
Metrics
Metrics marked with context indicate the measurement scope (local benchmark, prototype, qualitative estimate, etc.).
<3
min / hour of video
Processing Speed
Local benchmark — prototype infrastructure, not production-optimized
Linked evidence ↗
50+
YouTube channels
Channels Crawled
Technical, finance, and science content — prototype scope
200k+
semantic chunks
Chunks Indexed
Prototype run — not a sustained production index
~70%
compute reduction
Whisper Coverage Savings
Qualitative estimate — Whisper runs only on uncaptioned videos (~30% of corpus)
Context & Problem
High-signal video content from technical creators is locked in an unstructured, unsearchable format. A researcher wanting to find every time a specific model architecture was discussed across 50 channels has no practical tool to do this. Existing solutions either don't extract transcripts, don't make them queryable, or lose topic coherence through naive chunking.
Challenges Addressed
Must handle videos without YouTube captions using Whisper as a fallback (roughly 30% of technical content)
Must scale to full channel crawls (100+ videos) without API rate limit failures
Chunking must preserve semantic coherence across topic boundaries in long-form content
Storage costs must remain reasonable at prototype scale — under $0.01 per video target
Architecture
A Python async orchestrator fetches video metadata via YouTube Data API v3, downloads audio for Whisper transcription when captions are missing, runs semantic chunking with configurable overlap and cosine-similarity boundary detection (via sentence-transformers), generates embeddings via OpenAI batch API (100 chunks per request), and upserts to Pinecone with metadata. PostgreSQL stores chunk metadata for filtered retrieval. A FastAPI layer exposes `/search` (semantic), `/video/{id}/chunks`, and `/channel/{id}/videos` endpoints. Rate limiting uses a token bucket per API key.
Stack
PythonYouTube Data APIWhisperOpenAI EmbeddingsPineconeFastAPIPostgreSQL
Technical Decisions
Decision
Whisper as fallback transcription, not primary
Roughly 30% of high-value technical videos lack auto-captions. Running Whisper on everything is expensive and unnecessary — YouTube captions are faster and free. The hybrid approach maximizes coverage while minimizing compute.
Decision
Semantic chunking with sentence-transformer boundary detection
Fixed-size chunking breaks topic coherence: a 500-token window often splits mid-argument. Cosine similarity between consecutive sentence embeddings identifies topic shift boundaries, producing semantically coherent chunks that improve retrieval precision.
Decision
Pinecone namespaced by channel
Namespace partitioning allows per-channel filtering without index-level segregation. This supports channel-scoped queries (common use case) without needing separate Pinecone indexes per channel.
Running Whisper on every video would cost ~70% more in compute. The hybrid approach achieves near-100% coverage at a fraction of the cost, with no quality loss (Whisper transcription quality matches or exceeds YouTube auto-captions on technical content).
Tradeoff
Chunking coherence vs. retrieval granularity
Chose: Semantic chunks with 20% overlap
Very large chunks improve coherence but hurt retrieval granularity (the query matches the chunk, but the relevant content is buried). Semantic boundary detection + 20% overlap balances both.
Future Work
01Speaker diarization to attribute claims to specific speakers across panel discussions
02Temporal indexing: queries return timestamps, not just chunk text
03Multi-modal indexing of slide frames extracted from screen-recorded talks
04Incremental channel sync: re-process only new videos, not full channel re-crawls