Intro
Derived from Travis Fischer's All-In Podcast Semantic Search, adapted to search Videos of Launch School Capstone Project Presentations.
Complementary to this is a Streamlit app David Dickinson put together to view summaries of the Capstone Projects made from the video transcripts with GPT-3.5 and GPT-4. Check it out here.
This project uses the latest models from OpenAI to build a semantic search index across every Capstone presentation from 2020 to 2023. It allows you to find the exact moments in each video where a topic was discussed with Google-level accuracy and find the exact clips you're interested in.
You can use it to power advanced search across any YouTube channel or playlist.
Example Queries
- api load testing
- graphql observability
- elastic container service
- automated canary deployments
- kubernetes vs docker swarm
- websockets
- implementation challenges future work design decisions
How It Works
This project is open source! 😄
Under the hood, it uses:
- OpenAI - We're using the brand new text-embedding-ada-002 embedding model, which captures deeper information about text in a latent space with 1536 dimensions
- This allows us to go beyond keyword search and search by higher-level topics.
- Pinecone - Hosted vector search which enables us to efficiently perform k-NN searches across these embeddings
- Vercel - Hosting and API functions
- Next.js - React web framework
We use Node.js and the YouTube API v3 to fetch the videos of our target playlist. In this case, we're focused on the Launch School Capstone Presentations Playlist, which contains 58 videos at the time of writing.
npx tsx src/bin/resolve-yt-playlist.ts
We download the English transcripts for each episode using a hacky HTML scraping solution, since the YouTube API doesn't allow non-OAuth access to captions. Note that a few episodes don't have automated English transcriptions available, so we're just skipping them at the moment. A better solution would be to use Whisper to transcribe each episode's audio.
Once we have all of the transcripts and metadata downloaded locally, we pre-process each video's transcripts, breaking them up into reasonably sized chunks of ~100 tokens and fetch it's text-embedding-ada-002 embedding from OpenAI. This results in ~200 embeddings per episode.
All of these embeddings are then upserted into a Pinecone index with a dimensionality of 1536. There are ~9,575 embeddings in total across ~58 Capstone Presentations.
npx tsx src/bin/process-yt-playlist.ts
Once our Pinecone search index is set up, we can start querying it either via the webapp or via the example CLI:
npx tsx src/bin/query.ts
We also support generating timestamp-based thumbnails of every YouTube video in the playlist. Thumbnails are generated using headless Puppeteer and are uploaded to Google Cloud Storage. We also post-process each thumbnail with lqip-modern to generate nice preview placeholder images.
If you want to generate thumbnails (optional), run:
npx tsx src/bin/generate-thumbnails.ts
Note that thumbnail generation takes ~2 hours and requires a pretty stable internet connection.
The frontend is a Next.js webapp deployed to Vercel that uses our Pinecone index as a primary data store.
Feedback
Have an idea on how this webapp could be improved? Find a particularly fun search query?
Feel free to send me feedback, either on [GitHub or Twitter. 💯
License
This project is open source. MIT © Travis Fischer
This project is not affiliated with Launch School. It just pulls data from their videos and transcripts.