Step 4: Configure crawl settings:
- Max Depth: Set how many levels deep to crawl (e.g., 2-3 for following related videos)
- Parallel Tabs: Control concurrent page processing (2-4 recommended)
- Nav-Link Filter: Select "Same Domain" to stay within YouTube, or "Custom Rules" for specific patterns
Step 5: Set advanced limits (optional):
- Page Load Delay - set to 2-3 sec as Youtube loads links asynchronously we need to wait a bit untill all links available.
- Max Pages: Limit total pages to crawl (e.g., 100-500)
- Links Cap: Maximum links to discover (e.g., 10,000)
- Time Limit: Auto-stop after specified minutes
Step 6: Configure URL filtering for YouTube:
Open 'Nav-Link Filter' > 'Custome Rules'
- Include Patterns: Use patterns like `*/watch*` to focus on video links
- Exclude Patterns: Filter out unwanted URLs like `*/shorts/*` or `*/channel/*` if you only want regular videos
Step 7: Start the crawl and monitor progress in real-time
Step 8: Export results to CSV, JSON, or plain text when crawl completes
Perfect Use Cases for YouTube Link Crawling:
- Follow Related Videos: Start with one video and crawl through "watch next" recommendations
- Channel Deep Dive: Crawl a channel and follow all related video links recursively
- Playlist Networks: Extract links from playlists that link to other playlists
- Search Result Crawling: Start with search results and follow video links found
- Community Posts: Crawl YouTube community tab posts and extract all video links
- Competitor Analysis: Recursively extract all video URLs from competitor channels
⭐ Pro Tips:
- Use depth 1-2 for focused crawls, depth 3+ for comprehensive discovery
- Set "Same Domain" filter to ensure crawler stays within YouTube
- Use include pattern `youtube.com/watch*` to focus only on video pages
- Exclude `*/shorts/*` if you don't want YouTube Shorts
- Set reasonable limits to prevent runaway crawls on large channels