|
|
|
|
|
# Ideas/Questions for future code
|
|
|
|
|
|
|
|
|
|
|
|
> Brainstorming here :P
|
|
|
|
|
|
|
|
|
|
|
|
### MongoDB
|
|
|
|
|
|
|
|
|
|
|
|
* Main database for crawled content
|
|
|
|
|
|
* YouTube is mostly non-relational
|
|
|
|
|
|
(except channels ↔ videos)
|
|
|
|
|
|
* Users can change videos (title, etc.):
|
|
|
|
|
|
Support for multiple crawls needed
|
|
|
|
|
|
* In one document as array?
|
|
|
|
|
|
Like `{videoid: xxx, crawls: []…`
|
|
|
|
|
|
* Pros: Easy history query
|
|
|
|
|
|
* Cons: (Title) indices might be harder to maintain
|
|
|
|
|
|
* Or as separate documents?
|
|
|
|
|
|
Like `{videoid: xxx, crawldate: …`
|
|
|
|
|
|
* Pros: Race conditions less likely
|
|
|
|
|
|
* Cons: Duplicates more likely?
|
|
|
|
|
|
* Avoiding duplicates?
|
|
|
|
|
|
* If the user hasn't changed video metadata,
|
|
|
|
|
|
crawling it again is a waste of disk space
|
|
|
|
|
|
* Rescan score: Should a video be rescanned?
|
|
|
|
|
|
* Viral videos should be crawled more often
|
|
|
|
|
|
* New videos shouldn't be instantly crawled again
|
|
|
|
|
|
* Very old videos are unlikely to change
|
|
|
|
|
|
* Maybe focus on views per week
|
|
|
|
|
|
* Machine learning?
|
|
|
|
|
|
* Hashing data from crawls to detect changes?
|
|
|
|
|
|
* Invalidates old data on API upgrade
|
|
|
|
|
|
* Could be used as an index tho
|
|
|
|
|
|
* Live data
|
|
|
|
|
|
* like views/comments/subscribers per day
|
|
|
|
|
|
* vs more persistent data: Title/Description/video Formats
|
|
|
|
|
|
* Are they worth crawling
|
|
|
|
|
|
* Additional data
|
|
|
|
|
|
* Like subtitles and annotations
|
|
|
|
|
|
* Need separate crawls
|
|
|
|
|
|
* Not as important as main data
|
|
|
|
|
|
|
|
|
|
|
|
### Types of bot
|
|
|
|
|
|
|
|
|
|
|
|
* __Discover bots__
|
|
|
|
|
|
* Find and push new video IDs to the queue
|
|
|
|
|
|
* Monitor channels for new content
|
|
|
|
|
|
* Discover new videos
|
|
|
|
|
|
* __Maintainer bots__
|
|
|
|
|
|
* Occasionally look at the database and
|
|
|
|
|
|
push backups/freezes to drive
|
|
|
|
|
|
* Decide which old video IDs to re-add to the queue
|
|
|
|
|
|
* __Worker bots__
|
|
|
|
|
|
* Get jobs from the Redis queue and crawl YT
|
|
|
|
|
|
* Remove processed entries from the queue
|
|
|
|
|
|
|
|
|
|
|
|
### Redis queue
|
|
|
|
|
|
|
|
|
|
|
|
* A redis queue lists video IDs that have been
|
|
|
|
|
|
discovered, but not crawled
|
|
|
|
|
|
* Discover bots bots push IDs if they find new ones
|
|
|
|
|
|
* Implement queue priority?
|
|
|
|
|
|
* Maintainer bots push IDs if they likely need rescans
|
|
|
|
|
|
* States of queued items
|
|
|
|
|
|
1. _Queued:_ Processing required
|
|
|
|
|
|
(no worker bot picked them up yet)
|
|
|
|
|
|
2. _Assigned:_ Worker claimed ID and processes it.
|
|
|
|
|
|
If the worker doesn't mark the ID as done in time
|
|
|
|
|
|
it gets tagged back as _Queued_ again
|
|
|
|
|
|
(should be hidden from other workers)
|
|
|
|
|
|
3. _Done:_ Worker submitted crawl to the database
|
|
|
|
|
|
(can be deleted from the queue)
|
|
|
|
|
|
* Single point of failure
|
|
|
|
|
|
* Potentially needs ton of RAM
|
|
|
|
|
|
* 800 M IDs at 100 bytes per entry = 80 GB
|
|
|
|
|
|
* Shuts down entire crawl system on failure
|
|
|
|
|
|
* Persistence: A crash can use all discovered IDs
|
|
|
|
|
|
* Alternative implementations
|
|
|
|
|
|
* SQLite in-memory?
|
