You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

78 lines
2.7 KiB

4 years ago
# Ideas/Questions for future code
> Brainstorming here :P
### MongoDB
* Main database for crawled content
* YouTube is mostly non-relational
(except channels ↔ videos)
* Users can change videos (title, etc.):
Support for multiple crawls needed
* In one document as array?
Like `{videoid: xxx, crawls: []…`
* Pros: Easy history query
* Cons: (Title) indices might be harder to maintain
* Or as separate documents?
Like `{videoid: xxx, crawldate: …`
* Pros: Race conditions less likely
* Cons: Duplicates more likely?
* Avoiding duplicates?
* If the user hasn't changed video metadata,
crawling it again is a waste of disk space
* Rescan score: Should a video be rescanned?
* Viral videos should be crawled more often
* New videos shouldn't be instantly crawled again
* Very old videos are unlikely to change
* Maybe focus on views per week
* Machine learning?
* Hashing data from crawls to detect changes?
* Invalidates old data on API upgrade
* Could be used as an index tho
* Live data
* like views/comments/subscribers per day
* vs more persistent data: Title/Description/video Formats
* Are they worth crawling
* Additional data
* Like subtitles and annotations
* Need separate crawls
* Not as important as main data
### Types of bot
* __Discover bots__
* Find and push new video IDs to the queue
* Monitor channels for new content
* Discover new videos
* __Maintainer bots__
* Occasionally look at the database and
push backups/freezes to drive
* Decide which old video IDs to re-add to the queue
* __Worker bots__
* Get jobs from the Redis queue and crawl YT
* Remove processed entries from the queue
### Redis queue
* A redis queue lists video IDs that have been
discovered, but not crawled
* Discover bots bots push IDs if they find new ones
* Implement queue priority?
* Maintainer bots push IDs if they likely need rescans
* States of queued items
1. _Queued:_ Processing required
(no worker bot picked them up yet)
2. _Assigned:_ Worker claimed ID and processes it.
If the worker doesn't mark the ID as done in time
it gets tagged back as _Queued_ again
(should be hidden from other workers)
3. _Done:_ Worker submitted crawl to the database
(can be deleted from the queue)
* Single point of failure
* Potentially needs ton of RAM
* 800 M IDs at 100 bytes per entry = 80 GB
* Shuts down entire crawl system on failure
* Persistence: A crash can use all discovered IDs
* Alternative implementations
* SQLite in-memory?