Viktor Lofgren 481f999b70 (crawler) Make DomainCrawlFrontier a bit less aggressive with throwing away excess links when it's approaching full. ... Also be a bit smarter about pre-allocating queues and sets based on depth rather than the number of provided URLs, which was always zero outside of tests.		2024-10-15 14:22:40 +02:00
..
marginalia/crawl	(crawler) Make DomainCrawlFrontier a bit less aggressive with throwing away excess links when it's approaching full.	2024-10-15 14:22:40 +02:00