Viktor Lofgren 0ca43f0c9c (live-crawler) Improve live crawler short-circuit logic ... We should not wait until we've fetched robots.txt to decide whether we have any data to fetch! This makes the live crawler very slow and leads to unnecessary requests.		2024-12-27 20:54:42 +01:00
..
nu/marginalia/livecrawler	(live-crawler) Improve live crawler short-circuit logic	2024-12-27 20:54:42 +01:00