Viktor Lofgren 552b246099 (live-crawl) Improve error handling for errors during robots.txt-retrieval ... Reduce log-spam and don't treat errors other than 404 as "all is permitted".		2024-11-22 14:15:32 +01:00
..
java/nu/marginalia/livecrawler	(live-crawl) Improve error handling for errors during robots.txt-retrieval	2024-11-22 14:15:32 +01:00
test/nu/marginalia/livecrawler	(live-crawler) Keep track of bad URLs	2024-11-22 00:55:46 +01:00
build.gradle	(refac) Move export tasks to a process and clean up process initialization for all ProcessMainClass descendents	2024-11-21 16:00:09 +01:00