Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3ddcebaa36
MarginaliaSearch/code/processes/live-crawling-process/test/nu
History
Viktor Lofgren 52eb5bc84f (live-crawler) Keep track of bad URLs 
To avoid hammering the same invalid URLs for up to two months, URLs that fail to fetch correctly are on a dice roll added to a bad URLs table, that prevents further attempts at fetching them.
2024-11-22 00:55:46 +01:00
..
marginalia/livecrawler (live-crawler) Keep track of bad URLs 2024-11-22 00:55:46 +01:00