Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-25 05:38:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
e44ecd6d69
MarginaliaSearch/code/processes/crawling-process/model/java
History
Viktor Lofgren 0ca43f0c9c (live-crawler) Improve live crawler short-circuit logic 
We should not wait until we've fetched robots.txt to decide whether we have any data to fetch!  This makes the live crawler very slow and leads to unnecessary requests.
2024-12-27 20:54:42 +01:00
..
nu/marginalia (live-crawler) Improve live crawler short-circuit logic 2024-12-27 20:54:42 +01:00
org/netpreserve/jwarc (wip) Extract and encode spans data 2024-07-27 11:44:13 +02:00