Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
567e4e1237
MarginaliaSearch/code/processes/crawling-process/java/nu/marginalia
History
Viktor Lofgren 567e4e1237 (crawler) Fast detection and bail-out for crawler traps 
Improve logging and exclude robots.txt from this logic.
2025-01-18 15:28:54 +01:00
..
crawl (crawler) Fast detection and bail-out for crawler traps 2025-01-18 15:28:54 +01:00