Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 13:09:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(live-crawl) Flag URLs that don't pass robots.txt as bad so we don't keep fetching robots.txt every day for an empty link list

Browse Source
This commit is contained in:
Viktor Lofgren 2024-11-23 17:07:16 +01:00
parent 923ebbac81
commit 88caca60f9
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
code/processes/live-crawling-process/java/nu/marginalia/livecrawler/SimpleLinkScraper.java
View File

@ -92,6 +92,7 @@ public class SimpleLinkScraper implements AutoCloseable {
EdgeUrl parsedUrl = optParsedUrl.get();
if (!rules.isAllowed(url)) {
maybeFlagAsBad(parsedUrl);
continue;
}