Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(crawler) Set more lenient parameters for recrawl

Browse Source
This commit is contained in:
Viktor Lofgren 2023-10-30 11:35:30 +01:00
parent fd5a7eac87
commit 8f74dbdbb4
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
code/processes/crawling-process/src/main/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
View File

@ -227,9 +227,9 @@ public class CrawlerRetreiver {
continue; continue;
if (recrawled > 10 if (recrawled > 5
&& retained > 0.9 * recrawled && retained > 0.9 * recrawled
&& Math.random() < 0.75) && Math.random() < 0.9)
{ {
// Since it looks like most of these documents haven't changed, // Since it looks like most of these documents haven't changed,
// we'll load the documents directly; but we do this in a random // we'll load the documents directly; but we do this in a random