Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 13:09:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(crawler) Set more lenient parameters for recrawl

Browse Source
This commit is contained in:
Viktor Lofgren 2023-10-30 11:35:30 +01:00
parent fd5a7eac87
commit 8f74dbdbb4
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
code/processes/crawling-process/src/main/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
View File

@ -227,9 +227,9 @@ public class CrawlerRetreiver {
continue;
if (recrawled > 10
if (recrawled > 5
&& retained > 0.9 * recrawled
&& Math.random() < 0.75)
&& Math.random() < 0.9)
{
// Since it looks like most of these documents haven't changed,
// we'll load the documents directly; but we do this in a random