Viktor Lofgren f4d79c203d (crawler) Adjust revisit logic ... The revisit logic wasn't sufficiently dampening the recrawl rate for websites that largely have not changed. Modified it to be more reactive to the degree to which the content has changed, while applying upper and lower limits depending on the size of the crawl set.		2024-07-16 15:12:38 +02:00
..
fetcher	(crawler) Update WARC builder to use SHA-256 for digests	2024-06-12 09:14:12 +02:00
revisit	(crawler) Adjust revisit logic	2024-07-16 15:12:38 +02:00
sitemap	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
Cookies.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlDataReference.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlDelayTimer.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawledDocumentFactory.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlerRetreiver.java	(crawler) Introduce absolute upper limit to crawl depth growth	2024-07-16 14:40:45 +02:00
CrawlerWarcResynchronizer.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
DomainCrawlFrontier.java	(crawler) Introduce absolute upper limit to crawl depth growth	2024-07-16 14:40:45 +02:00
DomainProber.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
LinkFilterSelector.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
RateLimitException.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00