Viktor Lofgren ec600b967d (crawler) Adjust domain locking ... Turns out throttling to only 1 lock per domain means the crawler chokes hard on large hosting websites such as wordpress. Giving these a slightly larger allowance.		2024-07-27 11:54:46 +02:00
..
fetcher	(crawler) Update WARC builder to use SHA-256 for digests	2024-06-12 09:14:12 +02:00
revisit	(crawler) Adjust revisit logic	2024-07-16 15:12:38 +02:00
sitemap	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
Cookies.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlDataReference.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlDelayTimer.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawledDocumentFactory.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
CrawlerRetreiver.java	(crawler) Adjust domain locking	2024-07-27 11:54:46 +02:00
CrawlerWarcResynchronizer.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
DomainCrawlFrontier.java	(crawler) Introduce absolute upper limit to crawl depth growth	2024-07-16 14:40:45 +02:00
DomainLocks.java	(crawler) Adjust domain locking	2024-07-27 11:54:46 +02:00
DomainProber.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
LinkFilterSelector.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00
RateLimitException.java	(refac) Remove src/main from all source code paths.	2024-02-23 16:13:40 +01:00