Viktor Lofgren bae44497fe (crawler) Add a new system property crawler.maxFetchSize ... This gives the same upper limit to the live crawler and the big boy crawler, though the live crawler will reject items too large, and the big crawler will truncate at that point.		2024-12-30 15:10:11 +01:00
..
LiveCrawlDataSet.java	(live-crawler) Keep track of bad URLs	2024-11-22 00:55:46 +01:00
LiveCrawlerMain.java	(live-crawler) Add Accept-Encoding: gzip to outbound requests	2024-12-27 03:59:34 +01:00
LiveCrawlerModule.java	(refac) Move export tasks to a process and clean up process initialization for all ProcessMainClass descendents	2024-11-21 16:00:09 +01:00
SimpleLinkScraper.java	(crawler) Add a new system property crawler.maxFetchSize	2024-12-30 15:10:11 +01:00