4 Commits

Author	SHA1	Message	Date
Viktor Lofgren	1673fc284c	(converter) Reduce lock contention in converter by separating the processing of full and simple-track domains	2025-01-26 13:21:46 +01:00
Viktor Lofgren	3714104976	Add loader for slop data in converter. ... Also alter CrawledDocument to not require String parsing of the underlying byte[] data. This should reduce the number of large memory allocations quite significantly, hopefully reducing the GC churn a bit.	2024-12-17 15:40:24 +01:00
Viktor Lofgren	52eb5bc84f	(live-crawler) Keep track of bad URLs ... To avoid hammering the same invalid URLs for up to two months, URLs that fail to fetch correctly are on a dice roll added to a bad URLs table, that prevents further attempts at fetching them.	2024-11-22 00:55:46 +01:00
Viktor Lofgren	89d8af640d	(live-crawl) Rename the live crawler code module to be more consistent with the other processes	2024-11-20 15:55:15 +01:00