Viktor Lofgren 2ea34767d8 (crawler) Use the response URL when resolving relative links ... The crawler was incorrectly using the request URL as the base URL when resolving relative links. This caused problems when encountering redirects. For example if we fetch /log, redirecting to /log/ and find links to foo/, and bar/; these would resolve to /foo and /bar, and not /log/foo and /log/bar.		2025-01-31 12:40:13 +01:00
..
fetcher	(crawler) Change the header 'User-agent' to 'User-Agent'	2025-01-28 15:34:16 +01:00
logic	(crawler) Clean up the crawler code a bit, removing vestigial abstractions and historical debris	2024-10-15 17:27:59 +02:00
retreival	(crawler) Use the response URL when resolving relative links	2025-01-31 12:40:13 +01:00
warc	(refac) Move export tasks to a process and clean up process initialization for all ProcessMainClass descendents	2024-11-21 16:00:09 +01:00
CrawlerMain.java	(crawler) Reduce long retention of CrawlDataReference objects and their associated SerializableCrawlDataStreams	2025-01-26 15:40:17 +01:00
CrawlerModule.java	(chore) Remove lombok	2024-11-11 21:14:38 +01:00
DomainStateDb.java	(crawler) Add default CT when it's missing for icons	2025-01-22 13:55:47 +01:00