MarginaliaSearch/code/processes/crawling-process/java/nu
Viktor Lofgren 2ea34767d8 (crawler) Use the response URL when resolving relative links
The crawler was incorrectly using the request URL as the base URL when resolving relative links.  This caused problems when encountering redirects.

 For example if we fetch /log, redirecting to  /log/ and find links to foo/, and bar/; these would resolve to /foo and /bar, and not /log/foo and /log/bar.
2025-01-31 12:40:13 +01:00
..
marginalia/crawl (crawler) Use the response URL when resolving relative links 2025-01-31 12:40:13 +01:00