6 Commits

Author	SHA1	Message	Date
Viktor Lofgren	a86b596897	(crawler) Code quality	2024-04-24 14:44:39 +02:00
Viktor Lofgren	6dd87b0378	(crawler) Use the probe-result to reduce the likelihood of crawling both http and https ... This should drastically reduce the number of fetched documents on many domains	2024-04-24 14:44:39 +02:00
Viktor Lofgren	c9f029c214	(crawler) Strip W/-prefix from the etag when supplied as If-None-Match	2024-04-24 14:44:39 +02:00
Viktor Lofgren	6b88db10ad	(crawler) Ensure all appropriate headers are recorded on the request	2024-04-24 14:44:39 +02:00
Viktor Lofgren	8a891c2159	(crawler/converter) Remove legacy junk from parquet migration	2024-04-24 14:44:39 +02:00
Viktor Lofgren	1d34224416	(refac) Remove src/main from all source code paths. ... Look, this will make the git history look funny, but trimming unnecessary depth from the source tree is a very necessary sanity-preserving measure when dealing with a super-modularized codebase like this one. While it makes the project configuration a bit less conventional, it will save you several clicks every time you jump between modules. Which you'll do a lot, because it's *modul*ar. The src/main/java convention makes a lot of sense for a non-modular project though. This ain't that.	2024-02-23 16:13:40 +01:00