Commit Graph

8 Commits

Author SHA1 Message Date
Viktor Lofgren
e1c9313396 (crawler) Emulate if-modified-since for domains that don't support the header
This will help reduce the strain on some server software, in particular Discourse.
2024-04-24 14:44:39 +02:00
Viktor Lofgren
f430a084e8 (crawler) Remove accidental log spam 2024-04-24 14:44:39 +02:00
Viktor Lofgren
a86b596897 (crawler) Code quality 2024-04-24 14:44:39 +02:00
Viktor Lofgren
6dd87b0378 (crawler) Use the probe-result to reduce the likelihood of crawling both http and https
This should drastically reduce the number of fetched documents on many domains
2024-04-24 14:44:39 +02:00
Viktor Lofgren
c9f029c214 (crawler) Strip W/-prefix from the etag when supplied as If-None-Match 2024-04-24 14:44:39 +02:00
Viktor Lofgren
6b88db10ad (crawler) Ensure all appropriate headers are recorded on the request 2024-04-24 14:44:39 +02:00
Viktor Lofgren
8a891c2159 (crawler/converter) Remove legacy junk from parquet migration 2024-04-24 14:44:39 +02:00
Viktor Lofgren
1d34224416 (refac) Remove src/main from all source code paths.
Look, this will make the git history look funny, but trimming unnecessary depth from the source tree is a very necessary sanity-preserving measure when dealing with a super-modularized codebase like this one.

While it makes the project configuration a bit less conventional, it will save you several clicks every time you jump between modules.  Which you'll do a lot, because it's *modul*ar.  The src/main/java convention makes a lot of sense for a non-modular project though.  This ain't that.
2024-02-23 16:13:40 +01:00