Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 21:29:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,432 Commits 4 Branches 108 Tags 17 MiB
4b16022556
Commit Graph

4 Commits

Author SHA1 Message Date
Viktor Lofgren
ecb5eedeae (crawler, EXPERIMENT) Disable content type probing and use Accept header instead 
There's reason to think this may speed up crawling quite significantly, and the benefits of the probing aren't quite there.
 2024-09-30 14:53:01 +02:00
Viktor Lofgren
40512511af (crawler) Refactor boundary between CrawlerRetreiver and HttpFetcherImpl 
This code is still a bit too complex, but it's slowly getting better.
 2024-09-24 15:08:22 +02:00
Viktor Lofgren
162fc25ebc (minor) Fix accidental commit errors 2024-09-23 18:03:09 +02:00
Viktor Lofgren
e9854f194c (crawler) Refactor 
* Restructure the code to make a bit more sense
* Store full headers in crawl data
* Fix bug in retry-after header that assumed the timeout was in milliseconds, and then clamped it to a lower bound of 500ms, meaning this was almost always handled wrong
 2024-09-23 17:51:07 +02:00