Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 21:29:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,386 Commits 4 Branches 108 Tags 17 MiB
40512511af
Commit Graph

3 Commits

Author SHA1 Message Date
Viktor Lofgren
40512511af (crawler) Refactor boundary between CrawlerRetreiver and HttpFetcherImpl 
This code is still a bit too complex, but it's slowly getting better.
 2024-09-24 15:08:22 +02:00
Viktor Lofgren
162fc25ebc (minor) Fix accidental commit errors 2024-09-23 18:03:09 +02:00
Viktor Lofgren
e9854f194c (crawler) Refactor 
* Restructure the code to make a bit more sense
* Store full headers in crawl data
* Fix bug in retry-after header that assumed the timeout was in milliseconds, and then clamped it to a lower bound of 500ms, meaning this was almost always handled wrong
 2024-09-23 17:51:07 +02:00