Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
481f999b70
MarginaliaSearch/code/processes/crawling-process/java/nu/marginalia/crawl/fetcher
History
Viktor Lofgren ecb5eedeae (crawler, EXPERIMENT) Disable content type probing and use Accept header instead 
There's reason to think this may speed up crawling quite significantly, and the benefits of the probing aren't quite there.
2024-09-30 14:53:01 +02:00
..
socket (crawler) Refactor 2024-09-23 17:51:07 +02:00
warc (crawler) Refactor boundary between CrawlerRetreiver and HttpFetcherImpl 2024-09-24 15:08:22 +02:00
ContentTags.java (crawler) Refactor 2024-09-23 17:51:07 +02:00
Cookies.java (crawler) Refactor 2024-09-23 17:51:07 +02:00
HttpFetcher.java (crawler) Refactor boundary between CrawlerRetreiver and HttpFetcherImpl 2024-09-24 15:08:22 +02:00
HttpFetcherImpl.java (crawler, EXPERIMENT) Disable content type probing and use Accept header instead 2024-09-30 14:53:01 +02:00
SitemapRetriever.java (crawler) Refactor 2024-09-23 17:51:07 +02:00