Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 21:29:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3bc99639a0
MarginaliaSearch/code/processes/live-crawling-process/test/nu/marginalia/livecrawler
History
Viktor Lofgren 927bc0b63c (live-crawler) Add Accept-Encoding: gzip to outbound requests 
This change adds `Accept-Encoding: gzip` to all outbound requests from the live crawler and feed fetcher, and the corresponding decoding logic for the compressed response data.

The change addresses issue #136, save for making the fetcher's requests conditional.
2024-12-27 03:59:34 +01:00
..
LiveCrawlDataSetTest.java (live-crawler) Keep track of bad URLs 2024-11-22 00:55:46 +01:00
SimpleLinkScraperTest.java (live-crawler) Add Accept-Encoding: gzip to outbound requests 2024-12-27 03:59:34 +01:00