Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
072b5fcd12
MarginaliaSearch/code/common/model
History
Viktor Lofgren 072b5fcd12 Implement Warc-recording wrapper for OkHttp3 client 
This is a first step of using WARC as an intermediate flight recorder style step in the crawler, ultimately aimed at being able to resume crawls if the crawler is restarted.  This component is currently not hooked into anything.

The OkHttp3 client wrapper class 'WarcRecordingFetcherClient' was implemented for web archiving. This allows for the recording of HTTP requests and responses. New classes were introduced, 'WarcDigestBuilder', 'IpInterceptingNetworkInterceptor', and 'WarcProtocolReconstructor'.

The JWarc dependency was added to the build.gradle file, and relevant unit tests were also introduced. Some HttpFetcher-adjacent structural changes were also done for better organization.
2023-12-08 13:49:16 +01:00
..
src Implement Warc-recording wrapper for OkHttp3 client 2023-12-08 13:49:16 +01:00
build.gradle (build) Move unit test configuration to root build.gradle 2023-10-04 12:46:22 +02:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

readme.md

Model

This package contains common models to the search engine

Central Classes