Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 21:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
6f830f0e08
MarginaliaSearch/code/process-models/crawling-model
History
Viktor Lofgren 22c8fb3f59 (crawler) Fix a bug where reference copies of crawl data was written without etag and last-modified 
This commit also adds a band-aid to ParquetSerializableCrawlDataStream to fetch this from the 304-entity.  This can be removed in a few months.
2024-01-18 16:02:27 +01:00
..
src (crawler) Fix a bug where reference copies of crawl data was written without etag and last-modified 2024-01-18 16:02:27 +01:00
build.gradle (warc) Filter WarcResponses based on X-Robots-Tags 2023-12-16 15:58:27 +01:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

readme.md

Crawling Models

Contains models shared by the crawling-process and converting-process.

Central Classes

Serialization