Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
a73f1ab0ac
MarginaliaSearch/code/process-models/crawling-model
History
Viktor Lofgren 440e097d78 (crawler) WIP integration of WARC files into the crawler and converter process. 
This commit is in a pretty rough state.  It refactors the crawler fairly significantly to offer better separation of concerns.  It replaces the zstd compressed json files used to store crawl data with WARC files entirely, and the converter is modified to be able to consume this data.  This works, -ish.

There appears to be some bug relating to reading robots.txt, and the X-Robots-Tag header is no longer processed either.

A problem is that the WARC files are a bit too large.  It will probably be likely to introduce a new format to store the crawl data long term, something like parquet; and use WARCs for intermediate storage to enable the crawler to be restarted without needing a recrawl.
2023-12-13 15:33:42 +01:00
..
src/main/java (crawler) WIP integration of WARC files into the crawler and converter process. 2023-12-13 15:33:42 +01:00
build.gradle (crawler) WIP integration of WARC files into the crawler and converter process. 2023-12-13 15:33:42 +01:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

readme.md

Crawling Models

Contains models shared by the crawling-process and converting-process.

Central Classes

Serialization