Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 21:29:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,541 Commits 4 Branches 108 Tags 17 MiB
47e58a21c6
Commit Graph

2 Commits

Author SHA1 Message Date
Viktor Lofgren
f6f036b9b1 Switch to new Slop format for crawl data storage and processing. 
Replaces Parquet output and processing with the new Slop-based format. Includes data migration functionality, updates to handling and writing of crawl data, and introduces support for SLOP in domain readers and converters.
 2024-12-15 19:34:03 +01:00
Viktor Lofgren
b510b7feb8 Spike for storing crawl data in slop instead of parquet 
This seems to reduce RAM overhead to 100s of MB (from ~2 GB), as well as roughly double the read speeds.  On disk size is virtually identical.
 2024-12-15 15:49:47 +01:00