Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
f6f036b9b1
MarginaliaSearch/code/processes/crawling-process/model/java/nu/marginalia/io
History
Viktor Lofgren f6f036b9b1 Switch to new Slop format for crawl data storage and processing. 
Replaces Parquet output and processing with the new Slop-based format. Includes data migration functionality, updates to handling and writing of crawl data, and introduces support for SLOP in domain readers and converters.
2024-12-15 19:34:03 +01:00
..
crawldata/format Switch to new Slop format for crawl data storage and processing. 2024-12-15 19:34:03 +01:00
CrawledDomainReader.java Switch to new Slop format for crawl data storage and processing. 2024-12-15 19:34:03 +01:00
CrawlerOutputFile.java Switch to new Slop format for crawl data storage and processing. 2024-12-15 19:34:03 +01:00
SerializableCrawlDataStream.java (*) Remove the crawl spec abstraction 2024-10-03 13:41:17 +02:00