mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 05:18:58 +00:00
d023e399d2
The term data iterator is quite hot and was performing buffer slice operations that were not necessary. Replacing with a fixed pointer alias that can be repositioned to the relevant data. The positions data was also being wrapped in a GammaCodedSequence only to be immediately un-wrapped. Removed this unnecessary step and move to copying the buffer directly instead. |
||
|---|---|---|
| .. | ||
| java/nu/marginalia/index/journal | ||
| test/nu/marginalia/index/journal | ||
| build.gradle | ||
| readme.md | ||
Index Journal
The index journal contains a list of entries with keywords and keyword metadata per document.
This journal is written by processes/loading-process and read when constructing the forward and reverse indices.
The journal format is a file header, followed by a zstd-compressed list of entries, each containing a header with document-level data, and a data section with keyword-level data.
The journal data may be split into multiple files, and the journal writers and readers are designed to handle this transparently via their Paging implementation.