90a2d4ae38
Ensure all data is written to writeChannel by looping until the buffer is fully drained. This prevents potential data loss during the close operation and maintains data integrity. |
||
|---|---|---|
| .. | ||
| java/nu/marginalia/index | ||
| test/nu/marginalia/index | ||
| build.gradle | ||
| index.svg | ||
| merging.svg | ||
| preindex.svg | ||
| readme.md | ||
Reverse Index
The reverse index contains a mapping from word to document id.
There are two tiers of this index.
- A priority index which only indexes terms that are flagged with priority flags1.
- A full index that indexes all terms.
The full index also provides access to term-level metadata, while the priority index is a binary index that only offers information about which documents has a specific word.
The priority index is also compressed, while the full index at this point is not.
[1] See WordFlags in common/model and KeywordMetadata in converting-process/ft-keyword-extraction.
Construction
The reverse index is constructed by first building a series of preindexes. Preindexes consist of a Segment and a Documents object. The segment contains information about which word identifiers are present and how many, and the documents contain information about in which documents the words can be found.
These would typically not fit in RAM, so the index journal is paged and the preindexes are constructed small enough to fit in memory, and then merged. Merging sorted arrays is a very fast operation that does not require additional RAM.
Once merged into one large preindex, indexes are added to the preindex data to form a finalized reverse index.
Central Classes
Full index:
- FullPreindex intermediate reverse index state.
- FullIndexConstructor constructs the index.
- FullReverseIndexReader interrogates the index.
Prio index:
- PrioPreindex intermediate reverse index state.
- PrioIndexConstructor constructs the index.
- PrioIndexReader interrogates the index.