mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 13:19:02 +00:00
51e46ad2b0
Since some of the export tasks have been memory hungry, sometimes killing the executor-services, they've been moved to a separate process that can be given a larger Xmx. While doing this, the ProcessMainClass was given utilities for the boilerplate surrounding receiving mq requests and responding to them, some effort was also put toward making the process boot process a bit more uniform. It's still a bit heterogeneous between different processes, but a bit less so for now. |
||
|---|---|---|
| .. | ||
| java/nu/marginalia/index/forward | ||
| test/nu/marginalia/index/forward | ||
| build.gradle | ||
| readme.md | ||
Forward Index
The forward index contains a mapping from document id to various forms of document metadata.
In practice, the forward index consists of two files, an id file and a data file.
The id file contains a list of sorted document ids, and the data file contains
metadata for each document id, in the same order as the id file, with a fixed
size record containing data associated with each document id.
Each record contains a binary encoded DocumentMetadata object, as well as a HtmlFeatures bitmask.
Unlike the reverse index, the forward index is not split into two tiers, and the data is in the same order as it is in the source data, and the cardinality of the document IDs is assumed to fit in memory, so it's relatively easy to construct.
Central Classes
- ForwardIndexConverter constructs the index.
- ForwardIndexReader interrogates the index.