Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
aebb2652e8
MarginaliaSearch/code/index/index-forward
History
Viktor Lofgren aebb2652e8 (wip) Extract and encode spans data 
Refactoring keyword extraction to extract spans information.

Modifying the intermediate storage of converted data to use the new slop library, which is allows for easier storage of ad-hoc binary data like spans and positions.

This is a bit of a katamari damacy commit that ended up dragging along a bunch of other fairly tangentially related changes that are hard to break out into separate commits after the fact.  Will push as-is to get back to being able to do more isolated work.
2024-07-27 11:44:13 +02:00
..
java/nu/marginalia/index/forward (wip) Extract and encode spans data 2024-07-27 11:44:13 +02:00
test/nu/marginalia/index/forward (wip) Extract and encode spans data 2024-07-27 11:44:13 +02:00
build.gradle (wip) Extract and encode spans data 2024-07-27 11:44:13 +02:00
readme.md (docs) Begin un-fucking the docs after refactoring 2024-02-27 21:22:21 +01:00

readme.md

Forward Index

The forward index contains a mapping from document id to various forms of document metadata.

In practice, the forward index consists of two files, an id file and a data file.

The id file contains a list of sorted document ids, and the data file contains metadata for each document id, in the same order as the id file, with a fixed size record containing data associated with each document id.

Each record contains a binary encoded DocumentMetadata object, as well as a HtmlFeatures bitmask.

Unlike the reverse index, the forward index is not split into two tiers, and the data is in the same order as it is in the source data, and the cardinality of the document IDs is assumed to fit in memory, so it's relatively easy to construct.

Central Classes