2023-03-06 17:32:13 +00:00
|
|
|
# Index Journal
|
|
|
|
|
|
|
|
|
|
The index journal contains a list of entries with keywords and keyword metadata per document.
|
|
|
|
|
|
2023-03-13 16:39:53 +00:00
|
|
|
This journal is written by [processes/loading-process](../../processes/loading-process) and read
|
2023-03-06 17:32:13 +00:00
|
|
|
when constructing the [forward](../index-forward) and [reverse](../index-reverse)
|
|
|
|
|
indices.
|
|
|
|
|
|
2024-09-13 09:01:05 +00:00
|
|
|
The journal uses the [Slop library](https://github.com/MarginaliaSearch/SlopData) to store data
|
|
|
|
|
in a columnar fashion.
|
2024-02-15 09:51:49 +00:00
|
|
|
|
2024-09-13 09:01:05 +00:00
|
|
|
The journal will may be split into multiple files to help index
|
|
|
|
|
construction, as a merge strategy is used to reduce the amount
|
|
|
|
|
of RAM required during index construction.
|
2024-02-15 09:51:49 +00:00
|
|
|
|
2024-09-13 09:01:05 +00:00
|
|
|
Unlike most slop data stores, the index journal allows direct access
|
|
|
|
|
to the underlying columns, as the needs of the index construction processes
|
|
|
|
|
are fairly varied.
|
