mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 13:09:00 +00:00
51e46ad2b0
Since some of the export tasks have been memory hungry, sometimes killing the executor-services, they've been moved to a separate process that can be given a larger Xmx. While doing this, the ProcessMainClass was given utilities for the boilerplate surrounding receiving mq requests and responding to them, some effort was also put toward making the process boot process a bit more uniform. It's still a bit heterogeneous between different processes, but a bit less so for now. |
||
|---|---|---|
| .. | ||
| java/nu/marginalia/index | ||
| build.gradle | ||
| readme.md | ||
The index construction process is responsible for creating the indexes used by the search engine.
There are three types of indexes:
- The forward index, which maps documents to words.
- The full reverse index, which maps words to documents; and includes all words.
- The priority reverse index, which maps words to documents; but includes only the most "important" words (such as those appearing in the title, or with especially high TF-IDF scores).
This is a very light-weight module that delegates the actual work to the modules:
Their respective readme files contain more information about the indexes themselves and how they are constructed.
The process is glued together within IndexConstructorMain, which is the only class of interest in this module.