mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 13:19:02 +00:00
467ba5be20
This change splits the previous 'repartition' action into two steps, one for recalculating the domain rankings, and one for recalculating the other ranking sets. Since only the first is necessary before the index construction, the rest can be delayed until after... To avoid issues in handling the shotgun blast of MqNotifications, Service was switched over to use a synchronous message queue instead of an asynchronous one. The change also modifies the behavior so that only node 1 will push the changes to the EC_DOMAIN database table, to avoid unnecessary db locks and contention with the loader. Additionally, the change fixes a bug where the index construction code wasn't actually picking up the rankings data. Since the index construction used to be performed by the index-service, merely saving the data to memory was enough for it to be accessible within the index-construction logic, but since it's been broken out into a separate process, the new process just injected an empty DomainRankings object instead. To fix this, DomainRankings can now be persisted to disk, and a pre-loaded version of the object is injected into the index-construction process. |
||
|---|---|---|
| .. | ||
| domain-ranking | ||
| index-forward | ||
| index-journal | ||
| index-query | ||
| index-reverse | ||
| result-ranking | ||
| readme.md | ||
Index
These are components that offer functionality for the index-service.
Indexes
There are two indexes with accompanying tools for constructing them.
-
index-reverse is code for
word->documentindexes. There are two such indexes, one containing only document-word pairs that are flagged as important, e.g. the word appears in the title or has a high TF-IDF. This allows good results to be discovered quickly without having to sift through ten thousand bad ones first. -
index-forward is the
document->wordindex containing metadata about each word, such as its position. It is used after identifying candidate search results via the reverse index to fetch metadata and rank the results.
These indices rely heavily on the libraries/btree and libraries/array components.
Algorithms
- domain-ranking contains domain ranking algorithms.
- result-ranking contains logic for ranking search results by relevance.
Libraries
- index-query contains structures for evaluating search queries.
- index-journal contains tools for writing and reading index data.