mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 13:09:00 +00:00
66c1281301
Cleaning out a lot of old junk from the code, and one thing lead to another...
* Build is improved, now constructing docker images with 'jib'. Clean build went from 3 minutes to 50 seconds.
* The ProcessService's spawning is smarter. Will now just spawn a java process instead of relying on the application plugin's generated outputs.
* Project is migrated to GraalVM
* gRPC clients are re-written with a neat fluent/functional style. e.g.
```channelPool.call(grpcStub::method)
.async(executor) // <-- optional
.run(argument);
```
This change is primarily to allow handling ManagedChannel errors, but it turned out to be a pretty clean API overall.
* For now the project is all in on zookeeper
* Service discovery is now based on APIs and not services. Theoretically means we could ship the same code either a monolith or a service mesh.
* To this end, began modularizing a few of the APIs so that they aren't strongly "living" in a service. WIP!
Missing is documentation and testing, and some more breaking apart of code.
|
||
|---|---|---|
| .. | ||
| domain-ranking | ||
| index-forward | ||
| index-journal | ||
| index-query | ||
| index-reverse | ||
| result-ranking | ||
| readme.md | ||
Index
These are components that offer functionality for the index-service.
Indexes
There are two indexes with accompanying tools for constructing them.
-
index-reverse is code for
word->documentindexes. There are two such indexes, one containing only document-word pairs that are flagged as important, e.g. the word appears in the title or has a high TF-IDF. This allows good results to be discovered quickly without having to sift through ten thousand bad ones first. -
index-forward is the
document->wordindex containing metadata about each word, such as its position. It is used after identifying candidate search results via the reverse index to fetch metadata and rank the results.
These indices rely heavily on the libraries/btree and libraries/array components.
Algorithms
- domain-ranking contains domain ranking algorithms.
- result-ranking contains logic for ranking search results by relevance.
Libraries
- index-query contains structures for evaluating search queries.
- index-journal contains tools for writing and reading index data.