mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 21:18:58 +00:00
66c1281301
Cleaning out a lot of old junk from the code, and one thing lead to another...
* Build is improved, now constructing docker images with 'jib'. Clean build went from 3 minutes to 50 seconds.
* The ProcessService's spawning is smarter. Will now just spawn a java process instead of relying on the application plugin's generated outputs.
* Project is migrated to GraalVM
* gRPC clients are re-written with a neat fluent/functional style. e.g.
```channelPool.call(grpcStub::method)
.async(executor) // <-- optional
.run(argument);
```
This change is primarily to allow handling ManagedChannel errors, but it turned out to be a pretty clean API overall.
* For now the project is all in on zookeeper
* Service discovery is now based on APIs and not services. Theoretically means we could ship the same code either a monolith or a service mesh.
* To this end, began modularizing a few of the APIs so that they aren't strongly "living" in a service. WIP!
Missing is documentation and testing, and some more breaking apart of code.
|
||
|---|---|---|
| .. | ||
| src/main/java/nu/marginalia/index | ||
| build.gradle | ||
| readme.md | ||
The index construction process is responsible for creating the indexes used by the search engine.
There are three types of indexes:
- The forward index, which maps documents to words.
- The full reverse index, which maps words to documents; and includes all words.
- The priority reverse index, which maps words to documents; but includes only the most "important" words (such as those appearing in the title, or with especially high TF-IDF scores).
This is a very light-weight module that delegates the actual work to the modules:
Their respective readme files contain more information about the indexes themselves and how they are constructed.
The process is glued together within IndexConstructorMain, which is the only class of interest in this module.