mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 05:18:58 +00:00
0307c55f9f
To avoid having to either hard-code or manually configure service addresses (possibly several dozen), and to reduce the project's dependency on docker to deal with routing and discovery, the option to use [Zookeeper](https://zookeeper.apache.org/) to manage services and discovery has been added. A service registry interface was added, with a Zookeeper implementation and a basic implementation that only works on docker and hard-codes everything. The last remaining REST service, the assistant-service, has been migrated to gRPC. This also proved a good time to clear out primordial technical debt from the root of the codebase. The 'service-client' library has been taken behind the barn and given a last farewell. It's replaced by a small library for managing gRPC channels. Since it's no longer used by anything, RxJava has been removed as a dependency from the project. Although the current state seems reasonably stable, this is a work-in-progress commit. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
Big String
Microlibrary that offers string compression. This is useful when having to load tens of thousands of HTML documents in memory during conversion. XML has been described as the opposite of a compression scheme, and as a result, HTML compresses ridiculously well.
Configuration
If the Java property 'bigstring.disabled' is set to true, the BigString class will not compress strings.
Demo
List<BigString> manyBigStrings = new ArrayList<>();
for (var file : files) {
// BigString.encode may or may not compress the string
// depeneding on its size
manyBigStrings.add(BigString.encode(readFile(file)));
}
for (var bs : manyBigStrings) {
String decompressedString = bs.decompress();
byte[] bytes = bs.getBytes();
int len = bs.getLength();
}