mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 13:09:00 +00:00
aebb2652e8
Refactoring keyword extraction to extract spans information. Modifying the intermediate storage of converted data to use the new slop library, which is allows for easier storage of ad-hoc binary data like spans and positions. This is a bit of a katamari damacy commit that ended up dragging along a bunch of other fairly tangentially related changes that are hard to break out into separate commits after the fact. Will push as-is to get back to being able to do more isolated work. |
||
|---|---|---|
| .. | ||
| array | ||
| blocking-thread-pool | ||
| braille-block-punch-cards | ||
| btree | ||
| coded-sequence | ||
| easy-lsh | ||
| geo-ip | ||
| guarded-regex | ||
| language-processing | ||
| message-queue | ||
| random-write-funnel | ||
| slop | ||
| term-frequency-dict | ||
| test-helpers | ||
| LICENSE.txt | ||
| readme.md | ||
Libraries
These are libraries that are not strongly coupled to the search engine's business logic. These libraries may not depend on features, services, processes, models, etc.
NOTE: These libraries are co-licensed under the MIT license.
Libraries
- The array library is for memory mapping large memory-areas, which Java has bad support for. It's designed to be able to easily replaced when Java's Foreign Function And Memory API is released.
- The btree library offers a static BTree implementation based on the array library.
- language-processing contains primitives for sentence extraction and POS-tagging.
- The message-queue library.
Micro libraries
- easy-lsh is a simple locality-sensitive hash for document deduplication
- guarded-regex makes predicated regular expressions clearer
- random-write-funnel is a tool for reducing write amplification when constructing large files out of order.
- next-prime naive brute force prime sieve.
- braille-block-punch-cards renders bit masks into human-readable dot matrices using the braille block.