mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 21:18:58 +00:00
53c575db3f
To cope with writing large files out of order, the system needs some form of strategy to avoid writing them directly to disk, as this causes insane amounts of disk thrashing. By default, the data is just buffered in RAM. This works well on a large server, but smaller systems struggle. To help systems with small RAM process large amounts of data, the old RandomWriteFunnel is brought back if the system property 'system.conserve-memory' is set to true. RandomWriteFunnel is buffering the construction by creating a series of small files that pigeonhole the writes into rough neighborhoods, and then it goes over the files one by one to construct one area of the file at a time. This is relatively slow and uses more than twice the disk size. A new interface RandomFileAssembler is introduced as an abstraction for this operation. A third strategy, direct mmaps, is also introduced if the file is very small (less than 1 GB). In this domain, disk thrashing is unlikely since it will comfortably fit in RAM. |
||
|---|---|---|
| .. | ||
| array | ||
| big-string | ||
| blocking-thread-pool | ||
| braille-block-punch-cards | ||
| btree | ||
| easy-lsh | ||
| geo-ip | ||
| guarded-regex | ||
| language-processing | ||
| message-queue | ||
| next-prime | ||
| random-write-funnel | ||
| term-frequency-dict | ||
| test-helpers | ||
| LICENSE.txt | ||
| readme.md | ||
Libraries
These are libraries that are not strongly coupled to the search engine's business logic. These libraries may not depend on features, services, processes, models, etc.
NOTE: These libraries are co-licensed under the MIT license.
Libraries
- The array library is for memory mapping large memory-areas, which Java has bad support for. It's designed to be able to easily replaced when Java's Foreign Function And Memory API is released.
- The btree library offers a static BTree implementation based on the array library.
- language-processing contains primitives for sentence extraction and POS-tagging.
- The message-queue library.
Micro libraries
- easy-lsh is a simple locality-sensitive hash for document deduplication
- guarded-regex makes predicated regular expressions clearer
- big-string offers seamless string compression
- random-write-funnel is a tool for reducing write amplification when constructing large files out of order.
- next-prime naive brute force prime sieve.
- braille-block-punch-cards renders bit masks into human-readable dot matrices using the braille block.