Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(doc) Update RandomWriteFunnel documentation

Browse Source
This commit is contained in:
Viktor Lofgren 2024-02-06 12:35:24 +01:00
parent f89274d1ea
commit d1aeb030f2
1 changed files with 18 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
code/libraries/random-write-funnel/readme.md
View File

@ -1,12 +1,25 @@
# Random Write Funnel This micro-library with strategies for solving the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when
writing large files out of order to disk. It offers a simple API to write data to a file in a
random order, while localizing the writes.
This micro-library solves the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when Several strategies are available from the [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java)
writing large files out of order to disk. It does this by bucketing the writes into several temporary files, interface.
* Writing to a memory mapped file (non-solution, for small files)
* Writing to a memory buffer (for systems with enough memory)
* [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java) - Not bound by memory.
The data is written in a native byte order.
## RandomWriteFunnel
The RandomWriteFunnel solves the problem by bucketing the writes into several temporary files,
which are then evaluated to construct the larger file with a more predictable order of writes. which are then evaluated to construct the larger file with a more predictable order of writes.
Even though it effectively writes 2.5x as much data to disk than simply attempting to Even though it effectively writes 2.5x as much data to disk than simply attempting to
construct the file directly, it is *much* faster than thrashing an SSD with dozens of gigabytes construct the file directly, it is *much* faster than thrashing an SSD with dozens of gigabytes
of small random writes. of small random writes, which is what tends to happen if you naively mmap a file that is larger
than the system RAM, and write to it in a random order.
## Demo ## Demo
```java ```java
@ -28,4 +41,5 @@ catch (IOException ex) {
## Central Classes ## Central Classes
* [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java)
* [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java) * [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java)