Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 21:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(doc) Update RandomWriteFunnel documentation

Browse Source
This commit is contained in:
Viktor Lofgren 2024-02-06 12:35:24 +01:00
parent f89274d1ea
commit d1aeb030f2
1 changed files with 18 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
code/libraries/random-write-funnel/readme.md
View File

@ -1,12 +1,25 @@
# Random Write Funnel
This micro-library with strategies for solving the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when
writing large files out of order to disk. It offers a simple API to write data to a file in a
random order, while localizing the writes.
This micro-library solves the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when
writing large files out of order to disk. It does this by bucketing the writes into several temporary files,
Several strategies are available from the [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java)
interface.
* Writing to a memory mapped file (non-solution, for small files)
* Writing to a memory buffer (for systems with enough memory)
* [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java) - Not bound by memory.
The data is written in a native byte order.
## RandomWriteFunnel
The RandomWriteFunnel solves the problem by bucketing the writes into several temporary files,
which are then evaluated to construct the larger file with a more predictable order of writes.
Even though it effectively writes 2.5x as much data to disk than simply attempting to
construct the file directly, it is *much* faster than thrashing an SSD with dozens of gigabytes
of small random writes.
of small random writes, which is what tends to happen if you naively mmap a file that is larger
than the system RAM, and write to it in a random order.
## Demo
```java
@ -28,4 +41,5 @@ catch (IOException ex) {
## Central Classes
* [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java)
* [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java)