From d1aeb030f222d80648fc08480a8ce1e1ef6b9d1e Mon Sep 17 00:00:00 2001 From: Viktor Lofgren Date: Tue, 6 Feb 2024 12:35:24 +0100 Subject: [PATCH] (doc) Update RandomWriteFunnel documentation --- code/libraries/random-write-funnel/readme.md | 22 ++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/code/libraries/random-write-funnel/readme.md b/code/libraries/random-write-funnel/readme.md index 4581e17e..fc02b955 100644 --- a/code/libraries/random-write-funnel/readme.md +++ b/code/libraries/random-write-funnel/readme.md @@ -1,12 +1,25 @@ -# Random Write Funnel +This micro-library with strategies for solving the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when +writing large files out of order to disk. It offers a simple API to write data to a file in a +random order, while localizing the writes. -This micro-library solves the problem of [write amplification](https://en.wikipedia.org/wiki/Write_amplification) when -writing large files out of order to disk. It does this by bucketing the writes into several temporary files, +Several strategies are available from the [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java) +interface. + +* Writing to a memory mapped file (non-solution, for small files) +* Writing to a memory buffer (for systems with enough memory) +* [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java) - Not bound by memory. + +The data is written in a native byte order. + +## RandomWriteFunnel + +The RandomWriteFunnel solves the problem by bucketing the writes into several temporary files, which are then evaluated to construct the larger file with a more predictable order of writes. Even though it effectively writes 2.5x as much data to disk than simply attempting to construct the file directly, it is *much* faster than thrashing an SSD with dozens of gigabytes -of small random writes. +of small random writes, which is what tends to happen if you naively mmap a file that is larger +than the system RAM, and write to it in a random order. ## Demo ```java @@ -28,4 +41,5 @@ catch (IOException ex) { ## Central Classes +* [RandomFileAssembler](src/main/java/nu/marginalia/rwf/RandomFileAssembler.java) * [RandomWriteFunnel](src/main/java/nu/marginalia/rwf/RandomWriteFunnel.java) \ No newline at end of file