5 Commits

Author	SHA1	Message	Date
Viktor Lofgren	787a20cbaa	(crawling-model) Implement a parquet format for crawl data ... This is not hooked into anything yet. The change also makes modifications to the parquet-floor library to support reading and writing of byte[] arrays. This is desirable since we may in the future want to support inputs that are not text-based, and codifying the assumption that each document is a string will definitely cause us grief down the line.	2023-12-13 16:22:19 +01:00
Viktor Lofgren	9f672a0cf4	(parquet-floor) Modify the parquet library to permit list-fields.	2023-09-13 15:56:35 +02:00
Viktor Lofgren	a00cabe223	(parquet-floor) Patch in support for writing and reading repeated values	2023-09-11 14:06:43 +02:00
Viktor Lofgren	dbe974f510	(parquet) Use ZSTD compression by default.	2023-09-11 09:02:58 +02:00
Viktor Lofgren	a284682deb	(parquet) Add parquet library ... This small library, while great, will require some modifications to fit the project's needs, so it goes into third-party directly.	2023-09-05 10:38:51 +02:00