MarginaliaSearch/code/process-mqapi/java/nu/marginalia/mqapi/converting/ConvertAction.java

package nu.marginalia.mqapi.converting;

public enum ConvertAction {
    ConvertCrawlData,
    SideloadEncyclopedia,
    SideloadDirtree,
    SideloadWarc,
    SideloadReddit,
    SideloadStackexchange
}
(WIP) Make it possible to sideload encyclopedia data. This is mostly a pilot track for sideloading other large websites. Also change coverter to produce a more compact output (java serialization instead of json). 2023-07-28 16:14:43 +00:00			`package nu.marginalia.mqapi.converting;`

			`public enum ConvertAction {`
			`ConvertCrawlData,`
(*) last touches 2023-08-07 10:57:38 +00:00			`SideloadEncyclopedia,`
(converter) Make it possible to sideload documents from a directory tree 2023-09-17 12:35:06 +00:00			`SideloadDirtree,`
(convert) Add basic support for Warc file sideloading This update includes the integration of the jwarc library and implements support for Warc file sideloading, as a first trial integration with this library. 2023-12-06 17:43:55 +00:00			`SideloadWarc,`
(converter) Loader for reddit data Adds experimental sideloading support for pusshift.io style reddit data. This dataset is limited to data older than 2023, due to licensing changes making large-scale data extraction difficult. Since the median post quality on reddit is not very good, he sideloader will only load a subset of self-texts and top-level comments that have sufficiently many upvotes. Empirically this appears to mostly return good matches, even if it probably could index more. Tests were written for this, but all require local reddit data which can't be distributed with the source code. If these can not be found, the tests will shortcircuit as OK. They're mostly there for debugging, and it's fine if they don't always run. The change also refactors the sideloading a bit since it was a bit messy. 2024-02-14 10:11:23 +00:00			`SideloadReddit,`
(*) last touches 2023-08-07 10:57:38 +00:00			`SideloadStackexchange`
(WIP) Make it possible to sideload encyclopedia data. This is mostly a pilot track for sideloading other large websites. Also change coverter to produce a more compact output (java serialization instead of json). 2023-07-28 16:14:43 +00:00			`}`