mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 05:18:58 +00:00
fab36d6e63
Adds experimental sideloading support for pusshift.io style reddit data. This dataset is limited to data older than 2023, due to licensing changes making large-scale data extraction difficult. Since the median post quality on reddit is not very good, he sideloader will only load a subset of self-texts and top-level comments that have sufficiently many upvotes. Empirically this appears to mostly return good matches, even if it probably could index more. Tests were written for this, but all require local reddit data which can't be distributed with the source code. If these can not be found, the tests will shortcircuit as OK. They're mostly there for debugging, and it's fine if they don't always run. The change also refactors the sideloading a bit since it was a bit messy. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
Control Service
The control service provides an operator's user interface. By default this interface is exposed on port 8081. It does not offer any sort of access control or authentication.
The control service will itself execute tasks that affect the entire system, but delegate node-specific tasks to the corresponding executor-service via the executor-api.
Conceptually the application is broken into three parts:
- Application specific tasks relate to the high level abstractions such as blacklisting and API keys
- System tasks relate to low level abstractions such as the message queue and event log.
- Node tasks relate to index node specific tasks, such as crawling and indexing.
Central Classes
See Also
- processes
- libraries/message-queue - The Message Queue and Actor abstractions