MarginaliaSearch/code
Viktor Lofgren d0aa754252 (long-array) Implement java.lang.foreign.Arena based lifecycle control for LongArray.
Further de-ByteBuffer:ing of these classes is to be done, but this is the smallest most urgently needed benefit.

This commit is a WIP but in a fully working state, pushing due to the importance of the changes to offer lifecycle control over mmaps.
2023-09-24 10:40:06 +02:00
..
api (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
common (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
features-convert (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
features-crawl (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
features-index (long-array) Implement java.lang.foreign.Arena based lifecycle control for LongArray. 2023-09-24 10:40:06 +02:00
features-search (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
libraries (long-array) Implement java.lang.foreign.Arena based lifecycle control for LongArray. 2023-09-24 10:40:06 +02:00
process-models (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
processes (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
services-core (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
services-satellite (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
tools (*) Upgrade to JDK21 with preview enabled. 2023-09-24 10:38:59 +02:00
readme.md (crawl-spec) Parquetify crawl spec 2023-09-17 09:41:34 +02:00

Code

This is a pretty large and diverse project with many moving parts.

You'll find a short description in each module of what it does and how it relates to other modules. The modules each have names like "library" or "process" or "feature". These have specific meanings. See doc/module-taxonomy.md.

Overview

A map of the most important components and how they relate can be found below.

image

Services

Processes

Processes are batch jobs that deal with data retrieval, processing and loading.

Tools

Features

Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent, but isolated.

Libraries and primitives

Libraries are stand-alone code that is independent of the domain logic.

  • common elements for creating a service, a client etc.
  • libraries containing non-search specific code.
    • array - large memory mapped area library
    • btree - static btree library