edc1acbb7e
The EC_DOMAIN_LINK MariaDB table stores links between domains. This is problematic, as both updating and querying this table is very slow in relation to how small the data is (~10 GB). This slowness is largely caused by the database enforcing ACID guarantees we don't particularly need. This changeset replaces the EC_DOMAIN_LINK table with a file in each index node containing 32 bit integer pairs corresponding to links between two domains. This file is loaded in memory in each node, and can be queried via the Query Service. A migration step is needed before this file is created in each node. Until that happens, the actual data is loaded from the EC_DOMAIN_LINK table, but accessed as though it was a file. The changeset also migrates/renames the links.db file to documents.db to avoid naming confusion between the two. |
||
|---|---|---|
| .. | ||
| api | ||
| common | ||
| features-convert | ||
| features-crawl | ||
| features-index | ||
| features-qs | ||
| features-search | ||
| libraries | ||
| process-models | ||
| processes | ||
| services-application | ||
| services-core | ||
| tools | ||
| readme.md | ||
Code
This is a pretty large and diverse project with many moving parts.
You'll find a short description in each module of what it does and how it relates to other modules. The modules each have names like "library" or "process" or "feature". These have specific meanings. See doc/module-taxonomy.md.
Overview
A map of the most important components and how they relate can be found below.
The core part of the search engine is the index service, which is responsible for storing and retrieving the document data. The index serive is partitioned, along with the executor service, which is responsible for executing processes. At least one instance of each service must be run, but more can be run alongside. Multiple partitions is desirable in production to distribute load across multiple physical drives, as well as reducing the impact of downtime.
Search queries are delegated via the query service, which is a proxy that fans out the query to all eligible index services. The control service is responsible for distributing commands to the executor service, and for monitoring the health of the system. It also offers a web interface for operating the system.
Services
- core services Most of these services are stateful, memory hungry, and doing heavy lifting.
- application services Mostly stateless gateways providing access to the core services.
-
- api - public API
-
- search - marginalia search application
- an internal API
Processes
Processes are batch jobs that deal with data retrieval, processing and loading. These are spawned and orchestrated by the executor service, which is controlled by the control service.
Tools
Features
Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent, but isolated.
Libraries and primitives
Libraries are stand-alone code that is independent of the domain logic.