MarginaliaSearch/code/common
Viktor Lofgren 467ba5be20 (index-construction) Split repartition into two actions
This change splits the previous 'repartition' action into two steps, one for recalculating the domain rankings, and one for recalculating the other ranking sets.  Since only the first is necessary before the index construction, the rest can be delayed until after...

To avoid issues in handling the shotgun blast of MqNotifications, Service was switched over to use a synchronous message queue instead of an asynchronous one.

The change also modifies the behavior so that only node 1 will push the changes to the EC_DOMAIN database table, to avoid unnecessary db locks and contention with the loader.

Additionally, the change fixes a bug where the index construction code wasn't actually picking up the rankings data.

Since the index construction used to be performed by the index-service, merely saving the data to memory was enough for it to be accessible within the index-construction logic, but since it's been broken out into a separate process, the new process just injected an empty DomainRankings object instead.

To fix this, DomainRankings can now be persisted to disk, and a pre-loaded version of the object is injected into the index-construction process.
2024-02-06 17:20:07 +01:00
..
config (minor) Fix test and improve error message 2024-01-31 11:53:41 +01:00
db (*) Fix broken test, clean up code 2024-01-24 12:50:41 +01:00
linkdb (test) Clean up test usage of migrations 2024-01-12 15:55:50 +01:00
model (*) Fix bug in EdgeDomain where it would permit domains with a trailing period, DNS style. 2023-12-29 16:36:01 +01:00
process (control) New GUI for exporting crawl data samples 2024-01-23 17:08:21 +01:00
renderer (search) Fix acknowledgement page for domain complaints rendering as plain text 2024-01-10 09:26:34 +01:00
service (index-construction) Split repartition into two actions 2024-02-06 17:20:07 +01:00
service-client (search) Move site information out of the search service and into assistant. 2023-12-09 16:30:06 +01:00
service-discovery (mqapi/control) Repair repartition endpoint, deprecate notify endpoints. 2023-11-27 16:01:12 +01:00
readme.md Update readme.md 2023-03-25 15:27:11 +01:00

Common

These are packages containing the basic building blocks for running a service as well as shared models.

  • db contains SQL code and some database-related utilities.
  • config contains some @Injectables.
  • renderer contains utility code for rendering website templates.
  • service is the shared base classes for main methods and web services.
  • service-client is the shared base class for RPC.
  • service-discovery contains tools that lets the services find each other.
  • process contains boiler plate for batch processes.