Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 13:19:02 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
46423612e3
MarginaliaSearch/code/services-core/assistant-service/build.gradle

63 lines
1.7 KiB
Groovy
Raw Normal View History

Restructuring the git repo
2023-03-04 12:19:01 +00:00
plugins {
id 'java'
(*) Upgrade to JDK21 with preview enabled. ... also move some common configuration into the root build.gradle-file. Support for JDK21 in lombok is a bit sketchy at the moment, but it seems to work. This upgrade is kind of important as the new index construction really benefits from Arena based lifecycle control over off-heap memory.
2023-09-23 09:01:43 +00:00
Restructuring the git repo
2023-03-04 12:19:01 +00:00
id 'application'
id 'jvm-test-suite'
(zk-registry) epic jak shaving WIP Cleaning out a lot of old junk from the code, and one thing lead to another... * Build is improved, now constructing docker images with 'jib'. Clean build went from 3 minutes to 50 seconds. * The ProcessService's spawning is smarter. Will now just spawn a java process instead of relying on the application plugin's generated outputs. * Project is migrated to GraalVM * gRPC clients are re-written with a neat fluent/functional style. e.g. ```channelPool.call(grpcStub::method) .async(executor) // <-- optional .run(argument); ``` This change is primarily to allow handling ManagedChannel errors, but it turned out to be a pretty clean API overall. * For now the project is all in on zookeeper * Service discovery is now based on APIs and not services. Theoretically means we could ship the same code either a monolith or a service mesh. * To this end, began modularizing a few of the APIs so that they aren't strongly "living" in a service. WIP! Missing is documentation and testing, and some more breaking apart of code.
2024-02-22 13:01:23 +00:00
id 'com.google.cloud.tools.jib' version '3.4.0'
Restructuring the git repo
2023-03-04 12:19:01 +00:00
}
application {
mainClass = 'nu.marginalia.assistant.AssistantMain'
applicationName = 'assistant-service'
}
Gradle shouldn't generate dist zips, zipping jar files is slow and also just ridiculous when you realize jar files are zip files and you can't compress a file twice using the same algo.
2023-03-11 10:34:51 +00:00
tasks.distZip.enabled = false
Restructuring the git repo
2023-03-04 12:19:01 +00:00
java {
toolchain {
(*) Upgrade to JDK21 with preview enabled. ... also move some common configuration into the root build.gradle-file. Support for JDK21 in lombok is a bit sketchy at the moment, but it seems to work. This upgrade is kind of important as the new index construction really benefits from Arena based lifecycle control over off-heap memory.
2023-09-23 09:01:43 +00:00
languageVersion.set(JavaLanguageVersion.of(21))
Restructuring the git repo
2023-03-04 12:19:01 +00:00
}
}
(refac) Remove src/main from all source code paths. Look, this will make the git history look funny, but trimming unnecessary depth from the source tree is a very necessary sanity-preserving measure when dealing with a super-modularized codebase like this one. While it makes the project configuration a bit less conventional, it will save you several clicks every time you jump between modules. Which you'll do a lot, because it's *modul*ar. The src/main/java convention makes a lot of sense for a non-modular project though. This ain't that.
2024-02-23 15:13:40 +00:00
apply from: "$rootProject.projectDir/srcsets.gradle"
(gradle) Add task alias 'docker' for 'jibDockerBuild' The change also moves the jib boilerplate to an include.
2024-02-28 10:59:15 +00:00
apply from: "$rootProject.projectDir/docker.gradle"
(refac) Remove src/main from all source code paths. Look, this will make the git history look funny, but trimming unnecessary depth from the source tree is a very necessary sanity-preserving measure when dealing with a super-modularized codebase like this one. While it makes the project configuration a bit less conventional, it will save you several clicks every time you jump between modules. Which you'll do a lot, because it's *modul*ar. The src/main/java convention makes a lot of sense for a non-modular project though. This ain't that.
2024-02-23 15:13:40 +00:00
Restructuring the git repo
2023-03-04 12:19:01 +00:00
dependencies {
The refactoring will continue until morale improves.
2023-03-12 09:04:48 +00:00
implementation project(':third-party:symspell')
(zk-registry) epic jak shaving WIP Cleaning out a lot of old junk from the code, and one thing lead to another... * Build is improved, now constructing docker images with 'jib'. Clean build went from 3 minutes to 50 seconds. * The ProcessService's spawning is smarter. Will now just spawn a java process instead of relying on the application plugin's generated outputs. * Project is migrated to GraalVM * gRPC clients are re-written with a neat fluent/functional style. e.g. ```channelPool.call(grpcStub::method) .async(executor) // <-- optional .run(argument); ``` This change is primarily to allow handling ManagedChannel errors, but it turned out to be a pretty clean API overall. * For now the project is all in on zookeeper * Service discovery is now based on APIs and not services. Theoretically means we could ship the same code either a monolith or a service mesh. * To this end, began modularizing a few of the APIs so that they aren't strongly "living" in a service. WIP! Missing is documentation and testing, and some more breaking apart of code.
2024-02-22 13:01:23 +00:00
implementation project(':code:functions:math')
implementation project(':code:functions:math:api')
implementation project(':code:functions:domain-info')
implementation project(':code:functions:domain-info:api')
Move all code to a code directory.
2023-03-07 16:14:32 +00:00
implementation project(':code:common:config')
implementation project(':code:common:service')
implementation project(':code:common:model')
(search) Move site information out of the search service and into assistant. This reduces the impact of restarting the search service, as the site information takes a few minutes to load during which it's not available. It also permits exposing this information via API in the future if there is interest in this. The assistant service was also modified to do a late load of the suggestions trie, as this is a major contributor to its start-up time. Finally, some changes were made to the client library, a new get() method was added that takes a TypeToken to allow deserialization of generics such as List<Foo>, and the scheduler was also modified to use virtual threads.
2023-12-09 15:29:35 +00:00
implementation project(':code:common:db')
Restructuring the git repo
2023-03-04 12:19:01 +00:00
The refactoring will continue until morale improves.
2023-03-12 09:50:31 +00:00
implementation project(':code:features-search:screenshots')
Restructuring the git repo
2023-03-04 12:19:01 +00:00
(*) Refactor GeoIP-related code In this commit, GeoIP-related classes are refactored and relocated to a common library as they are shared across multiple services. The crawler is refactored to enable the GeoIpBlocklist to use the new GeoIpDictionary as the base of its decisions. The converter is modified ot query this data to add a geoip:-keyword to documents to permit limiting a search to the country of the hosting server. The commit also adds due BY-SA attribution in the search engine footer for the source of the IP geolocation data.
2023-12-10 16:30:43 +00:00
implementation project(':code:libraries:geo-ip')
Yet more restructuring.
2023-03-13 22:40:26 +00:00
implementation project(':code:libraries:language-processing')
implementation project(':code:libraries:term-frequency-dict')
Restructuring the git repo
2023-03-04 12:19:01 +00:00
implementation libs.bundles.slf4j
implementation libs.prometheus
(refac) Zookeeper for service-discovery, kill service-client lib (WIP) To avoid having to either hard-code or manually configure service addresses (possibly several dozen), and to reduce the project's dependency on docker to deal with routing and discovery, the option to use [Zookeeper](https://zookeeper.apache.org/) to manage services and discovery has been added. A service registry interface was added, with a Zookeeper implementation and a basic implementation that only works on docker and hard-codes everything. The last remaining REST service, the assistant-service, has been migrated to gRPC. This also proved a good time to clear out primordial technical debt from the root of the codebase. The 'service-client' library has been taken behind the barn and given a last farewell. It's replaced by a small library for managing gRPC channels. Since it's no longer used by anything, RxJava has been removed as a dependency from the project. Although the current state seems reasonably stable, this is a work-in-progress commit.
2024-02-20 10:41:14 +00:00
implementation libs.bundles.grpc
Restructuring the git repo
2023-03-04 12:19:01 +00:00
implementation libs.notnull
implementation libs.guice
implementation libs.spark
implementation libs.opencsv
implementation libs.trove
implementation libs.fastutil
implementation libs.bundles.gson
implementation libs.bundles.mariadb
testImplementation libs.bundles.slf4j.test
testImplementation libs.bundles.junit
testImplementation libs.mockito
}
Reference in New Issue Copy Permalink