Viktor Lofgren
6e1aa7b391
(search) Make style.css depend on jte file changes
...
Also add a hack to ensure classes generated from java code get included in the stylesheet as intended.
2024-12-07 14:11:22 +01:00
Viktor Lofgren
deab9b9516
(search) Clean up start views for search and site-info
2024-12-07 14:11:22 +01:00
Viktor Lofgren
39d99a906a
(search) Add proper tailwind build and host fontawesome locally
2024-12-07 14:11:22 +01:00
Viktor Lofgren
6f72e6e0d3
(explore) Add lazy loading and alt attributes to images
2024-12-07 14:11:22 +01:00
Viktor Lofgren
d786d79483
(site-info) Add whitespace-nowrap to pubDay span in overview.jte
2024-12-07 14:11:22 +01:00
Viktor Lofgren
01510f6c2e
(serp) Add wayback link to search results
2024-12-07 14:11:22 +01:00
Viktor Lofgren
7ba43e9e3f
(site) Adjust sizing of navbars
2024-12-07 14:11:16 +01:00
Viktor Lofgren
97bfcd1353
(site) Layout changes site-info
2024-12-07 14:11:16 +01:00
Viktor Lofgren
aa3c85c196
(site) Mobile layout fixes
2024-12-07 14:11:16 +01:00
Viktor Lofgren
fb75a3827d
(site) Adjust coloration of search results
2024-12-05 16:58:00 +01:00
Viktor Lofgren
7d546d0e2a
(site) Make SearchParameters generate relative URLs instead of absolute
2024-12-05 16:47:22 +01:00
Viktor Lofgren
8fcb6ffd7a
(site-info) Increase contrast in search results for forums, wikis
2024-12-05 16:42:16 +01:00
Viktor Lofgren
f97de0c15a
(site-info) Fix layout
2024-12-05 16:33:46 +01:00
Viktor Lofgren
be9e192b78
(site-info) Fix pagination in backlinks and documents views
2024-12-05 16:26:11 +01:00
Viktor Lofgren
75ae1c9526
(site-info) Do not show 'suggest for crawling' when the ndoe affinity is already set to 0
...
This indicates the domain is already slated for crawling.
2024-12-05 16:18:46 +01:00
Viktor Lofgren
33761a0236
(site-info) Make the search box in the site viewer functional
2024-12-05 16:16:29 +01:00
Viktor Lofgren
19b69b1764
(site-info) Only show samples if feed is absent, never both.
2024-12-05 16:05:03 +01:00
Viktor Lofgren
8b804359a9
(serp) Layout fixes for mobile
2024-12-05 15:59:33 +01:00
Viktor Lofgren
f050bf5c4c
(WIP) Initial semi-working transformation to new tailwind UI
...
Still missing is a proper build, we're currently pulling in tailwind from a CDN, which is no bueno in prod.
There's also a lot of polish remaining everywhere, dead links, etc.
2024-12-05 14:00:17 +01:00
Viktor Lofgren
fdc3efa250
(setup) Remove OpenNLP tokenization model
...
This update eliminates all occurrences of the OpenNLP token model from the setup script, configuration, and test files, as this model file is no longer used.
2024-11-28 16:03:05 +01:00
Viktor Lofgren
9f47ce8d15
(chore) Remove lombok
...
There are likely some instances of delombok gore with this commit.
2024-11-11 21:14:38 +01:00
Viktor Lofgren
a5b4951f23
(chore) Remove use of deprecated STR.-style string templates
2024-11-11 18:02:28 +01:00
Viktor Lofgren
b8e0dc93d7
(search) Correctly show the feeds view when items are present
...
... otherwise show samples. This commit also removes the (Experimental) bit, as this is getting fairly mature.
2024-11-09 17:56:43 +01:00
Viktor Lofgren
bfeb9a4538
(feeds) Retire feedlot the feed bot, move RSS capture into the live-capture service
2024-11-09 17:56:43 +01:00
Viktor Lofgren
542690d9f6
(search-service) Hide pagination when there is only 1 page of results
2024-09-28 13:48:09 +02:00
Viktor Lofgren
fed33ed64a
(search-service) Update screenshot request handling
...
Always request the main site screenshot to ensure staleness checks and necessary updates. Limit additional screenshot requests for similar and linking domains to avoid overloading with a maximum of 5 requests per view.
2024-09-27 14:27:25 +02:00
Viktor Lofgren
23cce0c78a
Add a new function 'Live Capture' for on-demand screenshot capture
...
The screenshots are requested by the site-service, and triggered via the site-info view.
2024-09-27 13:46:34 +02:00
Viktor Lofgren
c757d116bf
(misc) Fix Broken Tests
2024-09-27 13:46:34 +02:00
Viktor Lofgren
0d2390fd13
(search-service) Only autofocus on the query when the query is empty
2024-09-25 14:27:03 +02:00
Viktor Lofgren
4a0356e26f
(search-service) Add pagination support to the search GUI
2024-09-25 14:26:49 +02:00
Viktor Lofgren
8b85a58fea
(search UX) Autofocus on the search form
2024-09-24 15:56:03 +02:00
Viktor Lofgren
8f367d96f8
Merge branch 'master' into term-positions
...
# Conflicts:
# code/index/java/nu/marginalia/index/results/model/ids/TermIdList.java
# code/processes/converting-process/java/nu/marginalia/converting/ConverterMain.java
# code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
# code/processes/crawling-process/java/nu/marginalia/crawl/retreival/fetcher/HttpFetcherImpl.java
# code/processes/crawling-process/model/java/nu/marginalia/io/crawldata/CrawledDomainReader.java
# code/processes/crawling-process/test/nu/marginalia/crawling/HttpFetcherTest.java
# code/processes/crawling-process/test/nu/marginalia/crawling/retreival/CrawlerMockFetcherTest.java
# code/services-application/search-service/java/nu/marginalia/search/svc/SearchQueryIndexService.java
2024-09-08 10:14:43 +02:00
Viktor Lofgren
7a69dff6cf
(search) Correct handling of languages on fandom
2024-09-01 13:46:01 +02:00
Viktor Lofgren
bfb7ed2c99
(search) Translate cursed medium URLs to scribe.rip links via the search application
2024-09-01 13:32:14 +02:00
Viktor Lofgren
e19dc9b13e
(search) Translate cursed fandom URLs to breezewiki links via the search application
2024-09-01 13:23:35 +02:00
Viktor Lofgren
77efce0673
(paper-doll) Fix compilation
2024-08-26 12:51:29 +02:00
Viktor Lofgren
b09e2dbeb7
(build) Fix dependency churn from testcontainers
...
Apparently you need to pull in commons-codec now in order to run testcontainers, through spooky action at a distance.
2024-08-25 10:35:48 +02:00
Viktor Lofgren
5d2b455572
(search) Clean up inconsistent usage of MathClient in SearchOperator
...
Also clean up SearchOperator and adjacent code
2024-08-24 10:39:31 +02:00
Viktor Lofgren
ea75ddc0e0
(search) Absorb SearchQueryIndexService into SearchOperator, and clean up SearchOperator
2024-08-22 11:50:52 +02:00
Viktor Lofgren
2db0e446cb
(search) Absorb SearchQueryIndexService into SearchOperator, and clean up SearchOperator
2024-08-22 11:49:29 +02:00
Viktor Lofgren
557bdaa694
(search) Clean up SearchQueryIndexService and surrounding code
2024-08-22 11:45:28 +02:00
Viktor Lofgren
9eb1f120fc
(index) Repair positions bitmask for search result presentation
2024-08-22 11:28:23 +02:00
Viktor Lofgren
016a4c62e1
(index) Bugs and error fixes, chasing and fixing mystery results that did not contain all relevant keywords
2024-08-10 09:51:03 +02:00
Viktor Lofgren
285e657f68
Merge branch 'master' into term-positions
...
# Conflicts:
# code/processes/crawling-process/java/nu/marginalia/crawl/CrawlerMain.java
# code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
2024-07-31 10:44:01 +02:00
Viktor Lofgren
046ffc7752
(build) Upgrade jib to 3.4.3
2024-07-31 10:39:50 +02:00
Viktor Lofgren
f19148132a
(search) Restrict site-search by passing domain id along with the site:-term
...
This will help these queries deal with domains that do not have a subdomain so that they do not drag up subdomains as well, as they are also given the special site:-keyword for their corresponding parent domain.
2024-07-30 21:41:07 +02:00
Viktor Lofgren
aebb2652e8
(wip) Extract and encode spans data
...
Refactoring keyword extraction to extract spans information.
Modifying the intermediate storage of converted data to use the new slop library, which is allows for easier storage of ad-hoc binary data like spans and positions.
This is a bit of a katamari damacy commit that ended up dragging along a bunch of other fairly tangentially related changes that are hard to break out into separate commits after the fact. Will push as-is to get back to being able to do more isolated work.
2024-07-27 11:44:13 +02:00
Viktor Lofgren
dfd19b5eb9
(index) Reduce the number of abstractions around result ranking
...
The change also restructures the internal API a bit, moving resultsFromDomain from RpcRawResultItem into RpcDecoratedResultItem, as the previous order was driving complexity in the code that generates these objects, and the consumer side of things puts all this data in the same object regardless.
2024-07-16 08:18:54 +02:00
Viktor
8ed5b51a32
Merge branch 'master' into term-positions
2024-07-15 07:05:31 +02:00
Viktor Lofgren
801cf4b5da
(search) Fix bad practice usage of innerHTML to set what should be text content.
2024-06-12 08:59:40 +02:00
Viktor Lofgren
36160988e2
(index) Integrate positions data with indexes WIP
...
This change integrates the new positions data with the forward and reverse indexes.
The ranking code is still only partially re-written.
2024-06-10 15:09:06 +02:00
Viktor Lofgren
4a8afa6b9f
(index, WIP) Position data partially integrated with forward and reverse indexes.
...
There's no graceful way of doing this in small commits, pushing to avoid the risk of data loss.
2024-06-06 12:54:52 +02:00
Sam Storment
9c06f446fb
(search) Styling tweaks. Make the filter button near the top right corener a bit bigger so it's easier to press on mobile
2024-06-05 19:55:17 -05:00
Sam Storment
2d076cbd67
(search) move data-has-js attribute from body to html element
2024-06-05 18:20:33 -05:00
Sam Storment
fb2eef24d6
Handle themeing when javascript is disabled. Hide the theme select and fallback to dark media query instead of data-theme attribute
2024-06-03 14:15:35 -05:00
Sam Storment
e2f68d9ccf
Add a theme select to the header that lets users toggle their theme independent of their OS theme
2024-06-02 21:02:52 -05:00
Viktor
4435f6245c
Merge pull request #94 from samstorment/search-dark-theme
...
Search Dark Theme
2024-06-02 16:21:52 +02:00
Viktor Lofgren
0e8300979b
(search) Update the no result text to request bug reports.
2024-05-23 20:18:16 +02:00
Viktor Lofgren
89aae93e60
(*) Lift jetty and guava-dependencies
2024-05-23 14:20:01 +02:00
Sam Storment
5659df4388
(search) Set link and form field colors manually to override browser defaults with poor dark mode contrast
2024-05-21 00:03:46 -05:00
Sam Storment
43489c98d8
(search) Minor dark theme tweaks after the new mocked UI elements were added
2024-05-19 01:06:54 -05:00
Sam Storment
a7c33809c4
Merge branch 'master' into search-dark-theme
2024-05-17 22:52:19 -05:00
Viktor Lofgren
d227a09fb1
(search) Extend paperdoll service mock with site info data and screenshots
...
It's a bit of a hack job but will do, random exploration is available but only through a "browse:random"-style query
2024-05-15 12:40:55 +02:00
Viktor Lofgren
c3e3a3dbc5
(search) Fix problem list in clustered search results
2024-05-14 13:05:52 +02:00
Sam Storment
bb315221ab
(search, WIP) Make the dark theme look generally nicer. Rename CSS custom properties a bit. Switch a lot of background colors to HSL to make it easy to change colors relative to one another.
2024-05-14 01:32:40 -05:00
Sam Storment
c38766c5a6
(search, WIP) Convert SCSS variables to CSS custom properties for dynamic theming
2024-05-08 22:13:24 -05:00
Viktor Lofgren
c837321df1
(search) Provide a notification when no search results are found.
2024-05-06 20:11:39 +02:00
Viktor Lofgren
af7f6b89ec
(search) Delete vestigial stylesheet from the old design.
2024-05-06 19:52:29 +02:00
Viktor Lofgren
29a4d3df23
(search) Imrpove search-service paperdoll by mocking suggestions and news
2024-05-06 19:52:13 +02:00
Viktor Lofgren
5951c67a8b
(search) Center the search results page
2024-05-04 12:23:21 +02:00
Viktor Lofgren
c454007730
(search) Increase contrast for some UI elements
2024-05-04 12:02:52 +02:00
Viktor Lofgren
4e49cca43d
(search) Clean up SCSS code a bit
2024-05-04 11:58:54 +02:00
Viktor Lofgren
49a8c06095
(search) Improve contrast for text on random button
2024-05-04 11:51:19 +02:00
Viktor Lofgren
d01d9fa670
(search) Add screenreader-specific notification remark about when search results start.
2024-05-04 11:41:06 +02:00
Viktor Lofgren
a53a32f006
(search) Spell out website problems with "atomic elements" instead of having a hover that's inaccessible with keyboard navigation
2024-05-04 11:41:05 +02:00
Viktor Lofgren
3548d54cf6
(search) Add a screenreader-only alert when the search filters are updated to make it easier to understand what happens.
2024-05-04 11:41:04 +02:00
Viktor Lofgren
01f242ac7e
(search) Add stylesheet class for screenreader-only items
2024-05-04 11:41:03 +02:00
Viktor Lofgren
2840d9d403
(search) Add screenreader-only positions count text to search results
2024-05-04 11:41:03 +02:00
Viktor Lofgren
9fecfc5025
(search) Add autocomplete attribute to search-form
2024-05-04 11:41:02 +02:00
Viktor Lofgren
1b901e01f2
(search) Add bypass link that skips navigation
2024-05-04 11:41:01 +02:00
Viktor Lofgren
974aa35558
(search) Add proper alt-text to random exploration mode
2024-05-04 11:41:00 +02:00
Viktor Lofgren
4021a0ae98
(search) Add en-US language tags to all templates
2024-05-04 11:40:59 +02:00
Viktor Lofgren
b7a95be731
(search) Create a small mocking framework for running the search service in isolation.
2024-05-04 11:40:59 +02:00
Viktor Lofgren
2ad0bfda1e
(*) Fix boot orchestration for the services
...
This corrects an annoying bug that had the system crash and burn on first start-up due to a race condition in service initialization, where the services were attempting to access the database before it was properly migrated.
A fix was in principle already in place, but it was running too late and did not prevent attempts to access the as-yet uninitialized database. Move the first boot check into the MainClass instead of the Service constructor.
The change also adds more appropriate docker dependencies to the services to fix rare errors resolving the hostname of the database.
2024-05-01 12:39:48 +02:00
Viktor Lofgren
4772e0b59d
(service) Deprecate /public prefix on HTTP
...
Before the gRPC migration, the system would serve both public and internal requests over HTTP, but distinguish the two using path prefixes and a few HTTP Headers (X-Public, X-Context) added by the reverse proxy to prevent misconfigurations.
Since internal requests meaningfully no longer use HTTP, this convention is just an obstacle now, adding the need to always run the system behind a reverse proxy that rewrites the paths.
The change removes the path prefix, and updates the docker templates to reflect the change. This will require a migration for existing systems.
2024-04-30 14:46:18 +02:00
Viktor Lofgren
6690e9bde8
(service) Ensure the service discovery starts early
...
This is necessary as we use zookeeper to orchestrate first-time startup of the services, to ensure that the database is properly migrated by the control service before anything else is permitted to start.
2024-04-25 15:08:33 +02:00
Viktor Lofgren
32fe864a33
(build) Java 22 and its consequences has been a disaster for Marginalia Search
...
Roll back to JDK 21 for now, and make Java version configurable in the root build.gradle
The project has run into no less than three distinct show-stopping bugs in JDK22, across multiple vendors, and gradle still doesn't fully support it, meaning you need multiple JDK versions installed.
2024-04-24 14:44:39 +02:00
Viktor Lofgren
6efc0f21fe
(index) Clean up data model
...
The change set cleans up the data model for the term-level data. This used to contain a bunch of fields with document-level metadata. This data-duplication means a larger memory footprint and worse memory locality.
The ranking code is also modified to not accept SearchResultKeywordScores, but rather CompiledQueryLong and CqDataInts containing only the term metadata and the frequency information needed for ranking. This is again an effort to improve memory locality.
2024-04-24 14:44:39 +02:00
Viktor Lofgren
4fb86ac692
(search) Fix outdated assumptions about the results
...
We no longer break the query into "sets" of search terms and need to adapt the code to not use this assumption.
For the API service, we'll simulate the old behavior to keep the API stable.
For the search service, we'll introduce a new way of calculating positions through tree aggregation.
2024-04-24 14:44:38 +02:00
Viktor Lofgren
6cba6aef3b
(minor) Remove dead code
2024-04-24 14:44:38 +02:00
Viktor Lofgren
a3a6d6292b
(qs, index) New query model integrated with index service.
...
Seems to work, tests are green and initial testing finds no errors. Still a bit untested, committing WIP as-is because it would suck to lose weeks of work due to a drive failure or something.
2024-04-24 14:44:38 +02:00
Viktor Lofgren
3c75057dcd
(qs) Retire NGramBloomFilter, integrate new segmentation model instead
2024-04-24 14:44:17 +02:00
Viktor Lofgren
f434a8b492
(build) Upgrade jib plugin version
2024-04-16 15:25:23 +02:00
Joshua Holland
8e02f567d7
Update keywords docs use of explore to browse
...
I can't tell when this happened, but the proper keyword now seems to be browse and not explore.
2024-04-01 00:04:12 -05:00
Viktor Lofgren
fe8d583fdd
(sys) Upgrade to JDK22
...
This also entails upgrading JIB to 3.4.1 and Lombok to 1.18.32.
2024-03-21 14:27:13 +01:00
Viktor Lofgren
46423612e3
(refac) Merge service-discovery and service modules
...
Also adds a few tests to the server/client code.
2024-03-03 10:49:23 +01:00
Viktor Lofgren
d78e9e715f
(misc) Fix broken tests
2024-02-28 12:12:43 +01:00
Viktor Lofgren
20fc0ef13c
(gradle) Add task alias 'docker' for 'jibDockerBuild'
...
The change also moves the jib boilerplate to an include.
2024-02-28 11:59:15 +01:00
Viktor Lofgren
e696fd9e92
(docs) Begin un-fucking the docs after refactoring
2024-02-27 21:22:21 +01:00
Viktor Lofgren
8d0af9548b
(search) Bot mitigation
...
Add the ability to indicate to the search service that a request is malicious, and to poison the results by providing randomly reorered old results instead.
2024-02-27 21:22:19 +01:00
