Commit Graph

2499 Commits

Author SHA1 Message Date
Viktor
e5726a75d2
Merge pull request #120 from MarginaliaSearch/live-capture-function
Add a new function 'Live Capture' for on-demand screenshot capture
2024-09-27 13:48:53 +02:00
Viktor Lofgren
c757d116bf (misc) Fix Broken Tests 2024-09-27 13:46:34 +02:00
Viktor Lofgren
23cce0c78a Add a new function 'Live Capture' for on-demand screenshot capture
The screenshots are requested by the site-service, and triggered via the site-info view.
2024-09-27 13:46:34 +02:00
Viktor Lofgren
1bd29a586c (service-discovery) Add common base interface to all Grpc services
To be able to tell service discovery whether to enable a service on a particular runtime, a common base interface DiscoverableService extends BindableService was added.
2024-09-27 13:46:34 +02:00
Viktor Lofgren
4565bfe359 (crawler) Make the crawler report crawling progress correctly when stopped and resumed. 2024-09-26 18:30:29 +02:00
Viktor Lofgren
336d6fdd14 (index-client) Fix error when zero results are found 2024-09-25 20:23:13 +02:00
Viktor Lofgren
95cde242ca (assistant) Fix NPE when IP information is absent 2024-09-25 20:19:17 +02:00
Viktor
9224176202
Merge pull request #119 from MarginaliaSearch/result-pagination
Add pagination support for the search results
2024-09-25 14:29:24 +02:00
Viktor Lofgren
0d2390fd13 (search-service) Only autofocus on the query when the query is empty 2024-09-25 14:27:03 +02:00
Viktor Lofgren
4a0356e26f (search-service) Add pagination support to the search GUI 2024-09-25 14:26:49 +02:00
Viktor Lofgren
73f973cc06 (search-query) Add pagination to search query API and the direct query-service interface 2024-09-25 14:20:59 +02:00
Viktor Lofgren
e9e8580913 (converter) Fix NPE bugs in converter due to the reintroduction of CrawledDocument.headers 2024-09-25 12:18:56 +02:00
Viktor Lofgren
8b85a58fea (search UX) Autofocus on the search form 2024-09-24 15:56:03 +02:00
Viktor Lofgren
40512511af (crawler) Refactor boundary between CrawlerRetreiver and HttpFetcherImpl
This code is still a bit too complex, but it's slowly getting better.
2024-09-24 15:08:22 +02:00
Viktor
10d8fc4fe7
Update ROADMAP.md 2024-09-24 14:57:30 +02:00
Viktor
9899d45ea8
Merge pull request #118 from MarginaliaSearch/vlofgren-patch-1
Update ROADMAP.md
2024-09-24 14:13:47 +02:00
Viktor
3eea471ca6
Update ROADMAP.md 2024-09-24 14:13:32 +02:00
Viktor Lofgren
3dec4b6b34 (index) Fix bug where tcfFirstPosition lit up because one term was in the title and the other was missing from the document
This was because firstPosition calculation was not invalidated when positions were missing.
2024-09-24 13:33:37 +02:00
Viktor Lofgren
162fc25ebc (minor) Fix accidental commit errors 2024-09-23 18:03:09 +02:00
Viktor Lofgren
e9854f194c (crawler) Refactor
* Restructure the code to make a bit more sense
* Store full headers in crawl data
* Fix bug in retry-after header that assumed the timeout was in milliseconds, and then clamped it to a lower bound of 500ms, meaning this was almost always handled wrong
2024-09-23 17:51:07 +02:00
Viktor Lofgren
9c292a4f62 (doc) Fix outdated links in documentation 2024-09-22 13:56:17 +02:00
Viktor Lofgren
edb42836da (vcs) Fix shared state issues with VarintCodedSequence's iterators.
Also cleans up the code a bit.
2024-09-21 16:09:15 +02:00
Viktor Lofgren
1ff88ff0bc (vcs) Stopgap fix for quoted queries with the same term appearinc multiple times
There are reentrance issues with VarintCodedSequence, this hides the symptom but these need to be corrected properly.
2024-09-21 14:07:59 +02:00
Viktor Lofgren
28e7c8e5e0 Increase temporal bias weight to give the recent results filter a bit more recency 2024-09-17 18:11:40 +02:00
Viktor
463b3ed0ce
Merge pull request #99 from MarginaliaSearch/term-positions
Improve term positions accuracy
2024-09-17 15:30:04 +02:00
Viktor Lofgren
8e78286068 Merge branch 'master' into term-positions 2024-09-17 15:20:46 +02:00
Viktor Lofgren
f4eeef145e (index) Reduce fetch size to improve timeout characteristics 2024-09-17 15:20:41 +02:00
Viktor Lofgren
87aa869338 (index) Correct positions mask to take into account offsets when overlapping 2024-09-17 14:40:37 +02:00
Viktor Lofgren
60ad4786bc (index) Use MemorySegment.copy for LongArray->LongArray transfers 2024-09-17 13:56:31 +02:00
Viktor Lofgren
a74df7f905 (index) Increase buffer size for PrioDocIdsTransformer 2024-09-17 13:52:52 +02:00
Viktor Lofgren
9f9c6736ab (index) Use MemorySegment.copy for LongArray->LongArray transfers 2024-09-17 13:49:02 +02:00
Viktor Lofgren
b95646625f (index) Correct prio index construction with mmap
Accidentally snuck in behavior from full index
2024-09-17 13:39:08 +02:00
Viktor Lofgren
6e47eae903 (index) Correct strange close handling of PositionsFileConstructor 2024-09-13 16:34:14 +02:00
Viktor Lofgren
934af0dd4b (index) Correct units in log message when shrinking the documents file 2024-09-13 16:33:19 +02:00
Viktor Lofgren
a8bec13ed9 (index) Evaluate using mmap reads during index construction in favor of filechannel reads
It's likely that this will be faster, as the reads are on average small and sequential, and can't be buffered easily.
2024-09-13 16:14:56 +02:00
Viktor Lofgren
1cf62f5850 (doc) Correct dead links and stale information in the docs 2024-09-13 11:02:13 +02:00
Viktor Lofgren
8047e77757 (doc) Correct dead links and stale information in the docs 2024-09-13 11:01:05 +02:00
Viktor Lofgren
2a92de29ce (loader) Fix it so that the loader doesn't explode if it sees an invalid URL 2024-09-12 11:36:00 +02:00
Viktor Lofgren
99523ca079 (query-parser) Remove test that is no longer relevant 2024-09-10 10:35:56 +02:00
Viktor Lofgren
35f49bbb60 (coded-sequence) Add equals and hashCode to VCS 2024-09-10 10:33:56 +02:00
Viktor Lofgren
50ec922c2b (index) Fix broken index tests
Also cleaned up the tests to be less fragile to ranking algorithm changes.
2024-09-10 10:23:46 +02:00
Viktor Lofgren
cfbbeaa26e (ranking) Clean up ranking test code 2024-09-08 15:46:51 +02:00
Viktor Lofgren
a3b0189934 Fix build errors after merge 2024-09-08 10:22:32 +02:00
Viktor Lofgren
8f367d96f8 Merge branch 'master' into term-positions
# Conflicts:
#	code/index/java/nu/marginalia/index/results/model/ids/TermIdList.java
#	code/processes/converting-process/java/nu/marginalia/converting/ConverterMain.java
#	code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
#	code/processes/crawling-process/java/nu/marginalia/crawl/retreival/fetcher/HttpFetcherImpl.java
#	code/processes/crawling-process/model/java/nu/marginalia/io/crawldata/CrawledDomainReader.java
#	code/processes/crawling-process/test/nu/marginalia/crawling/HttpFetcherTest.java
#	code/processes/crawling-process/test/nu/marginalia/crawling/retreival/CrawlerMockFetcherTest.java
#	code/services-application/search-service/java/nu/marginalia/search/svc/SearchQueryIndexService.java
2024-09-08 10:14:43 +02:00
Viktor Lofgren
f78ef36cd4 (slop) Upgrade to 0.0.8, add encodings to string columns. 2024-09-04 15:19:00 +02:00
Viktor Lofgren
dc67c81f99 (summary) Fix a few cases where noscript tags would sometimes be used for document summary 2024-09-04 15:00:40 +02:00
Viktor Lofgren
50ba8fd099 (query-parsing) Correct handling of trailing parentheses 2024-09-03 11:45:14 +02:00
Viktor Lofgren
99b3b00b68 (query-parsing) Merge QueryTokenizer into QueryParser and add escaping of query grammar 2024-09-03 11:35:32 +02:00
Viktor Lofgren
f6d981761d (query-parsing) Drop search term elements that aren't indexed by the search engine 2024-09-03 11:24:05 +02:00
Viktor Lofgren
8290c19e24 (query-parsing) Drop search term elements that aren't indexed by the search engine 2024-09-03 11:21:01 +02:00