Viktor
9899d45ea8
Merge pull request #118 from MarginaliaSearch/vlofgren-patch-1
...
Update ROADMAP.md
2024-09-24 14:13:47 +02:00
Viktor
3eea471ca6
Update ROADMAP.md
2024-09-24 14:13:32 +02:00
Viktor Lofgren
3dec4b6b34
(index) Fix bug where tcfFirstPosition lit up because one term was in the title and the other was missing from the document
...
This was because firstPosition calculation was not invalidated when positions were missing.
2024-09-24 13:33:37 +02:00
Viktor Lofgren
162fc25ebc
(minor) Fix accidental commit errors
2024-09-23 18:03:09 +02:00
Viktor Lofgren
e9854f194c
(crawler) Refactor
...
* Restructure the code to make a bit more sense
* Store full headers in crawl data
* Fix bug in retry-after header that assumed the timeout was in milliseconds, and then clamped it to a lower bound of 500ms, meaning this was almost always handled wrong
2024-09-23 17:51:07 +02:00
Viktor Lofgren
9c292a4f62
(doc) Fix outdated links in documentation
2024-09-22 13:56:17 +02:00
Viktor Lofgren
edb42836da
(vcs) Fix shared state issues with VarintCodedSequence's iterators.
...
Also cleans up the code a bit.
2024-09-21 16:09:15 +02:00
Viktor Lofgren
1ff88ff0bc
(vcs) Stopgap fix for quoted queries with the same term appearinc multiple times
...
There are reentrance issues with VarintCodedSequence, this hides the symptom but these need to be corrected properly.
2024-09-21 14:07:59 +02:00
Viktor Lofgren
28e7c8e5e0
Increase temporal bias weight to give the recent results filter a bit more recency
2024-09-17 18:11:40 +02:00
Viktor
463b3ed0ce
Merge pull request #99 from MarginaliaSearch/term-positions
...
Improve term positions accuracy
2024-09-17 15:30:04 +02:00
Viktor Lofgren
8e78286068
Merge branch 'master' into term-positions
2024-09-17 15:20:46 +02:00
Viktor Lofgren
f4eeef145e
(index) Reduce fetch size to improve timeout characteristics
2024-09-17 15:20:41 +02:00
Viktor Lofgren
87aa869338
(index) Correct positions mask to take into account offsets when overlapping
2024-09-17 14:40:37 +02:00
Viktor Lofgren
60ad4786bc
(index) Use MemorySegment.copy for LongArray->LongArray transfers
2024-09-17 13:56:31 +02:00
Viktor Lofgren
a74df7f905
(index) Increase buffer size for PrioDocIdsTransformer
2024-09-17 13:52:52 +02:00
Viktor Lofgren
9f9c6736ab
(index) Use MemorySegment.copy for LongArray->LongArray transfers
2024-09-17 13:49:02 +02:00
Viktor Lofgren
b95646625f
(index) Correct prio index construction with mmap
...
Accidentally snuck in behavior from full index
2024-09-17 13:39:08 +02:00
Viktor Lofgren
6e47eae903
(index) Correct strange close handling of PositionsFileConstructor
2024-09-13 16:34:14 +02:00
Viktor Lofgren
934af0dd4b
(index) Correct units in log message when shrinking the documents file
2024-09-13 16:33:19 +02:00
Viktor Lofgren
a8bec13ed9
(index) Evaluate using mmap reads during index construction in favor of filechannel reads
...
It's likely that this will be faster, as the reads are on average small and sequential, and can't be buffered easily.
2024-09-13 16:14:56 +02:00
Viktor Lofgren
1cf62f5850
(doc) Correct dead links and stale information in the docs
2024-09-13 11:02:13 +02:00
Viktor Lofgren
8047e77757
(doc) Correct dead links and stale information in the docs
2024-09-13 11:01:05 +02:00
Viktor Lofgren
2a92de29ce
(loader) Fix it so that the loader doesn't explode if it sees an invalid URL
2024-09-12 11:36:00 +02:00
Viktor Lofgren
99523ca079
(query-parser) Remove test that is no longer relevant
2024-09-10 10:35:56 +02:00
Viktor Lofgren
35f49bbb60
(coded-sequence) Add equals and hashCode to VCS
2024-09-10 10:33:56 +02:00
Viktor Lofgren
50ec922c2b
(index) Fix broken index tests
...
Also cleaned up the tests to be less fragile to ranking algorithm changes.
2024-09-10 10:23:46 +02:00
Viktor Lofgren
cfbbeaa26e
(ranking) Clean up ranking test code
2024-09-08 15:46:51 +02:00
Viktor Lofgren
a3b0189934
Fix build errors after merge
2024-09-08 10:22:32 +02:00
Viktor Lofgren
8f367d96f8
Merge branch 'master' into term-positions
...
# Conflicts:
# code/index/java/nu/marginalia/index/results/model/ids/TermIdList.java
# code/processes/converting-process/java/nu/marginalia/converting/ConverterMain.java
# code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
# code/processes/crawling-process/java/nu/marginalia/crawl/retreival/fetcher/HttpFetcherImpl.java
# code/processes/crawling-process/model/java/nu/marginalia/io/crawldata/CrawledDomainReader.java
# code/processes/crawling-process/test/nu/marginalia/crawling/HttpFetcherTest.java
# code/processes/crawling-process/test/nu/marginalia/crawling/retreival/CrawlerMockFetcherTest.java
# code/services-application/search-service/java/nu/marginalia/search/svc/SearchQueryIndexService.java
2024-09-08 10:14:43 +02:00
Viktor Lofgren
f78ef36cd4
(slop) Upgrade to 0.0.8, add encodings to string columns.
2024-09-04 15:19:00 +02:00
Viktor Lofgren
dc67c81f99
(summary) Fix a few cases where noscript tags would sometimes be used for document summary
2024-09-04 15:00:40 +02:00
Viktor Lofgren
50ba8fd099
(query-parsing) Correct handling of trailing parentheses
2024-09-03 11:45:14 +02:00
Viktor Lofgren
99b3b00b68
(query-parsing) Merge QueryTokenizer into QueryParser and add escaping of query grammar
2024-09-03 11:35:32 +02:00
Viktor Lofgren
f6d981761d
(query-parsing) Drop search term elements that aren't indexed by the search engine
2024-09-03 11:24:05 +02:00
Viktor Lofgren
8290c19e24
(query-parsing) Drop search term elements that aren't indexed by the search engine
2024-09-03 11:21:01 +02:00
Viktor Lofgren
7a69dff6cf
(search) Correct handling of languages on fandom
2024-09-01 13:46:01 +02:00
Viktor Lofgren
bfb7ed2c99
(search) Translate cursed medium URLs to scribe.rip links via the search application
2024-09-01 13:32:14 +02:00
Viktor Lofgren
e19dc9b13e
(search) Translate cursed fandom URLs to breezewiki links via the search application
2024-09-01 13:23:35 +02:00
Viktor Lofgren
74148c790e
(crawler) Pull additional new domains from node-affinity 0
...
Previously a bit ambiguously defined, node affinity 0 is now indicative that a domain is up for grabs for the next crawler
2024-09-01 13:00:36 +02:00
Viktor Lofgren
3d77456110
(*) Add domain parking service to ip blocklist
2024-09-01 12:53:22 +02:00
Viktor Lofgren
ab6a4b1749
(control) Correct id value for domain addition tool
2024-09-01 12:25:15 +02:00
Viktor Lofgren
aeeb1d0cb7
(control) Add utility for adding domains from an external URL
2024-09-01 12:14:21 +02:00
Viktor Lofgren
185b79f2a5
(converter) Fix bug where sideloaded reddit content was errouneously categoriszed as wiki-generated.
2024-09-01 11:30:25 +02:00
Viktor Lofgren
8d0f9652c7
(crawler) Correct RSS-sitemap behavior
2024-08-31 11:38:34 +02:00
Viktor Lofgren
5353805cc6
(crawler) Correct RSS-sitemap behavior
2024-08-31 11:37:09 +02:00
Viktor Lofgren
5407da5650
(crawler) Grab favicons as part of root sniff
2024-08-31 11:32:56 +02:00
Viktor Lofgren
b1bfe6f76e
(control) New view for domains
...
Add capability to assign domains, and bulk-add new domains.
2024-08-30 17:06:48 +02:00
Viktor Lofgren
74e25370ca
(control) New view for domains
...
Still a work in progress, but at this point it's possible to use for viewing domains
2024-08-29 15:40:40 +02:00
Viktor Lofgren
bb5d946c26
(index, EXPERIMENTAL) Clean up ranking code
2024-08-29 11:34:23 +02:00
Viktor Lofgren
abab5bdc8a
(index, EXPERIMENTAL) Evaluate using Varint instead of GCS for position data
2024-08-26 14:20:39 +02:00
