Viktor Lofgren
df89661ed2
(index) In SearchResultItem, populate combinedId with combinedId and not its ranking-removed documentId cousin
2024-08-09 16:32:32 +02:00
Viktor Lofgren
2e89b55593
(wip) Repair qdebug utility and show new ranking details
2024-08-09 12:57:25 +02:00
Viktor Lofgren
7babdb87d5
(index) Remove intermediate models
2024-08-07 10:10:44 +02:00
Viktor Lofgren
df6a05b9a7
(index) Avoid hypothetical divide-by-zero in tcfAvgDist
2024-08-06 10:55:57 +02:00
Viktor Lofgren
8569bb8e11
(index) Avoid divide-by-zero when minDist returns 0
2024-08-06 10:34:05 +02:00
Viktor Lofgren
ca6e2db2b9
(index) Include external link texts in verbatim score
2024-08-06 10:23:23 +02:00
Viktor Lofgren
ee49c01d86
(index) Tune ranking for verbatim matches in the title, rewarding shorter titles
2024-08-03 14:47:23 +02:00
Viktor Lofgren
b21f8538a8
(index) Tune ranking for verbatim matches in the title, rewarding shorter titles
2024-08-03 14:41:38 +02:00
Viktor Lofgren
dd15676d33
(index) Tune ranking for verbatim matches in the title, rewarding shorter titles
2024-08-03 14:18:04 +02:00
Viktor Lofgren
ec5a17ad13
(index) Tune ranking for verbatim matches in the title, rewarding shorter titles
2024-08-03 14:07:02 +02:00
Viktor Lofgren
8462e88b8f
(index) Add min-dist factor and adjust rankings
2024-08-03 13:07:00 +02:00
Viktor Lofgren
bf26ead010
(index) Remove hasPrioTerm check as we should sort this out in ranking
2024-08-03 13:06:50 +02:00
Viktor Lofgren
c2cedfa83c
(index) Experimental ranking signals
2024-08-03 10:33:41 +02:00
Viktor Lofgren
c6c8b059bf
(index) Return some variant of the previously removed 'Bm25PrioGraphVisitor'
2024-08-03 10:10:12 +02:00
Viktor Lofgren
d8a99784e5
(index) Adding a few experimental relevance signals
2024-08-02 20:26:07 +02:00
Viktor Lofgren
e2107901ec
(index) Add span information for anchor tags, tweak ranking params
2024-08-01 11:46:30 +02:00
Viktor Lofgren
15745b692e
(index) Coherences need to be able to deal with null values among positions
2024-07-31 22:00:14 +02:00
Viktor Lofgren
b316b55be9
(index) Experimental initial integration of document spans into index
2024-07-30 12:01:53 +02:00
Viktor Lofgren
aebb2652e8
(wip) Extract and encode spans data
...
Refactoring keyword extraction to extract spans information.
Modifying the intermediate storage of converted data to use the new slop library, which is allows for easier storage of ad-hoc binary data like spans and positions.
This is a bit of a katamari damacy commit that ended up dragging along a bunch of other fairly tangentially related changes that are hard to break out into separate commits after the fact. Will push as-is to get back to being able to do more isolated work.
2024-07-27 11:44:13 +02:00
Viktor Lofgren
0b31c4cfbb
(coded-sequence) Replace GCS usage with an interface
2024-07-16 14:37:50 +02:00
Viktor Lofgren
dfd19b5eb9
(index) Reduce the number of abstractions around result ranking
...
The change also restructures the internal API a bit, moving resultsFromDomain from RpcRawResultItem into RpcDecoratedResultItem, as the previous order was driving complexity in the code that generates these objects, and the consumer side of things puts all this data in the same object regardless.
2024-07-16 08:18:54 +02:00
Viktor Lofgren
ad3857938d
(search-api, ranking) Update with new ranking parameters
...
Adding new ranking parameters to the API and routing them through the system, in order to permit integration of the new position data with the ranking algorithm.
The change also cleans out several parameters that no longer filled any function.
2024-07-15 04:49:40 +02:00
Viktor Lofgren
fa36689597
(index-reverse) Simplify priority index
...
* Do not emit a documents file
* Do not interlace metadata or offsets with doc ids
2024-07-06 18:04:08 +02:00
Viktor Lofgren
0e4dd3d76d
(minor) Remove accidentally committed debug printf
2024-06-27 13:40:53 +02:00
Viktor Lofgren
95b9af92a0
(index) Implement working optional TermCoherences
2024-06-26 12:22:06 +02:00
Viktor Lofgren
8ee64c0771
(index) Correct TermCoherence requirements
2024-06-25 22:18:10 +02:00
Viktor Lofgren
dae22ccbe0
(test) Integration test from crawl->query
2024-06-25 22:17:26 +02:00
Viktor Lofgren
9d00243d7f
(index) Partial re-implementation of position constraints
2024-06-24 15:55:54 +02:00
Viktor Lofgren
36160988e2
(index) Integrate positions data with indexes WIP
...
This change integrates the new positions data with the forward and reverse indexes.
The ranking code is still only partially re-written.
2024-06-10 15:09:06 +02:00
Viktor Lofgren
4fcd4a8197
(index) Refactor to reduce the level of indirection
2024-05-19 12:40:33 +02:00
Viktor Lofgren
19163fa883
(array) Clean up the Array library
...
IntArray gets the YAGNI axe. The array library had two implementations, one for longs which was used, and one for ints, which only ever saw bit rot. Removing the latter, as all it ever did was clutter up the codebase and add technical debt. If we need int arrays, we fork LongArray again (or add int capabilities to it)
Also cleaning up the interfaces, removing layers of redundant abstractions and adding javadocs.
Finally adding sz=2 specializations to the quick- and insertion sort algorithms. It seems the JIT isn't optimizing these particularly well, this is an attempt to help it out a bit.
2024-05-18 13:23:06 +02:00
Viktor
2d49071e96
Merge branch 'master' into run-outside-docker
2024-04-25 18:53:26 +02:00
Viktor Lofgren
e4b34b6ee6
(index) Correctly detect the presence of an all-virtual path through the query
2024-04-25 14:01:46 +02:00
Viktor Lofgren
f46733a47a
(ranking) TermCoherenceFactory should be run for size=2 queries
2024-04-24 14:44:39 +02:00
Viktor Lofgren
5165cf6d15
(ranking) Set regularMask correctly
2024-04-24 14:44:39 +02:00
Viktor Lofgren
4489b21528
(ranking) Cleanup
2024-04-24 14:44:39 +02:00
Viktor Lofgren
f623b37577
(ranking) Suppress NaN:s in ranking output
2024-04-24 14:44:39 +02:00
Viktor Lofgren
f4a2fea451
(ranking, bugfix) Use bm25NgramWeight and not full weight for bM25N
2024-04-24 14:44:39 +02:00
Viktor Lofgren
a748fc5448
(index, bugfix) Pass url quality to query service
2024-04-24 14:44:39 +02:00
Viktor Lofgren
0dcca0cb83
(index) Fix TCF bug where the ngram terms would be considered instead of the regular ones due to a logical derp
2024-04-24 14:44:39 +02:00
Viktor Lofgren
b80a83339b
(qs) Additional info in query debug UI
2024-04-24 14:44:39 +02:00
Viktor Lofgren
eb74d08f2a
(qs) Additional info in query debug UI
2024-04-24 14:44:39 +02:00
Viktor Lofgren
462aa9af26
(query) Update ranking parameters with new variables for bm25 ngrams and tcf mutual jaccard
...
The change also makes it so that as long as the values are defaults, they don't need to be sent over the wire and decoded.
2024-04-24 14:44:39 +02:00
Viktor Lofgren
44b33798f3
(index) Clean up jaccard index term code and down-tune the parameter's importance a bit
2024-04-24 14:44:39 +02:00
Viktor Lofgren
2f0b648fad
(index) Add jaccard index term to boost results based on term overlap
2024-04-24 14:44:39 +02:00
Viktor Lofgren
de0e56f027
(index) Remove position overlap check, coherences will do the work instead
2024-04-24 14:44:39 +02:00
Viktor Lofgren
973ced7b13
(index) Omit absent terms from coherence checks
2024-04-24 14:44:39 +02:00
Viktor Lofgren
cb4b824a85
(index) Split ngram and regular keyword bm25 calculation and add ngram score as a bonus
2024-04-24 14:44:39 +02:00
Viktor Lofgren
e0224085b4
(index) Improve recall for small queries
...
Partially reverse the previous commit and add a query head for the priority index when there are few query interpretations.
2024-04-24 14:44:39 +02:00
Viktor Lofgren
44c1e1d6d9
(index) Remove dead code
...
Since the performance fix in 3359f72239
2024-04-24 14:44:39 +02:00
