Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 13:09:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Revert "(experiment) Modify atags exporter to permit duplicates from different source domains"

Browse Source 
This reverts commit 5c858a2b94.
This commit is contained in:
Viktor Lofgren 2024-12-07 14:01:50 +01:00
parent 5c858a2b94
commit ee2d5496d0
1 changed files with 6 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
code/processes/export-task-process/java/nu/marginalia/extractor/AtagExporter.java
View File

@ -155,9 +155,12 @@ public class AtagExporter implements ExporterIf {
}
// Deduplicate by hash; we've already checked that the strings are ASCII printable so we don't
// need to be concerned about using the fast ASCII hash. Note we don't consider the destination URL
// here, but the source domain instead.
return !hashes.add(hash.hashLowerBytes(linkText) ^ hash.hashLowerBytes(baseUrl.domain.toString()));
// need to be concerned about using the fast ASCII hash
if (hashes.add(hash.hashLowerBytes(linkText) ^ hash.hashLowerBytes(urlString))) {
return false;
}
return true;
}
}