Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(link-parser) Filter out URLs with binary file suffixes in LinkParser

Browse Source 
Added an additional filter step to ensure URLs with binary suffixes are excluded during crawling. This prevents unnecessary processing of non-HTML content, improving the efficiency of the link parsing process.
This commit is contained in:
Viktor Lofgren 2024-12-11 16:42:47 +01:00
parent a97c05107e
commit 3b99cffb3d
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
code/processes/crawling-process/ft-link-parser/java/nu/marginalia/link_parser/LinkParser.java
View File

@ -42,7 +42,8 @@ public class LinkParser {
.flatMap(this::createURI) .flatMap(this::createURI)
.map(URI::normalize) .map(URI::normalize)
.map(this::renormalize) .map(this::renormalize)
.flatMap(this::createEdgeUrl); .flatMap(this::createEdgeUrl)
.filter(url -> !hasBinarySuffix(url.path));
} }
@Contract(pure=true) @Contract(pure=true)