Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(converter) Stopgap fix for some cases of lost crawl data due to HTTP 304. The root cause needs further investigation.

Browse Source
This commit is contained in:
Viktor Lofgren 2024-04-19 20:36:01 +02:00
parent 64baa41e64
commit 934167323d
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
code/processes/converting-process/java/nu/marginalia/converting/processor/DomainProcessor.java
View File

@ -193,6 +193,8 @@ public class DomainProcessor {
continue; continue;
if (doc.url == null) if (doc.url == null)
continue; continue;
if (doc.documentBody.isBlank())
continue;
if (!processedUrls.add(doc.url)) if (!processedUrls.add(doc.url))
continue; continue;