Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 21:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(converter) Stopgap fix for some cases of lost crawl data due to HTTP 304. The root cause needs further investigation.

Browse Source
This commit is contained in:
Viktor Lofgren 2024-04-19 20:36:01 +02:00
parent 2cc74c005a
commit 214551f1df
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
code/processes/converting-process/java/nu/marginalia/converting/processor/DomainProcessor.java
View File

@ -193,6 +193,8 @@ public class DomainProcessor {
continue;
if (doc.url == null)
continue;
if (doc.documentBody.isBlank())
continue;
if (!processedUrls.add(doc.url))
continue;