Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-24 05:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(crawler) Be a bit more conservative about pulling etags and so on if the previous fetch wasn't OK

Browse Source
This commit is contained in:
Viktor Lofgren 2023-12-25 00:55:05 +01:00
parent c5aab7e8db
commit 84563b0d46
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
code/processes/crawling-process/src/main/java/nu/marginalia/crawl/retreival/revisit/DocumentWithReference.java
View File

@ -49,6 +49,9 @@ public record DocumentWithReference(
if (null == doc) if (null == doc)
return ContentTags.empty(); return ContentTags.empty();
if (doc.documentBody == null || doc.httpStatus != 200)
return ContentTags.empty();
String lastmod = doc.getLastModified(); String lastmod = doc.getLastModified();
String etag = doc.getEtag(); String etag = doc.getEtag();