Mirror/MarginaliaSearch
1
0
Fork 0
mirror of https://github.com/MarginaliaSearch/MarginaliaSearch.git synced 2025-02-23 13:09:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

(crawler) Correct feed URLs in domain state db

Browse Source 
Discovered feed URLs were given a double slash after their domain name in the DB.  This will go away in the URL normalizer, so the URLs are still viable, but the commit fixes the issue regardless.
This commit is contained in:
Viktor Lofgren 2024-12-26 15:18:31 +01:00
parent 895cee7004
commit 89db69d360
1 changed files with 10 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
code/processes/crawling-process/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java
View File

@ -297,16 +297,16 @@ public class CrawlerRetreiver implements AutoCloseable {
} }
private final List<String> likelyFeedEndpoints = List.of( private final List<String> likelyFeedEndpoints = List.of(
"/rss.xml", "rss.xml",
"/atom.xml", "atom.xml",
"/feed.xml", "feed.xml",
"/index.xml", "index.xml",
"/feed", "feed",
"/rss", "rss",
"/atom", "atom",
"/feeds", "feeds",
"/blog/feed", "blog/feed",
"/blog/rss" "blog/rss"
); );
private Optional<String> guessFeedUrl(CrawlDelayTimer timer) throws InterruptedException { private Optional<String> guessFeedUrl(CrawlDelayTimer timer) throws InterruptedException {