3 Commits

Author	SHA1	Message	Date
Viktor Lofgren	3d68be64da	(crawler) Add default CT when it's missing for icons	2025-01-22 13:55:47 +01:00
Viktor Lofgren	98a340a0d1	(crawler) Add favicon data to domain state db in its own table	2025-01-22 11:41:20 +01:00
Viktor Lofgren	895cee7004	(crawler) Improved feed discovery, new domain state db per crawlset ... Feed discover is improved with by probing a few likely endpoints when no feed link tag is provided. To store the feed URLs, a sqlite database is added to each crawlset that stores a simple summary of the crawl job, including any feed URLs that have been discovered. Solves issue #135	2024-12-26 15:05:52 +01:00