5 Commits

Author	SHA1	Message	Date
Viktor Lofgren	89d8af640d	(live-crawl) Rename the live crawler code module to be more consistent with the other processes	2024-11-20 15:55:15 +01:00
Viktor Lofgren	6e4252cf4c	(live-crawl) Make the actor poll for feeds changes instead of being a one-shot thing. ... Also changes the live crawl process to store the live crawl data in a fixed directory in the storage base rather than versioned directories.	2024-11-20 15:36:25 +01:00
Viktor Lofgren	79ce4de2ab	(model) Remove deprecated fields from CrawledDocument and CrawledDomain	2024-11-20 15:27:05 +01:00
Viktor Lofgren	d6575dfee4	(live-crawler) Crude first-try process for live crawling #WIP ... Some refactoring is still needed, but an dummy actor is in place and a process that crawls URLs from the livecapture service's RSS endpoints; that makes it all the way to being indexable.	2024-11-19 21:00:18 +01:00
Viktor Lofgren	a91ab4c203	(live-crawler) Crude first-try process for live crawling #WIP ... Some refactoring is still needed, but an dummy actor is in place and a process that crawls URLs from the livecapture service's RSS endpoints; that makes it all the way to being indexable.	2024-11-19 19:35:01 +01:00