MarginaliaSearch/code/functions/live-capture
Viktor Lofgren 3bc99639a0 (feed-fetcher) Make feed fetcher requests conditional
Add `If-None-Match` and `If-Modified-Since` headers as appropriate to the feed fetcher's requests.  On well-configured web servers, this should short-circuit the request and reduce the amount of bandwidth and processing that is necessary.

A new table was added to the FeedDb to hold one etag per domain.

If-Modified-Since semantics are based on the creation date for the feed database, which should serve as a cutoff date for the earliest update we can have received.

This completes the changes for Issue #136.
2024-12-27 15:10:15 +01:00
..
api (live-crawler) Add refresh date to feeds API 2024-12-25 14:20:48 +01:00
java/nu/marginalia (feed-fetcher) Make feed fetcher requests conditional 2024-12-27 15:10:15 +01:00
test/nu/marginalia (feed-fetcher) Make feed fetcher requests conditional 2024-12-27 15:10:15 +01:00
build.gradle (live-crawler) Add Accept-Encoding: gzip to outbound requests 2024-12-27 03:59:34 +01:00
readme.md (live-capture) Add readme to live-capture function 2024-09-28 11:35:46 +02:00

This is a subsystem that allows on-demand screenshot capture of a website.

It uses the local browserless API to capture data. To use this module, you must have a browserless docker container running on machine, and then set the live-capture.browserless-uri system property to the address of the browserless container (e.g http://my-container:3000/).

When disabled, the subsystem will acknowledge the request, but will not act on it.

The module will only enable on the primary node of a service to simplify dealing with race conditions and duplicate requests.