MarginaliaSearch/code/processes/crawling-process/readme.md

# Crawling Process

The crawling process downloads HTML and saves them
into per-domain snapshots.

## Central Classes

* [CrawlerMain](src/main/java/nu/marginalia/crawl/CrawlerMain.java) orchestrates the crawling.
* [CrawlerRetreiver](src/main/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java)
  visits known addresses from a domain and downloads each document.
* [HttpFetcher](src/main/java/nu/marginalia/crawl/retreival/HttpFetcher.java)
  fetches a URL.

## See Also

* [features-convert](../../features-convert/)
More readmes, clean-up of associated code. 2023-03-04 15:42:31 +00:00			`# Crawling Process`

			`The crawling process downloads HTML and saves them`
			`into per-domain snapshots.`

			`## Central Classes`

			`* [CrawlerMain](src/main/java/nu/marginalia/crawl/CrawlerMain.java) orchestrates the crawling.`
			`* [CrawlerRetreiver](src/main/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java)`
			`visits known addresses from a domain and downloads each document.`
			`* [HttpFetcher](src/main/java/nu/marginalia/crawl/retreival/HttpFetcher.java)`
			`fetches a URL.`
More restructuring, big bug fixes in keyword extraction. 2023-03-13 16:39:53 +00:00
			`## See Also`

			`* [features-convert](../../features-convert/)`