MarginaliaSearch/code/features-crawl/readme.md

# Crawl Features

These are bits of search-engine related code that are relatively isolated pieces of business logic,
that benefit from the clarity of being kept separate from the rest of the crawling code.

* [content-type](content-type/) - Content Type identification
* [crawl-blocklist](crawl-blocklist/) - IP and URL blocklists
* [link-parser](link-parser/) - Code for parsing and normalizing links
The refactoring will continue until morale improves. 2023-03-12 09:50:31 +00:00			`# Crawl Features`

			`These are bits of search-engine related code that are relatively isolated pieces of business logic,`
			`that benefit from the clarity of being kept separate from the rest of the crawling code.`

(doc) Update docs 2024-02-06 15:29:55 +00:00			`* [content-type](content-type/) - Content Type identification`
The refactoring will continue until morale improves. 2023-03-12 10:42:07 +00:00			`* [crawl-blocklist](crawl-blocklist/) - IP and URL blocklists`
			`* [link-parser](link-parser/) - Code for parsing and normalizing links`