mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 21:18:58 +00:00
8 lines
341 B
Markdown
8 lines
341 B
Markdown
# Crawl Features
|
|
|
|
These are bits of search-engine related code that are relatively isolated pieces of business logic,
|
|
that benefit from the clarity of being kept separate from the rest of the crawling code.
|
|
|
|
* [crawl-blocklist](crawl-blocklist/) - IP and URL blocklists
|
|
* [link-parser](link-parser/) - Code for parsing and normalizing links
|