mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-24 13:19:02 +00:00
0152004c42
* Added new (optional) model file in $WMSA_HOME/data/atags.parquet * Converter gets a component for creating a projection of its domains onto the full atags parquet file * New WordFlag ExternalLink * These terms are also for now flagged as title words * Fixed a bug where Title words aliased with UrlDomain words * Fixed a bug in the encyclopedia sideloader that gave everything too high topology ranking |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
Converting Process
The converting process reads crawl data and extracts information to be fed into the index, such as keywords, metadata, urls, descriptions...
Central Classes
- ConverterMain orchestrates the conversion process.
- DocumentProcessor converts a single document.
-
- HtmlDocumentProcessorPlugin has HTML-specific logic related to a document, keywords and identifies features such as whether it has javascript.
-
- PlainTextDocumentProcessorPlugin has plain text-specific logic related to a document...
- DomainProcessor converts each document and generates domain-wide metadata such as link graphs.