mirror of
https://github.com/MarginaliaSearch/MarginaliaSearch.git
synced 2025-02-23 21:18:58 +00:00
f655ec5a5c
In this commit, GeoIP-related classes are refactored and relocated to a common library as they are shared across multiple services. The crawler is refactored to enable the GeoIpBlocklist to use the new GeoIpDictionary as the base of its decisions. The converter is modified ot query this data to add a geoip:-keyword to documents to permit limiting a search to the country of the hosting server. The commit also adds due BY-SA attribution in the search engine footer for the source of the IP geolocation data. |
||
|---|---|---|
| .. | ||
| array | ||
| big-string | ||
| blocking-thread-pool | ||
| braille-block-punch-cards | ||
| btree | ||
| easy-lsh | ||
| geo-ip | ||
| guarded-regex | ||
| language-processing | ||
| message-queue | ||
| next-prime | ||
| random-write-funnel | ||
| term-frequency-dict | ||
| LICENSE.txt | ||
| readme.md | ||
Libraries
These are libraries that are not strongly coupled to the search engine's business logic. These libraries may not depend on features, services, processes, models, etc.
NOTE: These libraries are co-licensed under the MIT license.
Libraries
- The array library is for memory mapping large memory-areas, which Java has bad support for. It's designed to be able to easily replaced when Java's Foreign Function And Memory API is released.
- The btree library offers a static BTree implementation based on the array library.
- language-processing contains primitives for sentence extraction and POS-tagging.
- The message-queue library.
Micro libraries
- easy-lsh is a simple locality-sensitive hash for document deduplication
- guarded-regex makes predicated regular expressions clearer
- big-string offers seamless string compression
- random-write-funnel is a tool for reducing write amplification when constructing large files out of order.
- next-prime naive brute force prime sieve.
- braille-block-punch-cards renders bit masks into human-readable dot matrices using the braille block.