Done
Done 2022-01-30
- generate better default thumbnail on the fly (/)
Done 2022-01-19
- public API gateway (/)
Done 2022-01-16
- overhaul CSS of MEMEX (/)
Done 2022-01-15
- Improved random (/)
INSERT INTO EC_RANDOM_DOMAINS
SELECT DISTINCT(EC_DOMAIN.ID) FROM EC_DOMAIN_NEIGHBORS A
INNER JOIN EC_DOMAIN_NEIGHBORS B ON B.NEIGHBOR_ID=A.DOMAIN_ID
INNER JOIN EC_DOMAIN_NEIGHBORS C ON C.NEIGHBOR_ID=B.DOMAIN_ID
INNER JOIN EC_DOMAIN ON A.DOMAIN_ID=EC_DOMAIN.ID
WHERE C.DOMAIN_ID IN (SELECT ID FROM EC_DOMAIN WHERE URL_PART IN (secret-sauce))
AND EC_DOMAIN.STATE>=0;
Done 2022-01-14
- Dark Mode (/)
- Screengrabs by domain (/)
- Revise exploration mode (/)
- Improve keyboard navigation (/)
Done 2022-01-12
- Search redesign (/)
- Fixed dictionary corruption bug (/)
Done 2022-01-04
- Improve site:-query QOL (/)
- Fix byte folder bug (/)
- refactor EC_URL (/)
ALTER TABLE EC_URL MODIFY COLUMN PROTO ENUM('http', 'https', 'gemini') NOT NULL;
-- put visit-metadata in separate table (/)
Done 2021-12-03
- fix bug in language detection (/)
-- re-fetching some pages (/)
Done 2021-12-02
- new approach for query rewriting (/)
Done 2021-11-14
- make site:-queries return a dummy entry when no site information is available (/)
Done 2021-11-11
- hybridized ordering of domains on reindex, F(previous rank, previous quality). (/)
- mark documents with audio, video, object tags (/)
Done 2021-11-10
- car service <2021-11-18> (/)
Done 2021-10-30
- Add auto redirects for guesswork rss/atom/feed-requests to /log/feed.xml (/)
Done 2021-10-29
- investigate extracting more keywords (/)
-- textrank (/)
-- tf-idf (x)
-- sideload additional keywords for most popular sites (/)
Done 2021-10-12
- refactor index converter (/)
- clean up code garbage (/)
Done 2021-10-05
- trial more vanilla PageRank approach as a tertiary algorithm (/)
- fix a search result priortization bugs for mixed rankings (/)
- fix search interface for firefox on android (x)
It is reportedly broken
-- figure out how to replicate this problem (x)
- fix potential DoS where certain search queries with a large number of common but mutually exclusive terms would take forever to process. (/)
test query: generic stores underground unusual
Done 2021-10-03
- prioritize n-gram matches over word matches (/)
- show informative error page when the index server reboots (/)
Done 2021-10-02
- Personalized Page Rank (/)
- Duelling Algorithms (/)
Done 2021-10-30
- Launch October Update (/)
Done 2021-09-26
- fix broken search use-cases (/)
-- c language (/)
-- 67 chevy (/)
-- 68000 (/)
-- c# (/)
-- @twitterhandle (/)
-- #hashtag (/)
- trial tar based archiving to save the poor ext4 fs (/)
- use words to tag document format etc (/)
- dynamic re-bucketing based on something like (/)
SELECT DEST.URL_PART,EXP(DEST.QUALITY)*SUM(EXP(SOURCE.QUALITY)) AS Q from EC_DOMAIN DEST INNER JOIN EC_DOMAIN_LINK ON DEST.ID=DEST_DOMAIN_ID INNER JOIN EC_DOMAIN SOURCE ON SOURCE.ID=SOURCE_DOMAIN_ID WHERE DEST.INDEXED>0 GROUP BY DEST_DOMAIN_ID
Done 2021-09-19
- Fix several indexing bugs that hid relevant search results (/)
Done 2021-09-17
- Added search profiles (/)
Done 2021-09-16
- Rephrased an error message that some people took to mean they weren't speaking a proper language (/)
Done 2021-09-15
- Using in-site domain link-names to add search terms (/)
- Fixed buggy default content-type (/)
- Even more aggressive unicode language dectection (/)
Done 2021-09-11
- Status flag for domains (/)
Indexed, Active, Blocked
- Improve topic detection (/)
Done 2021-09-09
- Tuned search results to demote very short results (/)
Done 2021-09-08
- Encyclopedia tries harder to find the right article if the case match isn't exact (/)
Done 2021-09-06
- Breaking changes for next Index-rebuild (/)
-- Change writer bucket scaling to 1/4 (/)
-- Move protocol and port from EdgeDomain to EdgeURL (/)
-- Change database schemas to reflect (/)
-- ISO-8859-1/UTF-8 charset sniffer (/)
-- Fixed a bug that would occasionally cause the crawler to re-index the same working set multiple times (/)
Done 2021-09-02
- improve edge-director throughput (/)
- give edge-director state for semi-blocking tasks (/)
Done 2021-08-31
- optimize URL index size (/)
Done 2021-08-28
- clean up gemini navigation (/)
- Atom feed for HTTPS and Gemini (/)
Done 2021-08-27
- Feed gemini server with rendered gmi-content (/)
-- Output the content (/)
-- Generate feeds (/)
-- Make the gemini server read it (x)
-- Switch over (/)
Done 2021-08-26
- Absorb gemini server into WMSA (/)
Done 2021-08-25
- wildcard domain for marginalia.nu (/)
-- move memex to memex-subdomain (/)
- feeds on FEED pragma (/)
Done 2021-08-24
- Top nav bar overhaul (/)
Done 2021-08-23
- add marker for which files are todo files (/)
Added %%%/pragmas for toggling behavior
-- Added template helpers for consuming pragmas (/)
-- Used to improve topic pages (/)
- Fixes for git (/)
Done 2021-08-22
- File manager (/)
-- Delete (/)
-- Delete Empty Dir (x)
-- Move/Rename (/)
--- System for tombstones/redirects (/)
- Edit for / does not work (/)
Needed better support for non-normalized URLs, e.g. //index.gmi
- Backlinks for index (/)
Done 2021-08-21
- Git Integration (/)
-- Use commit hooks to trigger pull (/)
https://git-scm.com/book/uz/v2/Appendix-B%3A-Embedding-Git-in-your-Applications-JGit- Recursive directory watch (/)
- Two column layout (/)
Done 2021-08-20
- Overhaul MEMEX navigation (/)
-- Navigation bar (/)
-- Generate site map (x)
-- Editing (/)
--- Add update-root link (/)
- Tombstones aren't generated properly on-delete (/)
The tombstone db wasn't properly reloaded after being updated.
- Just write static files to disk instead of using an intermediary backend server. (/)
-- Use alias directive to set different root for memex path. (/)
-- Content-type is finnicky (/)
I want to serve html-wrapped .gmi and .html
location ~* \.(gmi|png)$ {
types {
text/html gmi;
text/html png;
}
}
Done 2021-08-19
- Move away from statically generated HTML forms in memex (/)
- Fix stability of podcast scraper (/)
- Get crawling up again (/)
-- Monitoring (/)
--- Extraction (/)
--- Status page (/)
-- Scraper config (/)
-- DNS cache (?)
-- IP Block CDNs (/)
--- Parse CIDR (/)
Apache Commons.Net SubnetUtil seems to
do the job, although it can't deal
with IPV6 :-/
--- CloudFlare (/)
173.245.48.0/20
103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
141.101.64.0/18
108.162.192.0/18
190.93.240.0/20
188.114.96.0/20
197.234.240.0/22
198.41.128.0/17
162.158.0.0/15
172.64.0.0/13
131.0.72.0/22
104.16.0.0/13
104.24.0.0/14
2400:cb00::/32
2606:4700::/32
2803:f800::/32
2405:b500::/32
2405:8100::/32
2a06:98c0::/29
2c0f:f248::/32
--- Fastly (/)
23.235.32.0/20
43.249.72.0/22
103.244.50.0/24
103.245.222.0/23
103.245.224.0/24
104.156.80.0/20
146.75.0.0/17
151.101.0.0/16
157.52.64.0/18
167.82.0.0/17
167.82.128.0/20
167.82.160.0/20
167.82.224.0/20
172.111.64.0/18
185.31.16.0/22
199.27.72.0/21
199.232.0.0/16
- Refactor task management (/)
-- Fix prepend (/)
-- Add tests (/)
- Refactor Floyd-Steinberg ditherer (/)
- Todo move-to-done function puts header last in #Done (/)
Done 2021-08-16
- Pictures-in-HTML (/)
-- Implement compression via Floyd-Steinberg dithering (/)
https://encyclopedia.marginalia.nu/wiki/Floyd%E2%80%93Steinberg_ditheringhttp://image4j.sourceforge.net/javadoc/index.html?net/sf/image4j/util/ConvertUtil.html
--- Ensure 4 bit (/)
--- On upload (/)
--- Convert existing stuff on-read (x)
-- Render image views (/)
--- Add to index (/)
-- Upload form (/)
Done 2021-08-15
- CSS fixes for mobile (/)
-- text align for tasks (/)
-- indent overflowed tasks (/)
- Fix CME (/)
java.util.ConcurrentModificationException: null
at java.util.HashMap.forEach(HashMap.java:1428) ~[?:?]
at nu.marginalia.wmsa.memex.MemexData.forEach(MemexData.java:51) ~[WMSA-1628951793.jar:?]
at nu.marginalia.wmsa.memex.Memex.reRender(Memex.java:49) ~[WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.core.Scheduler$PeriodicDirectTask.run(Scheduler.java:566) [WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.core.Scheduler$Worker$PeriodicTask.run(Scheduler.java:513) [WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) [WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) [WMSA-1628951793.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
ERROR 2021-08-14 16:36:39,467 RxCachedThreadScheduler-2 MemexMain : Uncaught exception
java.util.ConcurrentModificationException: null
at java.util.HashMap.forEach(HashMap.java:1428) ~[?:?]
at nu.marginalia.wmsa.memex.MemexData.forEach(MemexData.java:51) ~[WMSA-1628951793.jar:?]
at nu.marginalia.wmsa.memex.Memex.reRender(Memex.java:49) ~[WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.core.Scheduler$PeriodicDirectTask.run(Scheduler.java:566) ~[WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.core.Scheduler$Worker$PeriodicTask.run(Scheduler.java:513) ~[WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) [WMSA-1628951793.jar:?]
at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) [WMSA-1628951793.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Done 2021-08-14
- Automatic TODO task categorization (/)
- Login API on separate service (/)
-- Set up service (/)
-- Route requests (/)
- Fix header auto-location (/)
- Display top tasks in index (/)
Done 2021-08-10
-- + in URLs? (/)
proxy_pass with / forces nginx to parse the url (why?)
Bad:
proxy_pass http://127.0.0.1:5025/public/wiki/
Good:
rewrite ^ $request_uri
rewrite ^/(.*) /public/$1 break;
return 400;
proxy_pass http://127.0.0.1:5025$uri;
- Encyclopedia (/)
-- Search API (/)
-- code tags (/)
Done 2021-08-06
- Memex (/)
-- GemtextParser (/)
-- Service skeleton (/)
-- Link extraction (/)
-- Rendering (/)
--- Stylesheet (/)
-- Metadata (-)
-- Uppdateringar (/)
--- API (/)
--- Formulär (/)
Done 2021-08-04
- Service Lockdown (/)
-- X-Public header in code (/)
-- Move endpoints (/)
--- Resource Store (/)
--- Search (/)
--- Assistant (/)
-- Update clients (/)
--- Resource Store (/)
--- Search Service (/)
--- Assistant (-)
-- Update nginx (/)
-- Update links on website (/)
- Tune wiki archive fs (/)
sudo tune2fs -O ^dir_index /dev/nvme0n1p2
- marginalia.nu:9999 "BBS" (/)
Done 2021-08-03
- encyclopedia.marginalia.nu (/)
- Verify automatic backup of git (/)
- Reddit frontend (/)
-- Scraper: (/)
-- API: Marginalia 2: (/)
- Wiki (/)
-- on Optane (/)
-- fix Hildegard of Bingen (/)
- Block bots on nginx (/)
https://kb.linuxlove.xyz/nginx-badbotblocker.html
Done 2021-08-02
- Install Optane (/)
-- Migrate MariaDB (/)
- Wiki (/)
-- redirects (/)
-- top notices (/)
- Bucket4J rate limiting (/)
- Service Monitoring (/)
Done 2021-08-01
- Update Cert (/)
- Backups for git (/)
Done 2021-07-30
- Load Wikidata from ZIM (/)
- Migrate Server to Debian Buster (/)
Done 2021-07-28
- Update description generation algorithm (/)
-- Recalculate descriptions (...) (/)
- Wiki data (/)
-- Load data (/)
-- Wrap wikipedia (/)
-- ZIM? (-)
-- Wikipedia Cleaner (/)
Done 2021-07-27
- Spell checker service? (/)
https://github.com/wolfgarbe/SymSpell- Calculations (/)
-- Detection (/)
-- Parser (/)
-- Unit conversion (/)
--- Temperature (/)
--- Distance (/)
--- Weight (/)
--- Area (/)
--- Volume (/)
Done 2021-07-26
- Save websites to disk? (/)
-- GZipped (/)
-- XFS (?)
- Local backlinks in GMI (/)
-- Parse GMI for links and titles (/)
-- Create tags system (/)
- Use prime sizing for HashMap! (/)
-- How to find primes (/)
- Arbitarary size HashMap (/)
Done 2021-07-25
- Syntax for orgmode + GMI in kate (/)
Use /usr/share/kde4/apps/katepart/syntax/markdown.xml
Done 2021-07-23
- Dictionary analysis in scraping (/)
It seems viable to estimate the lanaguage of a document based on the overlap with a N-most-common-words dictionary. Threshold 0.05 ok?
-- English (/)
-- Swedish (/)
-- Latin (/)
- Clean up tests (/)
Done 2021-07-22
GZip Compression stats:
63% old 21% new
- Hash map (/)
-- Contiguous memory bins (/)
- Key Folding (/)
-- For strings (/)
-- For integers (/)
-- For dates (x)
- Debian Desktop (/)
-- Docker (/)
-- Java 14 (/)
-- IntelliJ (/)
-- Code (/)
-- Gradle (/)
-- OrgMode (/)
Done 2021-07-21
- Bugfix: Domain Resolution (/)
Done 2021-07-20
- Index Changes (/)
-- Remove Junk Logging (/)
-- Split Query (/)
-- Implement in Frontend (/)
- Dictionary Service (/)
-- Add Index To Table (/)
-- Populate test db (/)
-- Build tests (/)
-- Integrate into frontend (/)
- Site Information (/)
-- Fetch (/)
-- 404 (/)