From bac69bec3b77e9fb20d48cccdde231ad85ccd5d9 Mon Sep 17 00:00:00 2001 From: Enki Date: Fri, 20 Dec 2024 20:01:15 +0000 Subject: [PATCH] Update README.md --- README.md | 185 +++++++++++++++++++++++++++++------------------------- 1 file changed, 100 insertions(+), 85 deletions(-) diff --git a/README.md b/README.md index c4f2e15..e7a9d7b 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,19 @@ # Strfry Nostr Spam Filter +A sophisticated spam filter for Strfry Nostr relays with reputation tracking, burst handling, and adaptive rate limiting. ## Features -- Reputation-based filtering system -- Per-user rate limiting +- Reputation-based filtering system with detailed logging +- Intelligent burst handling for follow lists +- Per-user and kind-specific rate limiting - Content similarity detection - New account protection - Bot behavior detection - Progressive penalty system - Automatic recovery mechanism -- Kind-specific rate limits - Configurable thresholds +- Memory-efficient cleanup system ## Installation @@ -26,7 +28,7 @@ chmod +x relay-spam-filter.js ``` 3. Configure your strfry relay to use the filter: -```bash +```yaml # Add to your strfry configuration plugins = [ { @@ -37,126 +39,139 @@ plugins = [ ## Configuration -The filter includes many configurable options. Here are the key settings: +### Rate Limits and Burst Handling -### Rate Limits ```javascript -maxRepliesPerMinute: 50, // Maximum replies per minute per user -maxRepliesPerHour: 400, // Maximum replies per hour per user -``` +// General rate limits +maxRepliesPerMinute: 50, +maxRepliesPerHour: 400, -### Event-Type Specific Limits -```javascript +// Kind-specific limits with burst handling kindSpecificLimits: { - 0: { maxPerHour: 10 }, // Profile updates - 3: { maxPerHour: 5 }, // Contact list updates - 1: { // Regular posts + 0: { // Profile updates + maxPerHour: 10, + minTimeBetween: 300 + }, + 1: { // Regular posts maxPerMinute: 30, maxPerHour: 200 + }, + 3: { // Follow lists + maxPerMinute: 120, + maxPerHour: 360, + minTimeBetween: 2, + ignoreSimilarity: true, + burstAllowance: { + maxBurst: 500, // Maximum events in burst + burstWindow: 300, // Window for burst (5 minutes) + cooldownPeriod: 3600 // Cooldown after burst (1 hour) + } + }, + 7: { // Reactions + maxPerMinute: 60, + maxPerHour: 300, + ignoreSimilarity: true } } ``` ### Reputation System + ```javascript reputationConfig: { - initialScore: 100, // Starting reputation for new users - goodEventBonus: 1, // Points gained for good events - spamPenalty: -15, // Base penalty for spam - recoveryRate: 3, // Points recovered per hour - minScore: -100, // Minimum possible reputation - maxScore: 1000, // Maximum possible reputation - blockThreshold: -50, // Users blocked at this score - blockRecoveryThreshold: -25 // Must recover to this to post again + initialScore: 100, // Starting reputation + goodEventBonus: 1, // Points for good events + spamPenalty: -15, // Base spam penalty + recoveryRate: 3, // Points/hour recovery + minScore: -100, // Minimum reputation + maxScore: 1000, // Maximum reputation + blockThreshold: -50, // Blocking threshold + blockRecoveryThreshold: -25, // Recovery threshold + + // Progressive penalties + penaltyScaling: { + 0: -15, // Above 0 reputation + "-25": -20, // Below -25 + "-50": -25 // Below -50 + }, + + // Variable recovery rates + recoveryScaling: { + 0: 3, // Above 0 + "-25": 2, // Below -25 + "-50": 1 // Below -50 + } } ``` ### New User Protection -```javascript -newPubkeyReplyThreshold: 60, // Seconds new users must wait to reply -newPubkeyMaxPostsIn5Min: 10 // Maximum posts in first 5 minutes -``` -### Content Analysis ```javascript -contentSimilarityThreshold: 0.8, // 80% similarity triggers spam detection -fastReplyThreshold: 30, // Minimum seconds between reply and original post +newPubkeyReplyThreshold: 60, // Seconds before first reply +newPubkeyMaxPostsIn5Min: 10 // Initial rate limit ``` ## How It Works -### Reputation System +### Enhanced Reputation System -The filter maintains a reputation score for each user: +The filter maintains a detailed reputation score for each user: -1. New users start at 100 points -2. Good behavior slowly increases reputation -3. Spam behavior results in penalties: - - -15 points above 0 reputation - - -20 points below -25 reputation - - -25 points below -50 reputation -4. Recovery rates: - - 3 points/hour above 0 - - 2 points/hour between -25 and 0 - - 1 point/hour below -50 -5. Users are blocked at -50 until they recover to -25 +- New users start at 100 points +- Good behavior increases reputation gradually +- Spam behavior triggers scaled penalties based on current reputation +- Recovery rates vary by reputation level +- Detailed logging of all reputation changes +- Special handling for follow list operations + +### Burst Handling + +Special consideration for bulk operations: + +- Follow list updates (kind 3) can operate in burst mode +- Up to 500 follows in a 5-minute window +- Cooldown period after burst completion +- Normal rate limits apply during cooldown +- Reputation affects burst allowances ### Spam Detection -The filter checks for several types of spam behavior: +Multiple detection mechanisms: -1. Rate Limiting - - Monitors post frequency per user - - Different limits for different event types - - Limits scale with reputation +- Adaptive Rate Limiting + - Per-user and per-kind limits + - Reputation-based scaling + - Burst mode for specific operations -2. Content Analysis - - Checks for duplicate content across users - - Detects suspiciously fast replies - - Identifies bot-like behavior patterns +- Content Analysis + - Cross-user content similarity + - Fast reply detection + - Bot pattern recognition -3. New Account Protection - - Stricter limits for new accounts - - Longer waiting periods for replies - - Limited initial posting rate +- New Account Protection + - Progressive rate limits + - Reply waiting periods + - Initial posting restrictions -4. Bot Detection - - Identifies automated posting patterns - - Checks for relay URL spam - - Monitors reply timing patterns +### Logging and Monitoring -### Progressive Enforcement +Comprehensive logging system: -The system becomes progressively stricter with problematic behavior: - -1. Initial warnings and small penalties -2. Increased penalties for continued violations -3. Stricter rate limits for low-reputation users -4. Eventual blocking for persistent offenders - -## Bypass Rules - -Certain event kinds can bypass specific checks: - -```javascript -allowedKinds: [3, 5, 10001, 10002, 30311], // Bypass content checks -bypassAllChecksKinds: [38383] // Bypass all checks ``` - -## Monitoring - -The filter provides detailed logging: - -```javascript [2024-12-19T10:15:30.123Z] Event abc123 from pubkey xyz789 (reputation: 85.50) -[2024-12-19T10:15:30.124Z] Rate limit exceeded for pubkey xyz789 +[2024-12-19T10:15:30.124Z] Reputation update: 85.50 -> 86.50 (change: +1.00) +[2024-12-19T10:15:30.125Z] Burst tracking - count: 42/500 (window: 300s remaining) ``` -## Performance Considerations +### Memory Management -- Memory usage is managed through periodic cleanup -- Old events and stats are automatically purged -- Reputation data persists for 24 hours of inactivity +Efficient resource usage: + +- Periodic cleanup of old data +- Configurable retention periods +- Automatic purging of inactive users +- Memory-efficient data structures +- Burst tracking cleanup ## Contributing