torrent-gateway/README.md

# BitTorrent Gateway

A comprehensive unified content distribution system that seamlessly integrates BitTorrent protocol, WebSeed technology, DHT peer discovery, built-in tracker, video transcoding, and Nostr announcements. This gateway provides intelligent content distribution by automatically selecting the optimal delivery method based on file size and network conditions, with automatic video transcoding for web-compatible streaming.

## Architecture Overview

The BitTorrent Gateway operates as a unified system with multiple specialized components working together:

### Core Components

**1. Gateway HTTP API Server (Port 9877)**
- Main web interface and API endpoints
- File upload/download management
- Smart proxy for reassembling chunked content
- WebSeed implementation with advanced LRU caching
- Rate limiting and abuse prevention

**2. Embedded Blossom Server (Port 8082)**
- Nostr-compatible blob storage protocol
- Direct blob storage for small files (<100MB)
- Integration with gateway for seamless operation

**3. DHT Node (Port 6883)**
- Distributed peer discovery
- BitTorrent DHT protocol implementation
- Bootstrap connectivity with major DHT networks
- Automatic torrent announcement and peer sharing

**4. Built-in BitTorrent Tracker**
- Full BitTorrent tracker implementation
- Announce/scrape protocol support
- P2P coordination and peer ranking
- Client compatibility optimizations for qBittorrent, Transmission, WebTorrent, Deluge, uTorrent

**5. Video Transcoding Engine**
- Automatic H.264/AAC conversion for web compatibility
- Smart serving: transcoded versions when ready, originals otherwise
- Background processing with priority queuing
- FFmpeg integration with progress tracking
- Multiple quality profiles and format support

### Smart Storage Strategy

The system uses an intelligent dual-storage approach with video optimization:

- **Small Files (<100MB)**: Stored directly as blobs using Blossom protocol
- **Large Files (≥100MB)**: Automatically chunked into 2MB pieces, stored as torrents with WebSeed fallback
- **Video Files**: Automatically queued for H.264/AAC transcoding to web-compatible MP4 format
- **Smart Serving**: Transcoded versions served when ready, original chunks as fallback

### P2P Coordination System

A sophisticated P2P coordinator manages all networking components:

- **Unified Peer Discovery**: Aggregates peers from tracker, DHT, and WebSeed sources
- **Smart Peer Ranking**: Geographic proximity and performance-based peer selection
- **Load Balancing**: Distributes load across multiple peer sources
- **Health Monitoring**: Real-time monitoring of all P2P components with automatic alerting

## Installation

### Prerequisites

- Go 1.21 or later
- SQLite3
- FFmpeg (for video transcoding, optional)
- 10MB+ available storage
- Linux/macOS/Windows

### Quick Start (Standalone)

```bash
# Clone repository
git clone https://git.sovbit.dev/enki/torrentGateway.git
cd torrentGateway

# Build the gateway
go build -o gateway ./cmd/gateway

# Run with default configuration
./gateway
```

The web interface will be available at http://localhost:9877

### Production Deployment (No Docker)

For production deployment without Docker, use the native installation script:

```bash
# Make installation script executable
chmod +x scripts/install_native.sh

# Install as system service
sudo ./scripts/install_native.sh

# Start the service
sudo systemctl start torrent-gateway
sudo systemctl enable torrent-gateway

# Check status
sudo systemctl status torrent-gateway
```

### Nginx Reverse Proxy Configuration

For production deployments, use Nginx as a reverse proxy:

```nginx
server {
    listen 80;
    server_name your-domain.com;
    
    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name your-domain.com;
    
    # SSL Configuration
    ssl_certificate /path/to/your/certificate.pem;
    ssl_certificate_key /path/to/your/private-key.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    
    # Security Headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
    
    # Gateway API and Web Interface
    location / {
        proxy_pass http://127.0.0.1:9877;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # WebSocket support for real-time features
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # Increase timeouts for large file uploads
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
    
    # Blossom Server (optional, if running separately)
    location /blossom/ {
        proxy_pass http://127.0.0.1:8082/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    # Increase client max body size for file uploads
    client_max_body_size 10G;
    
    # Enable gzip compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml+rss application/atom+xml image/svg+xml;
}
```

### SystemD Service Configuration

The native installer creates a systemd service. Manual configuration:

```bash
# Create service file
sudo tee /etc/systemd/system/torrent-gateway.service > /dev/null <<EOF
[Unit]
Description=Torrent Gateway Service
After=network.target

[Service]
Type=simple
User=torrent-gateway
Group=torrent-gateway
WorkingDirectory=/opt/torrent-gateway
ExecStart=/opt/torrent-gateway/gateway -config /etc/torrent-gateway/config.yaml
Restart=always
RestartSec=3
LimitNOFILE=65536

# Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/torrent-gateway/data
ReadWritePaths=/var/log/torrent-gateway

[Install]
WantedBy=multi-user.target
EOF

# Create dedicated user
sudo useradd --system --home /opt/torrent-gateway --shell /bin/false torrent-gateway

# Create directories and set permissions
sudo mkdir -p /opt/torrent-gateway/{data/{blobs,chunks},logs}
sudo mkdir -p /etc/torrent-gateway
sudo mkdir -p /var/log/torrent-gateway
sudo chown -R torrent-gateway:torrent-gateway /opt/torrent-gateway
sudo chown -R torrent-gateway:torrent-gateway /var/log/torrent-gateway

# Copy binary and config
sudo cp gateway /opt/torrent-gateway/
sudo cp configs/config.yaml /etc/torrent-gateway/
sudo chmod +x /opt/torrent-gateway/gateway

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable torrent-gateway
sudo systemctl start torrent-gateway
```

### Configuration

The default configuration is in `configs/config.yaml`. Customize settings there:

```yaml
# Unified Blossom-BitTorrent Gateway Configuration
mode: unified  # unified (all services), gateway-only, blossom-only, dht-only

# Gateway HTTP API server
gateway:
  enabled: true
  port: 9877
  max_upload_size: 10GB

# Embedded Blossom server
blossom_server:
  enabled: true
  port: 8082
  max_blob_size: 100MB
  rate_limit:
    requests_per_minute: 100
    burst_size: 20

# DHT node configuration
dht:
  enabled: true
  port: 6883
  bootstrap_nodes:
    - "router.bittorrent.com:6881"
    - "dht.transmissionbt.com:6881"
    - "router.utorrent.com:6881"
    - "dht.libtorrent.org:25401"
  announce_interval: 900s  # 15 minutes
  max_torrents: 10000
  max_peers_per_torrent: 200

# Shared storage configuration
storage:
  blob_threshold: 104857600    # 100MB in bytes
  chunk_size: 2097152          # 2MB chunks for large files  
  metadata_db: "./data/metadata.db"
  blob_storage: "./data/blobs"
  chunk_storage: "./data/chunks"

# Built-in BitTorrent tracker
tracker:
  enabled: true
  announce_interval: 1800      # 30 minutes
  min_interval: 900            # 15 minutes  
  default_numwant: 50          # peers to return
  max_numwant: 100             # maximum peers

# Nostr relay configuration
nostr:
  relays:
    - "wss://freelay.sovbit.host"

# Video transcoding configuration
transcoding:
  enabled: true                # Enable/disable transcoding (requires ffmpeg)
  concurrent_jobs: 2           # Number of parallel transcoding jobs
  work_dir: "./data/transcoded" # Directory for transcoded files
  auto_transcode: true         # Automatically transcode uploaded videos
  min_file_size: 50MB          # Don't transcode files smaller than this

# Admin configuration
admin:
  enabled: true
  pubkeys:
    - "your_admin_pubkey_here"  # Replace with actual admin pubkey
  default_user_storage_limit: "10GB"

# Rate limiting configuration
rate_limiting:
  upload:
    requests_per_second: 1.0    # Max uploads per second per IP
    burst_size: 5               # Burst allowance
    max_file_size: "3GB"        # Maximum file size
  download:
    requests_per_second: 50.0   # Global download rate limit
    burst_size: 100             # Global burst allowance
  stream:
    requests_per_second: 10.0   # Max streams per second per file
    burst_size: 20              # Stream burst allowance
    max_concurrent: 50          # Max concurrent streams
  auth:
    login_attempts_per_minute: 10  # Login attempts per IP per minute
    burst_size: 5                  # Login burst allowance
```

## API Reference

### Authentication

All endpoints support Nostr-based authentication via:
- **NIP-07**: Browser extension (Alby, nos2x)
- **NIP-46**: Remote signer/bunker URL

```bash
# Get challenge
curl http://localhost:9877/api/auth/challenge

# Login (requires signed Nostr event)
curl -X POST http://localhost:9877/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"auth_type": "nip07", "auth_event": "..."}'
```

### File Operations

```bash
# Upload file
curl -X POST http://localhost:9877/api/upload \
  -F "file=@example.mp4" \
  -F "announce_dht=true"

# Download file
curl http://localhost:9877/api/download/[hash] -o downloaded_file

# Get torrent
curl http://localhost:9877/api/torrent/[hash] -o file.torrent

# Stream video (HLS)
curl http://localhost:9877/api/stream/[hash]/playlist.m3u8

# Check transcoding status (requires auth)
curl http://localhost:9877/api/users/me/files/[hash]/transcoding-status \
  -H "Authorization: Bearer [session_token]"
```

### User Management

```bash
# Get user stats (requires auth)
curl http://localhost:9877/api/users/me/stats \
  -H "Authorization: Bearer [session_token]"

# List user files
curl http://localhost:9877/api/users/me/files \
  -H "Authorization: Bearer [session_token]"

# Delete file
curl -X DELETE http://localhost:9877/api/users/me/files/[hash] \
  -H "Authorization: Bearer [session_token]"
```

## Nostr Integration

The BitTorrent Gateway seamlessly integrates with the Nostr network to enable decentralized content discovery and social features:

### How Nostr Integration Works

1. **Content Announcements**: When files are uploaded, they're automatically announced to configured Nostr relays as structured events
2. **Decentralized Discovery**: Other nodes and users can discover new content by subscribing to these announcement events
3. **Social Layer**: Users can comment, react, and share content using standard Nostr clients
4. **No Central Authority**: Content discovery happens through the distributed Nostr relay network

### Technical Implementation

- **Event Type**: Uses kind `2003` per NIP-35 specification for torrent announcements
- **Content Addressing**: Files are identified by cryptographic hashes (SHA-256)
- **Metadata**: File size, name, magnet links, and WebSeed URLs included in events
- **Relay Redundancy**: Announces to multiple relays for reliability

### Nostr Event Structure

When a file is uploaded, the gateway creates a Nostr event like this:

```json
{
  "kind": 2003,
  "content": "New torrent: example-video.mp4",
  "tags": [
    ["title", "Example Video"],
    ["x", "d1c5a0f85a4e4f3d8e7d8f9c6b5a4e3f2d1c0b9a8e7d6f5c4b3a2e1d0c9b8a7f6"],
    ["file", "example-video.mp4", "157286400"],
    ["magnet", "magnet:?xt=urn:btih:..."],
    ["webseed", "https://gateway.example.com/api/webseed/..."],
    ["blossom", "sha256_blob_hash"],
    ["stream", "https://gateway.example.com/api/stream/hash"],
    ["hls", "https://gateway.example.com/api/stream/hash/playlist.m3u8"],
    ["transcoded", "https://gateway.example.com/api/stream/hash?transcoded=true"],
    ["duration", "3600"],
    ["video", "1920x1080", "30fps", "h264"],
    ["m", "video/mp4"],
    ["t", "torrent"],
    ["t", "video"],
    ["t", "streaming"],
    ["tcat", "video,streaming"]
  ]
}
```

### Benefits

- **Censorship Resistance**: No single point of control for content discovery
- **Network Effects**: Content spreads naturally through the Nostr social graph
- **Interoperability**: Standard Nostr clients can display and interact with content
- **Privacy**: Users maintain control over their identity and data

## Performance & Scaling

### Optimization Features
- **Concurrent Downloads**: Multiple parallel piece downloads
- **Geographic Peer Selection**: Prioritizes nearby peers for faster transfers  
- **Smart Caching**: LRU eviction with configurable cache sizes
- **Rate Limiting**: Prevents abuse while maintaining performance
- **Connection Pooling**: Efficient resource utilization

### Monitoring & Alerting
- **Component Health Scores**: 0-100 scoring for all P2P components
- **Performance Metrics**: Response times, throughput, error rates
- **Automatic Alerts**: Configurable thresholds for degraded performance
- **Diagnostic Endpoints**: Detailed system introspection

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive tests
4. Submit a pull request

## License

[Add your license information here]

## Support

- **Issues**: Report bugs and feature requests via GitHub issues
- **Documentation**: Additional documentation in `/docs`
- **Community**: [Add community links if available]