391 lines
13 KiB
Markdown
391 lines
13 KiB
Markdown
# BitTorrent Gateway
|
|
|
|
A comprehensive unified content distribution system that seamlessly integrates BitTorrent protocol, WebSeed technology, DHT peer discovery, built-in tracker, video transcoding, Blossom blob storage, and multi-relay Nostr announcements. This gateway provides intelligent content distribution by automatically selecting the optimal delivery method based on file size and network conditions, with automatic video transcoding, cleanup automation, and performance optimizations.
|
|
|
|
## Architecture Overview
|
|
|
|
The BitTorrent Gateway operates as a unified system with multiple specialized components working together:
|
|
|
|
### Core Components
|
|
|
|
**1. Gateway HTTP API Server (Port 9877)**
|
|
- Main web interface and API endpoints
|
|
- File upload/download management
|
|
- Smart proxy for reassembling chunked content
|
|
- WebSeed implementation with advanced LRU caching
|
|
- Rate limiting and abuse prevention
|
|
|
|
**2. Embedded Blossom Server (Port 8082)**
|
|
- Nostr-compatible blob storage protocol
|
|
- Direct blob storage for small files (<100MB)
|
|
- Integration with gateway for seamless operation
|
|
|
|
**3. DHT Node (Port 6883)**
|
|
- Distributed peer discovery
|
|
- BitTorrent DHT protocol implementation
|
|
- Bootstrap connectivity with major DHT networks
|
|
- Automatic torrent announcement and peer sharing
|
|
|
|
**4. Built-in BitTorrent Tracker**
|
|
- Full BitTorrent tracker implementation
|
|
- Announce/scrape protocol support
|
|
- P2P coordination and peer ranking
|
|
- Client compatibility optimizations for qBittorrent, Transmission, WebTorrent, Deluge, uTorrent
|
|
|
|
**5. Video Transcoding Engine**
|
|
- Automatic H.264/AAC conversion for web compatibility
|
|
- Smart serving: transcoded versions when ready, originals otherwise
|
|
- Background processing with priority queuing
|
|
- FFmpeg integration with progress tracking
|
|
- Multiple quality profiles (1080p, 720p, 480p) with configurable bitrates
|
|
|
|
**6. Automated Cleanup System**
|
|
- Intelligent file lifecycle management
|
|
- Automatic removal of old files (configurable age: 90d default)
|
|
- Orphaned chunk detection and cleanup
|
|
- Inactive user cleanup (180 days default)
|
|
- Smart proxy cache auto-cleanup (10-minute intervals)
|
|
- Database optimization with SQLite pragma tuning
|
|
|
|
**7. Multi-Relay Nostr Integration**
|
|
- NIP-35 compliant torrent announcements
|
|
- NIP-71 video event publishing (horizontal/vertical detection)
|
|
- WebTorrent extensions with WebSocket tracker support
|
|
- Blossom hash cross-referencing
|
|
- Configurable relay sets for different content types
|
|
- Fault-tolerant publishing with success/failure tracking
|
|
|
|
### Smart Storage Strategy
|
|
|
|
The system uses an intelligent dual-storage approach with video optimization:
|
|
|
|
- **Small Files (<100MB)**: Stored directly as blobs using Blossom protocol
|
|
- **Large Files (≥100MB)**: Automatically chunked into 2MB pieces, stored as torrents with WebSeed fallback
|
|
- **Video Files**: Automatically queued for H.264/AAC transcoding to web-compatible MP4 format
|
|
- **Smart Serving**: Transcoded versions served when ready, original chunks as fallback
|
|
|
|
### P2P Coordination System
|
|
|
|
A sophisticated P2P coordinator manages all networking components:
|
|
|
|
- **Unified Peer Discovery**: Aggregates peers from tracker, DHT, and WebSeed sources
|
|
- **Smart Peer Ranking**: Geographic proximity and performance-based peer selection
|
|
- **Load Balancing**: Distributes load across multiple peer sources
|
|
- **Health Monitoring**: Real-time monitoring of all P2P components with automatic alerting
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
|
|
- Go 1.21 or later
|
|
- SQLite3
|
|
- FFmpeg (for video transcoding, optional)
|
|
- 10MB+ available storage
|
|
- Linux/macOS/Windows
|
|
|
|
### Quick Start (Standalone)
|
|
|
|
```bash
|
|
# Clone repository
|
|
git clone https://git.sovbit.dev/enki/torrentGateway.git
|
|
cd torrentGateway
|
|
|
|
# Build the gateway
|
|
go build -o gateway ./cmd/gateway
|
|
|
|
# Run with default configuration
|
|
./gateway
|
|
```
|
|
|
|
The web interface will be available at http://localhost:9877
|
|
|
|
### Production Deployment
|
|
|
|
For production deployment, use the automated installation script:
|
|
|
|
```bash
|
|
# Basic installation (HTTP only)
|
|
sudo ./scripts/install_native.sh
|
|
|
|
# Full installation with SSL/HTTPS
|
|
sudo ./scripts/install_native.sh --domain gateway.example.com --email admin@example.com
|
|
|
|
# With monitoring stack
|
|
sudo ./scripts/install_native.sh --domain gateway.example.com --email admin@example.com --with-monitoring
|
|
```
|
|
|
|
This will automatically:
|
|
- Install all dependencies (Go, nginx, Redis, FFmpeg, etc.)
|
|
- Build and deploy the application
|
|
- Configure systemd service
|
|
- Set up nginx reverse proxy
|
|
- Configure SSL certificates (if domain provided)
|
|
- Start all services
|
|
|
|
See `INSTALL.md` for detailed installation instructions and domain setup.
|
|
|
|
### Manual Installation (Development)
|
|
|
|
For development or manual setup, you can run the application directly:
|
|
|
|
```bash
|
|
# Build the application
|
|
go build -o bin/gateway cmd/gateway/main.go
|
|
|
|
# Create basic directories
|
|
mkdir -p data/{blobs,chunks}
|
|
|
|
# Run with default config
|
|
./bin/gateway
|
|
|
|
# Or with custom config
|
|
./bin/gateway -config configs/config.yaml
|
|
```
|
|
|
|
For production deployments, use the automated installation script instead.
|
|
|
|
### Configuration
|
|
|
|
The default configuration is in `configs/config.yaml`. Customize settings there:
|
|
|
|
```yaml
|
|
# Unified Blossom-BitTorrent Gateway Configuration
|
|
mode: unified # unified (all services), gateway-only, blossom-only, dht-only
|
|
|
|
# Gateway HTTP API server
|
|
gateway:
|
|
enabled: true
|
|
port: 9877
|
|
max_upload_size: 10GB
|
|
|
|
# Embedded Blossom server
|
|
blossom_server:
|
|
enabled: true
|
|
port: 8082
|
|
max_blob_size: 100MB
|
|
rate_limit:
|
|
requests_per_minute: 100
|
|
burst_size: 20
|
|
|
|
# DHT node configuration
|
|
dht:
|
|
enabled: true
|
|
port: 6883
|
|
bootstrap_nodes:
|
|
- "router.bittorrent.com:6881"
|
|
- "dht.transmissionbt.com:6881"
|
|
- "router.utorrent.com:6881"
|
|
- "dht.libtorrent.org:25401"
|
|
announce_interval: 900s # 15 minutes
|
|
max_torrents: 10000
|
|
max_peers_per_torrent: 200
|
|
|
|
# Shared storage configuration
|
|
storage:
|
|
blob_threshold: 104857600 # 100MB in bytes
|
|
chunk_size: 2097152 # 2MB chunks for large files
|
|
metadata_db: "./data/metadata.db"
|
|
blob_storage: "./data/blobs"
|
|
chunk_storage: "./data/chunks"
|
|
|
|
# Built-in BitTorrent tracker
|
|
tracker:
|
|
enabled: true
|
|
announce_interval: 1800 # 30 minutes
|
|
min_interval: 900 # 15 minutes
|
|
default_numwant: 50 # peers to return
|
|
max_numwant: 100 # maximum peers
|
|
|
|
# Nostr relay configuration
|
|
nostr:
|
|
relays: # NIP-35 torrent announcements
|
|
- "wss://freelay.sovbit.host"
|
|
- "wss://relay.damus.io"
|
|
- "wss://nos.lol"
|
|
- "wss://relay.nostr.band"
|
|
video_relays: # NIP-71 video events (can be different)
|
|
- "wss://relay.damus.io"
|
|
- "wss://nos.lol"
|
|
- "wss://freelay.sovbit.host"
|
|
publish_nip35: true # Enable NIP-35 torrent events
|
|
publish_nip71: true # Enable NIP-71 video events
|
|
auto_publish: true # Auto-publish on upload
|
|
private_key: "" # Hex private key (auto-generate if empty)
|
|
|
|
# Video transcoding configuration
|
|
transcoding:
|
|
enabled: true # Enable/disable transcoding (requires ffmpeg)
|
|
concurrent_jobs: 2 # Number of parallel transcoding jobs
|
|
work_dir: "./data/transcoded" # Directory for transcoded files
|
|
auto_transcode: true # Automatically transcode uploaded videos
|
|
min_file_size: 50MB # Don't transcode files smaller than this
|
|
|
|
# Admin configuration
|
|
admin:
|
|
enabled: true
|
|
pubkeys:
|
|
- "your_admin_pubkey_here" # Replace with actual admin pubkey
|
|
auto_cleanup: true # Enable automated cleanup
|
|
cleanup_age: "90d" # Delete files older than 90 days
|
|
max_file_age: "365d" # Maximum file age before forced deletion
|
|
default_user_storage_limit: "10GB"
|
|
|
|
# Rate limiting configuration
|
|
rate_limiting:
|
|
upload:
|
|
requests_per_second: 1.0 # Max uploads per second per IP
|
|
burst_size: 5 # Burst allowance
|
|
max_file_size: "3GB" # Maximum file size
|
|
download:
|
|
requests_per_second: 50.0 # Global download rate limit
|
|
burst_size: 100 # Global burst allowance
|
|
stream:
|
|
requests_per_second: 10.0 # Max streams per second per file
|
|
burst_size: 20 # Stream burst allowance
|
|
max_concurrent: 50 # Max concurrent streams
|
|
auth:
|
|
login_attempts_per_minute: 10 # Login attempts per IP per minute
|
|
burst_size: 5 # Login burst allowance
|
|
```
|
|
|
|
## API Reference
|
|
|
|
### Authentication
|
|
|
|
All endpoints support Nostr-based authentication via:
|
|
- **NIP-07**: Browser extension (Alby, nos2x)
|
|
- **NIP-46**: Remote signer/bunker URL
|
|
|
|
```bash
|
|
# Get challenge
|
|
curl http://localhost:9877/api/auth/challenge
|
|
|
|
# Login (requires signed Nostr event)
|
|
curl -X POST http://localhost:9877/api/auth/login \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"auth_type": "nip07", "auth_event": "..."}'
|
|
```
|
|
|
|
### File Operations
|
|
|
|
```bash
|
|
# Upload file
|
|
curl -X POST http://localhost:9877/api/upload \
|
|
-F "file=@example.mp4" \
|
|
-F "announce_dht=true"
|
|
|
|
# Download file
|
|
curl http://localhost:9877/api/download/[hash] -o downloaded_file
|
|
|
|
# Get torrent
|
|
curl http://localhost:9877/api/torrent/[hash] -o file.torrent
|
|
|
|
# Stream video (HLS)
|
|
curl http://localhost:9877/api/stream/[hash]/playlist.m3u8
|
|
|
|
# Check transcoding status (requires auth)
|
|
curl http://localhost:9877/api/users/me/files/[hash]/transcoding-status \
|
|
-H "Authorization: Bearer [session_token]"
|
|
```
|
|
|
|
### User Management
|
|
|
|
```bash
|
|
# Get user stats (requires auth)
|
|
curl http://localhost:9877/api/users/me/stats \
|
|
-H "Authorization: Bearer [session_token]"
|
|
|
|
# List user files
|
|
curl http://localhost:9877/api/users/me/files \
|
|
-H "Authorization: Bearer [session_token]"
|
|
|
|
# Delete file
|
|
curl -X DELETE http://localhost:9877/api/users/me/files/[hash] \
|
|
-H "Authorization: Bearer [session_token]"
|
|
```
|
|
|
|
## Nostr Integration
|
|
|
|
The BitTorrent Gateway seamlessly integrates with the Nostr network to enable decentralized content discovery and social features:
|
|
|
|
### How Nostr Integration Works
|
|
|
|
1. **Content Announcements**: When files are uploaded, they're automatically announced to configured Nostr relays as structured events
|
|
2. **Decentralized Discovery**: Other nodes and users can discover new content by subscribing to these announcement events
|
|
3. **Social Layer**: Users can comment, react, and share content using standard Nostr clients
|
|
4. **No Central Authority**: Content discovery happens through the distributed Nostr relay network
|
|
|
|
### Technical Implementation
|
|
|
|
- **Event Type**: Uses kind `2003` per NIP-35 specification for torrent announcements
|
|
- **Content Addressing**: Files are identified by cryptographic hashes (SHA-256)
|
|
- **Metadata**: File size, name, magnet links, and WebSeed URLs included in events
|
|
- **Relay Redundancy**: Announces to multiple relays for reliability
|
|
|
|
### Nostr Event Structure
|
|
|
|
When a file is uploaded, the gateway creates a Nostr event like this:
|
|
|
|
```json
|
|
{
|
|
"kind": 2003,
|
|
"content": "New torrent: example-video.mp4",
|
|
"tags": [
|
|
["title", "Example Video"],
|
|
["x", "d1c5a0f85a4e4f3d8e7d8f9c6b5a4e3f2d1c0b9a8e7d6f5c4b3a2e1d0c9b8a7f6"],
|
|
["file", "example-video.mp4", "157286400"],
|
|
["magnet", "magnet:?xt=urn:btih:..."],
|
|
["webseed", "https://gateway.example.com/api/webseed/..."],
|
|
["blossom", "sha256_blob_hash"],
|
|
["stream", "https://gateway.example.com/api/stream/hash"],
|
|
["hls", "https://gateway.example.com/api/stream/hash/playlist.m3u8"],
|
|
["transcoded", "https://gateway.example.com/api/stream/hash?transcoded=true"],
|
|
["duration", "3600"],
|
|
["video", "1920x1080", "30fps", "h264"],
|
|
["m", "video/mp4"],
|
|
["t", "torrent"],
|
|
["t", "video"],
|
|
["t", "streaming"],
|
|
["tcat", "video,streaming"]
|
|
]
|
|
}
|
|
```
|
|
|
|
### Benefits
|
|
|
|
- **Censorship Resistance**: No single point of control for content discovery
|
|
- **Network Effects**: Content spreads naturally through the Nostr social graph
|
|
- **Interoperability**: Standard Nostr clients can display and interact with content
|
|
- **Privacy**: Users maintain control over their identity and data
|
|
|
|
## Performance & Scaling
|
|
|
|
### Optimization Features
|
|
- **Concurrent Downloads**: Multiple parallel piece downloads
|
|
- **Geographic Peer Selection**: Prioritizes nearby peers for faster transfers
|
|
- **Smart Caching**: LRU eviction with configurable cache sizes
|
|
- **Rate Limiting**: Prevents abuse while maintaining performance
|
|
- **Connection Pooling**: Efficient resource utilization
|
|
|
|
### Monitoring & Alerting
|
|
- **Component Health Scores**: 0-100 scoring for all P2P components
|
|
- **Performance Metrics**: Response times, throughput, error rates
|
|
- **Automatic Alerts**: Configurable thresholds for degraded performance
|
|
- **Diagnostic Endpoints**: Detailed system introspection
|
|
|
|
## Contributing
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Make your changes with comprehensive tests
|
|
4. Submit a pull request
|
|
|
|
## License
|
|
|
|
[Add your license information here]
|
|
|
|
## Support
|
|
|
|
- **Issues**: Report bugs and feature requests via GitHub issues
|
|
- **Documentation**: Additional documentation in `/docs`
|
|
- **Community**: [Add community links if available] |