mirror of
https://github.com/asmogo/nws.git
synced 2024-12-13 02:46:22 +00:00
commit
db232f8e16
40
README.md
40
README.md
@ -1,32 +1,37 @@
|
||||
# Nostr Web Services (NWS)
|
||||
|
||||
|
||||
NWS replaces the IP layer in TCP transport using Nostr, enabling a secure connection between
|
||||
clients and backend services.
|
||||
|
||||
Exit nodes are reachable through their [nprofiles](https://nostr-nips.com/nip-19), which are combinations of a Nostr public key and multiple relays.
|
||||
Exit node [domain names](#nws-domain-names) make private services accessible to entry nodes.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A list of Nostr relays that the exit node is connected to.
|
||||
- The Nostr private key of the exit node.
|
||||
|
||||
The exit node utilizes the private key and relay list to generate an [nprofile](https://nostr-nips.com/nip-19), which is printed in the console on startup.
|
||||
|
||||
## Overview
|
||||
|
||||
### NWS main components
|
||||
|
||||
1. **Entry node**: It forwards tcp packets to the exit node using a SOCKS proxy and creates encrypted events for the public key of the exit node.
|
||||
2. **Exit node**: It is a TCP reverse proxy that listens for incoming Nostr subscriptions and forwards the payload to the designated backend service.
|
||||
1. **Exit node**: It is a TCP reverse proxy that listens for incoming Nostr subscriptions and forwards the payload to your designated backend service.
|
||||
2. **Entry node**: It forwards tcp packets to the exit node using a SOCKS proxy and creates encrypted events for the exit node.
|
||||
|
||||
<img src="nws.png" width="900"/>
|
||||
|
||||
### NWS domain names
|
||||
|
||||
There are two types of domain names resolved by NWS entry nodes:
|
||||
1. `.nostr` domains have base32 encoded public key hostnames and base32 encoded relays as subdomains.
|
||||
2. [nprofiles](https://nostr-nips.com/nip-19) are combinations of a Nostr public key and multiple relays.
|
||||
|
||||
Both types of domains will be generated and printed in the console on startup
|
||||
|
||||
## Quickstart
|
||||
|
||||
Running NWS using Docker is recommended. For instructions on running NWS on your local machine, refer to the [Build from source](#build-from-source) section.
|
||||
|
||||
### Using Docker Compose
|
||||
### Using Docker-Compose
|
||||
|
||||
Please navigate to the `docker-compose.yaml` file and set `NOSTR_PRIVATE_KEY` to your own private key.
|
||||
Leaving it empty will generate a new private key on startup.
|
||||
@ -43,27 +48,28 @@ This will start an example environment, including:
|
||||
* [Cashu Nutshell](https://github.com/cashubtc/nutshell) (backend service)
|
||||
* [nostr-relay](https://github.com/scsibug/nostr-rs-relay)
|
||||
|
||||
You can run the following commands to receive your nprofiles:
|
||||
You can run the following commands to receive your NWS domain:
|
||||
|
||||
```bash
|
||||
docker logs exit-https 2>&1 | awk -F'profile=' '{if ($2) print $2}' | awk '{print $1}'
|
||||
docker logs exit-https 2>&1 | awk -F'domain=' '{if ($2) print $2}' | awk '{print $1}'
|
||||
```
|
||||
|
||||
```bash
|
||||
docker logs exit 2>&1 | awk -F'profile=' '{if ($2) print $2}' | awk '{print $1}`
|
||||
docker logs exit 2>&1 | awk -F'domain=' '{if ($2) print $2}' | awk '{print $1}`
|
||||
```
|
||||
|
||||
### Sending Requests to the Entry node
|
||||
### Sending requests to the entry node
|
||||
|
||||
With the log information from the previous step, you can use the following command to send a request to the nprofile:
|
||||
With the log information from the previous step, you can use the following command to send a request to the exit node domain:
|
||||
|
||||
```
|
||||
curl -v -x socks5h://localhost:8882 http://"$(docker logs exit 2>&1 | awk -F'profile=' '{if ($2) print $2}' | awk '{print $1}' | tail -n 1)"/v1/info --insecure
|
||||
curl -v -x socks5h://localhost:8882 http://"$(docker logs exit 2>&1 | awk -F'domain=' '{if ($2) print $2}' | awk '{print $1}' | tail -n 1)"/v1/info --insecure
|
||||
```
|
||||
|
||||
If the nprofile supports TLS, you can choose to connect using https scheme
|
||||
If the exit node supports TLS, you can choose to connect using https scheme
|
||||
|
||||
```
|
||||
curl -v -x socks5h://localhost:8882 https://"$(docker logs exit-https 2>&1 | awk -F'profile=' '{if ($2) print $2}' | awk '{print $1}' | tail -n 1)"/v1/info --insecure
|
||||
curl -v -x socks5h://localhost:8882 https://"$(docker logs exit-https 2>&1 | awk -F'domain=' '{if ($2) print $2}' | awk '{print $1}' | tail -n 1)"/v1/info --insecure
|
||||
```
|
||||
|
||||
When using https, the entry node can be used as a service, since the operator will not be able to see the request data.
|
||||
@ -72,7 +78,7 @@ When using https, the entry node can be used as a service, since the operator wi
|
||||
|
||||
The exit node must be set up to make your services reachable via Nostr.
|
||||
|
||||
### Exit node Configuration
|
||||
### Exit node
|
||||
|
||||
Configuration should be completed using environment variables.
|
||||
Alternatively, you can create a `.env` file in the current working directory with the following content:
|
||||
@ -97,7 +103,7 @@ If your backend services support TLS, your service can now start using TLS encry
|
||||
|
||||
---
|
||||
|
||||
### Entry node Configuration
|
||||
### Entry node
|
||||
|
||||
To run an entry node for accessing NWS services behind exit nodes, use the following command:
|
||||
```
|
||||
|
@ -6,15 +6,15 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var httpsPort int32
|
||||
var httpTarget string
|
||||
|
||||
const (
|
||||
usagePort = "set the https reverse proxy port"
|
||||
usageTarget = "set https reverse proxy target (your local service)"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
var httpsPort int32
|
||||
var httpTarget string
|
||||
rootCmd := &cobra.Command{Use: "exit", Run: startExitNode}
|
||||
rootCmd.Flags().Int32VarP(&httpsPort, "port", "p", 0, usagePort)
|
||||
rootCmd.Flags().StringVarP(&httpTarget, "target", "t", "", usageTarget)
|
||||
@ -25,9 +25,19 @@ func main() {
|
||||
}
|
||||
|
||||
// updateConfigFlag updates the configuration with the provided flags.
|
||||
func updateConfigFlag(cfg *config.ExitConfig) {
|
||||
func updateConfigFlag(cmd *cobra.Command, cfg *config.ExitConfig) error {
|
||||
|
||||
httpsPort, err := cmd.Flags().GetInt32("port")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
httpTarget, err := cmd.Flags().GetString("target")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.HttpsPort = httpsPort
|
||||
cfg.HttpsTarget = httpTarget
|
||||
return nil
|
||||
}
|
||||
|
||||
func startExitNode(cmd *cobra.Command, args []string) {
|
||||
@ -37,7 +47,7 @@ func startExitNode(cmd *cobra.Command, args []string) {
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
updateConfigFlag(cfg)
|
||||
updateConfigFlag(cmd, cfg)
|
||||
ctx := cmd.Context()
|
||||
exitNode := exit.NewExit(ctx, cfg)
|
||||
exitNode.ListenAndServe(ctx)
|
||||
|
92
exit/exit.go
92
exit/exit.go
@ -1,19 +1,24 @@
|
||||
package exit
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"encoding/base32"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"strings"
|
||||
|
||||
"github.com/asmogo/nws/config"
|
||||
"github.com/asmogo/nws/netstr"
|
||||
"github.com/asmogo/nws/protocol"
|
||||
"github.com/asmogo/nws/socks5"
|
||||
"github.com/btcsuite/btcd/btcec/v2"
|
||||
"github.com/btcsuite/btcd/btcec/v2/schnorr"
|
||||
"github.com/nbd-wtf/go-nostr"
|
||||
"github.com/nbd-wtf/go-nostr/nip04"
|
||||
"github.com/nbd-wtf/go-nostr/nip19"
|
||||
"github.com/puzpuzpuz/xsync/v3"
|
||||
"golang.org/x/net/context"
|
||||
"log/slog"
|
||||
"net"
|
||||
)
|
||||
|
||||
const (
|
||||
@ -100,9 +105,13 @@ func NewExit(ctx context.Context, exitNodeConfig *config.ExitConfig) *Exit {
|
||||
}
|
||||
exit.relays = append(exit.relays, relay)
|
||||
fmt.Printf("added relay connection to %s\n", relayUrl)
|
||||
}
|
||||
|
||||
slog.Info("created exit node", "profile", profile)
|
||||
}
|
||||
domain, err := exit.getDomain()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
slog.Info("created exit node", "profile", profile, "domain", domain)
|
||||
// setup subscriptions
|
||||
err = exit.setSubscriptions(ctx)
|
||||
if err != nil {
|
||||
@ -111,6 +120,51 @@ func NewExit(ctx context.Context, exitNodeConfig *config.ExitConfig) *Exit {
|
||||
return exit
|
||||
}
|
||||
|
||||
// getDomain returns the domain string used by the Exit node for communication with the Nostr relays.
|
||||
// It concatenates the relay URLs using base32 encoding with no padding, separated by dots.
|
||||
// The resulting domain is then appended with the base32 encoded public key obtained using the configured Nostr private key.
|
||||
// The final domain string is converted to lowercase and returned.
|
||||
// If any errors occur during the process, they are returned along with an
|
||||
func (e *Exit) getDomain() (string, error) {
|
||||
var domain string
|
||||
// first lets build the subdomains
|
||||
for _, relayUrl := range e.config.NostrRelays {
|
||||
if domain == "" {
|
||||
domain = base32.HexEncoding.WithPadding(base32.NoPadding).EncodeToString([]byte(relayUrl))
|
||||
} else {
|
||||
domain = fmt.Sprintf("%s.%s", domain, base32.HexEncoding.WithPadding(base32.NoPadding).EncodeToString([]byte(relayUrl)))
|
||||
}
|
||||
}
|
||||
// create base32 encoded public key
|
||||
decoded, err := GetPublicKeyBase32(e.config.NostrPrivateKey)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// use public key as host. add TLD
|
||||
domain = strings.ToLower(fmt.Sprintf("%s.%s.nostr", domain, decoded))
|
||||
return domain, nil
|
||||
}
|
||||
|
||||
// GetPublicKeyBase32 decodes the private key string from hexadecimal format
|
||||
// and returns the base32 encoded public key obtained using the provided private key.
|
||||
// The base32 encoding has no padding. If there is an error decoding the private key
|
||||
// or generating the public key, an error is returned.
|
||||
//
|
||||
// Parameters:
|
||||
// - sk: The private key string in hexadecimal format
|
||||
//
|
||||
// Returns:
|
||||
// - The base32 encoded public key as a string
|
||||
// - Any error that occurred during the process
|
||||
func GetPublicKeyBase32(sk string) (string, error) {
|
||||
b, err := hex.DecodeString(sk)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, pk := btcec.PrivKeyFromBytes(b)
|
||||
return base32.HexEncoding.WithPadding(base32.NoPadding).EncodeToString(schnorr.SerializePubKey(pk)), nil
|
||||
}
|
||||
|
||||
// setSubscriptions sets up subscriptions for the Exit node to receive incoming events from the specified relays.
|
||||
// It first obtains the public key using the configured Nostr private key.
|
||||
// Then it calls the `handleSubscription` method to open a subscription to the relays with the specified filters.
|
||||
@ -140,7 +194,8 @@ func (e *Exit) handleSubscription(ctx context.Context, pubKey string, since nost
|
||||
Since: &since,
|
||||
Tags: nostr.TagMap{
|
||||
"p": []string{pubKey},
|
||||
}},
|
||||
},
|
||||
},
|
||||
})
|
||||
e.incomingChannel = incomingEventChannel
|
||||
return nil
|
||||
@ -182,9 +237,9 @@ func (e *Exit) processMessage(ctx context.Context, msg nostr.IncomingEvent) {
|
||||
}
|
||||
switch protocolMessage.Type {
|
||||
case protocol.MessageConnect:
|
||||
e.handleConnect(ctx, msg, protocolMessage, false)
|
||||
e.handleConnect(ctx, msg, protocolMessage)
|
||||
case protocol.MessageConnectReverse:
|
||||
e.handleConnectReverse(ctx, protocolMessage, false)
|
||||
e.handleConnectReverse(protocolMessage)
|
||||
case protocol.MessageTypeSocks5:
|
||||
e.handleSocks5ProxyMessage(msg, protocolMessage)
|
||||
}
|
||||
@ -197,7 +252,10 @@ func (e *Exit) processMessage(ctx context.Context, msg nostr.IncomingEvent) {
|
||||
// If the connection cannot be established, it logs an error and returns.
|
||||
// It then stores the connection in the nostrConnectionMap and creates two goroutines
|
||||
// to proxy the data between the connection and the backend.
|
||||
func (e *Exit) handleConnect(ctx context.Context, msg nostr.IncomingEvent, protocolMessage *protocol.Message, isTLS bool) {
|
||||
func (e *Exit) handleConnect(
|
||||
ctx context.Context,
|
||||
msg nostr.IncomingEvent,
|
||||
protocolMessage *protocol.Message) {
|
||||
e.mutexMap.Lock(protocolMessage.Key.String())
|
||||
defer e.mutexMap.Unlock(protocolMessage.Key.String())
|
||||
receiver, err := nip19.EncodeProfile(msg.PubKey, []string{msg.Relay.String()})
|
||||
@ -211,12 +269,7 @@ func (e *Exit) handleConnect(ctx context.Context, msg nostr.IncomingEvent, proto
|
||||
netstr.WithUUID(protocolMessage.Key),
|
||||
)
|
||||
var dst net.Conn
|
||||
if isTLS {
|
||||
conf := tls.Config{InsecureSkipVerify: true}
|
||||
dst, err = tls.Dial("tcp", e.config.BackendHost, &conf)
|
||||
} else {
|
||||
dst, err = net.Dial("tcp", e.config.BackendHost)
|
||||
}
|
||||
dst, err = net.Dial("tcp", e.config.BackendHost)
|
||||
if err != nil {
|
||||
slog.Error("could not connect to backend", "error", err)
|
||||
return
|
||||
@ -228,7 +281,7 @@ func (e *Exit) handleConnect(ctx context.Context, msg nostr.IncomingEvent, proto
|
||||
go socks5.Proxy(connection, dst, nil)
|
||||
}
|
||||
|
||||
func (e *Exit) handleConnectReverse(ctx context.Context, protocolMessage *protocol.Message, isTLS bool) {
|
||||
func (e *Exit) handleConnectReverse(protocolMessage *protocol.Message) {
|
||||
e.mutexMap.Lock(protocolMessage.Key.String())
|
||||
defer e.mutexMap.Unlock(protocolMessage.Key.String())
|
||||
connection, err := net.Dial("tcp", protocolMessage.Destination)
|
||||
@ -236,12 +289,7 @@ func (e *Exit) handleConnectReverse(ctx context.Context, protocolMessage *protoc
|
||||
return
|
||||
}
|
||||
var dst net.Conn
|
||||
if isTLS {
|
||||
conf := tls.Config{InsecureSkipVerify: true}
|
||||
dst, err = tls.Dial("tcp", e.config.BackendHost, &conf)
|
||||
} else {
|
||||
dst, err = net.Dial("tcp", e.config.BackendHost)
|
||||
}
|
||||
dst, err = net.Dial("tcp", e.config.BackendHost)
|
||||
if err != nil {
|
||||
slog.Error("could not connect to backend", "error", err)
|
||||
return
|
||||
|
@ -3,9 +3,12 @@ package netstr
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base32"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"github.com/asmogo/nws/protocol"
|
||||
"github.com/btcsuite/btcd/btcec/v2/schnorr"
|
||||
"github.com/google/uuid"
|
||||
"github.com/nbd-wtf/go-nostr"
|
||||
"github.com/nbd-wtf/go-nostr/nip04"
|
||||
@ -13,6 +16,7 @@ import (
|
||||
"github.com/samber/lo"
|
||||
"log/slog"
|
||||
"net"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@ -216,6 +220,10 @@ func (nc *NostrConnection) handleNostrWrite(b []byte, err error) (int, error) {
|
||||
// If the prefix is "nprofile", the public key and relays are extracted.
|
||||
// Returns the public key, relays (if any), and any error encountered.
|
||||
func ParseDestination(destination string) (string, []string, error) {
|
||||
// check if destination ends with .nostr
|
||||
if strings.HasSuffix(destination, ".nostr") {
|
||||
return ParseDestinationDomain(destination)
|
||||
}
|
||||
// destination can be npub or nprofile
|
||||
prefix, pubKey, err := nip19.Decode(destination)
|
||||
|
||||
@ -237,6 +245,38 @@ func ParseDestination(destination string) (string, []string, error) {
|
||||
return publicKey, relays, nil
|
||||
}
|
||||
|
||||
func ParseDestinationDomain(destination string) (string, []string, error) {
|
||||
url, err := protocol.Parse(destination)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
if !url.IsDomain {
|
||||
// return "", nil, fmt.Errorf("destination is not a domain")
|
||||
}
|
||||
var subdomains []string
|
||||
split := strings.Split(url.SubName, ".")
|
||||
for _, subdomain := range split {
|
||||
decodedSubDomain, err := base32.HexEncoding.WithPadding(base32.NoPadding).DecodeString(strings.ToUpper(subdomain))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
subdomains = append(subdomains, string(decodedSubDomain))
|
||||
}
|
||||
|
||||
// base32 decode the subdomain
|
||||
decodedPubKey, err := base32.HexEncoding.WithPadding(base32.NoPadding).DecodeString(strings.ToUpper(url.Name))
|
||||
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
pk, err := schnorr.ParsePubKey(decodedPubKey)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
// todo -- check if this is correct
|
||||
return hex.EncodeToString(pk.SerializeCompressed())[2:], subdomains, nil
|
||||
}
|
||||
|
||||
func (nc *NostrConnection) Close() error {
|
||||
nc.cancel()
|
||||
return nil
|
||||
|
@ -8,7 +8,6 @@ import (
|
||||
"github.com/nbd-wtf/go-nostr"
|
||||
"log/slog"
|
||||
"net"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type DialOptions struct {
|
||||
@ -26,7 +25,6 @@ type DialOptions struct {
|
||||
// Finally, it returns the Connection and nil error. If there are any errors, nil connection and the error are returned.
|
||||
func DialSocks(options DialOptions) func(ctx context.Context, net_, addr string) (net.Conn, error) {
|
||||
return func(ctx context.Context, net_, addr string) (net.Conn, error) {
|
||||
addr = strings.ReplaceAll(addr, ".", "")
|
||||
key := nostr.GeneratePrivateKey()
|
||||
connection := NewConnection(ctx,
|
||||
WithPrivateKey(key),
|
||||
|
380
protocol/domain.go
Normal file
380
protocol/domain.go
Normal file
@ -0,0 +1,380 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/net/idna"
|
||||
"golang.org/x/net/publicsuffix"
|
||||
)
|
||||
|
||||
const (
|
||||
ipV6URINotationPrefix = "["
|
||||
ipV6URINotationSuffix = "]"
|
||||
)
|
||||
|
||||
var ErrEmptyURL = errors.New("url to be parsed is empty")
|
||||
|
||||
// URL represents a URL with additional fields and methods.
|
||||
type URL struct {
|
||||
SubName, Name, TLD, Port string
|
||||
IsDomain bool
|
||||
*url.URL
|
||||
}
|
||||
|
||||
// String returns the string representation of the URL.
|
||||
// It includes the scheme if `includeScheme` is true.
|
||||
func (url URL) String(includeScheme bool) string {
|
||||
s := url.URL.String()
|
||||
if !includeScheme {
|
||||
s = RemoveScheme(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Domain returns the domain name of the URL. If includeSub is true and there is a subdomain, it includes the subdomain
|
||||
// in the returned string. Otherwise, it only includes the domain.
|
||||
func (url URL) Domain(includeSub bool) string {
|
||||
if includeSub && url.SubName != "" {
|
||||
return fmt.Sprintf("%s.%s.%s", url.SubName, url.Name, url.TLD)
|
||||
}
|
||||
return fmt.Sprintf("%s.%s", url.Name, url.TLD)
|
||||
}
|
||||
|
||||
// NoWWW returns the domain name without the "www" subdomain.
|
||||
// If the subdomain is not "www" or is empty, it returns the domain name as is.
|
||||
// The returned domain name is a string in the format "subname.name.tld".
|
||||
func (url URL) NoWWW() string {
|
||||
if url.SubName != "www" && url.SubName != "" {
|
||||
return fmt.Sprintf("%s.%s.%s", url.SubName, url.Name, url.TLD)
|
||||
}
|
||||
return fmt.Sprintf("%s.%s", url.Name, url.TLD)
|
||||
}
|
||||
|
||||
// WWW returns the domain name with the "www" subdomain.
|
||||
// If the subdomain is not "www", it returns the domain name as is.
|
||||
// The returned domain name is a string in the format "subname.name.tld".
|
||||
func (url URL) WWW() string {
|
||||
if url.SubName != "" {
|
||||
return fmt.Sprintf("%s.%s.%s", url.SubName, url.Name, url.TLD)
|
||||
}
|
||||
return fmt.Sprintf("%s.%s.%s", "www", url.Name, url.TLD)
|
||||
}
|
||||
|
||||
// HTTPS returns the URL with HTTPS Scheme but leaves the URL itself untouched.
|
||||
func (url URL) HTTPS() string {
|
||||
rememberScheme := url.Scheme
|
||||
url.Scheme = "https"
|
||||
httpsURL := url.String(true)
|
||||
url.Scheme = rememberScheme
|
||||
return httpsURL
|
||||
}
|
||||
|
||||
// StripWWW returns the URL without "www" subdomain, but leaves the URL itself untouched.
|
||||
// This function returns the whole URL with its path, in contrast to NoWWW().
|
||||
func (url URL) StripWWW(includeScheme bool) string {
|
||||
if url.SubName == "www" {
|
||||
return strings.Replace(url.String(includeScheme), "www.", "", 1)
|
||||
}
|
||||
return url.String(includeScheme)
|
||||
}
|
||||
|
||||
// StripQueryParams removes query parameters and fragments from the URL and returns
|
||||
// the URL as a string. If includeScheme is true, it includes the scheme in the returned URL.
|
||||
func (url URL) StripQueryParams(includeScheme bool) string {
|
||||
// Remember the original values of query parameters and fragments
|
||||
rememberRawQuery := url.RawQuery
|
||||
rememberFragment := url.Fragment
|
||||
rememberRawFragment := url.RawFragment
|
||||
|
||||
// Clear the query parameters and fragments
|
||||
url.RawQuery = ""
|
||||
url.RawFragment = ""
|
||||
url.Fragment = ""
|
||||
|
||||
// Get the URL without query parameters
|
||||
urlWithoutQuery := url.String(includeScheme)
|
||||
|
||||
// Restore the original values of query parameters and fragments
|
||||
url.RawQuery = rememberRawQuery
|
||||
url.RawFragment = rememberRawFragment
|
||||
url.Fragment = rememberFragment
|
||||
|
||||
return urlWithoutQuery
|
||||
}
|
||||
|
||||
// IsLocal checks if the URL is a local address.
|
||||
// It returns true if the URL's top-level domain (TLD) is "localhost" or if the URL's
|
||||
// hostname resolves to a loopback IP address.
|
||||
func (url URL) IsLocal() bool {
|
||||
ip := net.ParseIP(strings.TrimPrefix(strings.TrimSuffix(url.Name, ipV6URINotationSuffix), ipV6URINotationPrefix))
|
||||
return url.TLD == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
|
||||
// Parse parses a string representation of a URL and returns a *URL and error.
|
||||
// It mirrors the net/url.Parse function but returns a tld.URL, which contains extra fields.
|
||||
func Parse(urlString string) (*URL, error) {
|
||||
urlString = strings.TrimSpace(urlString)
|
||||
|
||||
// if the url to be parsed is empty after trimming, we return an error
|
||||
if len(urlString) == 0 {
|
||||
return nil, ErrEmptyURL
|
||||
}
|
||||
|
||||
urlString = AddDefaultScheme(urlString)
|
||||
parsedURL, err := url.Parse(urlString)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not parse url: %w", err)
|
||||
}
|
||||
// always lowercase subdomain.domain.tld (host property)
|
||||
parsedURL.Host = strings.ToLower(parsedURL.Host)
|
||||
if parsedURL.Host == "" {
|
||||
return &URL{URL: parsedURL}, nil
|
||||
}
|
||||
dom, port := domainPort(parsedURL.Host)
|
||||
var domName, tld, sub string
|
||||
ip := net.ParseIP(strings.TrimPrefix(strings.TrimSuffix(dom, ipV6URINotationSuffix), ipV6URINotationPrefix))
|
||||
switch {
|
||||
case ip != nil:
|
||||
domName = dom
|
||||
case dom == "localhost":
|
||||
tld = dom
|
||||
default:
|
||||
etld1, err := publicsuffix.EffectiveTLDPlusOne(dom)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to extract eTLD+1: %w", err)
|
||||
}
|
||||
i := strings.Index(etld1, ".")
|
||||
domName = etld1[0:i]
|
||||
tld = etld1[i+1:]
|
||||
sub = ""
|
||||
if rest := strings.TrimSuffix(dom, "."+etld1); rest != dom {
|
||||
sub = rest
|
||||
}
|
||||
}
|
||||
urlString, err = idna.ToASCII(dom)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to convert domain to ASCII: %w", err)
|
||||
}
|
||||
return &URL{
|
||||
SubName: sub,
|
||||
Name: domName,
|
||||
TLD: tld,
|
||||
Port: port,
|
||||
URL: parsedURL,
|
||||
IsDomain: IsDomainName(urlString),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// FromParsed mirrors the net/url.Parse function,
|
||||
// but instead of returning a *url.URL, it returns a *URL,
|
||||
// which is a struct that contains additional fields.
|
||||
//
|
||||
// The function first checks if the parsedUrl.Host field is empty.
|
||||
// If it is empty, it returns a *URL with the URL field set to parsedUrl
|
||||
// and all other fields set to their zero values.
|
||||
//
|
||||
// If the parsedUrl.Host field is not empty, it extracts the domain and port
|
||||
// using the domainPort function.
|
||||
//
|
||||
// It then calculates the effective top-level domain plus one (etld+1)
|
||||
// using the publicsuffix.EffectiveTLDPlusOne function.
|
||||
//
|
||||
// The etld+1 is then split into the domain name (domName) and the top-level domain (tld).
|
||||
//
|
||||
// It further determines the subdomain (sub) by checking if the domain is a subdomain of the etld+1.
|
||||
//
|
||||
// The domain name (domName) is then converted to ASCII using the idna.ToASCII function.
|
||||
//
|
||||
// Finally, it returns a *URL with the extracted values and the URL field set to parsedUrl.
|
||||
// The IsDomain field is set to the result of the IsDomainName function called with the ASCII domain name.
|
||||
// The SubName field is set to sub, the Name field is set to domName, and the T.
|
||||
func FromParsed(parsedURL *url.URL) (*URL, error) {
|
||||
if parsedURL.Host == "" {
|
||||
return &URL{URL: parsedURL}, nil
|
||||
}
|
||||
dom, port := domainPort(parsedURL.Host)
|
||||
// etld+1
|
||||
etld1, err := publicsuffix.EffectiveTLDPlusOne(dom)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to extract eTLD+1: %w", err)
|
||||
}
|
||||
// convert to domain name, and tld
|
||||
i := strings.Index(etld1, ".")
|
||||
domName := etld1[0:i]
|
||||
tld := etld1[i+1:]
|
||||
// and subdomain
|
||||
sub := ""
|
||||
if rest := strings.TrimSuffix(dom, "."+etld1); rest != dom {
|
||||
sub = rest
|
||||
}
|
||||
asciiDom, err := idna.ToASCII(dom)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to convert domain to ASCII: %w", err)
|
||||
}
|
||||
return &URL{
|
||||
SubName: sub,
|
||||
Name: domName,
|
||||
TLD: tld,
|
||||
Port: port,
|
||||
URL: parsedURL,
|
||||
IsDomain: IsDomainName(asciiDom),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// domainPort extracts the domain and port from the host part of a URL.
|
||||
// If the host contains a port, it returns the domain without the port and the port as strings.
|
||||
// If the host does not contain a port, it returns the domain and an empty string for the port.
|
||||
// If the host is all numeric characters, it returns the host itself and an empty string for the port.
|
||||
// Note that the net/url package should prevent the string from being all numeric characters.
|
||||
func domainPort(host string) (string, string) {
|
||||
for i := len(host) - 1; i >= 0; i-- {
|
||||
if host[i] == ':' {
|
||||
return host[:i], host[i+1:]
|
||||
} else if host[i] < '0' || host[i] > '9' {
|
||||
return host, ""
|
||||
}
|
||||
}
|
||||
// will only land here if the string is all digits,
|
||||
// net/url should prevent that from happening
|
||||
return host, ""
|
||||
}
|
||||
|
||||
// IsDomainName checks if a string represents a valid domain name.
|
||||
//
|
||||
// It follows the rules specified in RFC 1035 and RFC 3696 for domain name validation.
|
||||
//
|
||||
// The input string is first processed with the RemoveScheme function to remove any scheme prefix.
|
||||
// The domain name is then split into labels using the dot separator.
|
||||
// The function checks that the number of labels is at least 2 and that the total length of the string is between 1 and
|
||||
// 254 characters.
|
||||
//
|
||||
// The function iterates over the characters of the string and performs checks based on the character type.
|
||||
// Valid characters include letters (a-zA-Z), digits (0-9), underscore (_), and hyphen (-).
|
||||
// Each label can contain up to 63 characters and the last label cannot end with a hyphen.
|
||||
// The function also checks that the byte before a dot or a hyphen is not a dot or a hyphen, respectively.
|
||||
// Non-numeric characters are tracked to ensure the presence of at least one non-numeric character in the domain name.
|
||||
//
|
||||
// If any of the checks fail, the function returns false. Otherwise, it returns true.
|
||||
//
|
||||
// Example usage:
|
||||
// s := "mail.google.com"
|
||||
// isValid := IsDomainName(s).
|
||||
func IsDomainName(name string) bool { //nolint:cyclop
|
||||
name = RemoveScheme(name)
|
||||
// See RFC 1035, RFC 3696.
|
||||
// Presentation format has dots before every label except the first, and the
|
||||
// terminal empty label is optional here because we assume fully-qualified
|
||||
// (absolute) input. We must therefore reserve space for the first and last
|
||||
// labels' length octets in wire format, where they are necessary and the
|
||||
// maximum total length is 255.
|
||||
// So our _effective_ maximum is 253, but 254 is not rejected if the last
|
||||
// character is a dot.
|
||||
split := strings.Split(name, ".")
|
||||
|
||||
// Need a TLD and a domain.
|
||||
if len(split) < 2 { //nolint:gomnd
|
||||
return false
|
||||
}
|
||||
l := len(name)
|
||||
if l == 0 || l > 254 || l == 254 && name[l-1] != '.' {
|
||||
return false
|
||||
}
|
||||
|
||||
last := byte('.')
|
||||
nonNumeric := false // true once we've seen a letter or hyphen
|
||||
partlen := 0
|
||||
for i := 0; i < len(name); i++ {
|
||||
char := name[i]
|
||||
switch {
|
||||
default:
|
||||
return false
|
||||
case 'a' <= char && char <= 'z' || 'A' <= char && char <= 'Z' || char == '_':
|
||||
nonNumeric = true
|
||||
partlen++
|
||||
case '0' <= char && char <= '9':
|
||||
// fine
|
||||
partlen++
|
||||
case char == '-':
|
||||
// Byte before dash cannot be dot.
|
||||
if last == '.' {
|
||||
return false
|
||||
}
|
||||
partlen++
|
||||
nonNumeric = true
|
||||
case char == '.':
|
||||
// Byte before dot cannot be dot, dash.
|
||||
if last == '.' || last == '-' {
|
||||
return false
|
||||
}
|
||||
if partlen > 63 || partlen == 0 {
|
||||
return false
|
||||
}
|
||||
partlen = 0
|
||||
}
|
||||
last = char
|
||||
}
|
||||
if last == '-' || partlen > 63 {
|
||||
return false
|
||||
}
|
||||
|
||||
return nonNumeric
|
||||
}
|
||||
|
||||
// RemoveScheme removes the scheme from a URL string.
|
||||
// If the URL string includes a scheme (e.g., "http://"), the scheme will be removed and the remaining string will be returned.
|
||||
// If the URL string includes a default scheme (e.g., "//"), the default scheme will be removed and the remaining string will be returned.
|
||||
// If the URL string does not include a scheme, the original string will be returned unchanged.
|
||||
func RemoveScheme(s string) string {
|
||||
if strings.Contains(s, "://") {
|
||||
return removeScheme(s)
|
||||
}
|
||||
if strings.Contains(s, "//") {
|
||||
return removeDefaultScheme(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// add default scheme if string does not include a scheme.
|
||||
func AddDefaultScheme(s string) string {
|
||||
if !strings.Contains(s, "//") ||
|
||||
(!strings.Contains(s, "//") && !strings.Contains(s, ":") && !strings.Contains(s, "@")) {
|
||||
return addDefaultScheme(s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func AddScheme(s, scheme string) string {
|
||||
if scheme == "" {
|
||||
return AddDefaultScheme(s)
|
||||
}
|
||||
if strings.Index(s, "//") == -1 {
|
||||
return fmt.Sprintf("%s://%s", scheme, s)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// addDefaultScheme returns a new string with a default scheme added.
|
||||
// The default scheme format is "//<original_string>".
|
||||
func addDefaultScheme(s string) string {
|
||||
return fmt.Sprintf("//%s", s)
|
||||
}
|
||||
|
||||
// removeDefaultScheme removes the default scheme from a string.
|
||||
func removeDefaultScheme(s string) string {
|
||||
return s[index(s, "//"):]
|
||||
}
|
||||
|
||||
func removeScheme(s string) string {
|
||||
return s[index(s, "://"):]
|
||||
}
|
||||
|
||||
// index returns the starting index of the first occurrence of the specified scheme in the given string.
|
||||
// If the scheme is not found, it returns -1.
|
||||
// The returned index is incremented by the length of the scheme to obtain the starting position of the remaining string.
|
||||
func index(s, scheme string) int {
|
||||
return strings.Index(s, scheme) + len(scheme)
|
||||
}
|
45
protocol/domain_test.go
Normal file
45
protocol/domain_test.go
Normal file
@ -0,0 +1,45 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type args struct {
|
||||
s string
|
||||
}
|
||||
|
||||
type parseTest struct {
|
||||
name string
|
||||
args args
|
||||
want *URL
|
||||
wantErr bool
|
||||
}
|
||||
|
||||
func TestParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
for _, test := range createParseTests() {
|
||||
testCopy := test
|
||||
t.Run(testCopy.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
got, err := Parse(testCopy.args.s)
|
||||
if (err != nil) != testCopy.wantErr {
|
||||
t.Errorf("Parse() error = %v, wantErr %v", err, testCopy.wantErr)
|
||||
return
|
||||
}
|
||||
if !reflect.DeepEqual(got, testCopy.want) {
|
||||
t.Errorf("Parse() got = %v, want %v", got, testCopy.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func createParseTests() []parseTest {
|
||||
return []parseTest{
|
||||
{name: "1", args: args{s: "http://D1Q78S3J78NIURJFEDQ74BJQCLH6AP35CKN66R3FELI0.9B7NTQSU4PBM2JJQJ0CMGHUENQON4GB28RLGQCH3D3NK2AQVFE70.nostr"}, want: &URL{IsDomain: true, TLD: "nostr", Name: "9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70", SubName: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0", URL: &url.URL{Host: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0.9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70.nostr", Scheme: "http"}}, wantErr: false}, //nolint:lll
|
||||
{name: "1", args: args{s: "http://d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0.9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70.nostr"}, want: &URL{IsDomain: true, TLD: "nostr", Name: "9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70", SubName: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0", URL: &url.URL{Host: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0.9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70.nostr", Scheme: "http"}}, wantErr: false}, //nolint:lll
|
||||
{name: "1", args: args{s: "https://d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0.9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70.nostr"}, want: &URL{IsDomain: true, TLD: "nostr", Name: "9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70", SubName: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0", URL: &url.URL{Host: "d1q78s3j78niurjfedq74bjqclh6ap35ckn66r3feli0.9b7ntqsu4pbm2jjqj0cmghuenqon4gb28rlgqch3d3nk2aqvfe70.nostr", Scheme: "https"}}, wantErr: false}, //nolint:lll
|
||||
|
||||
}
|
||||
}
|
Loading…
Reference in New Issue
Block a user