robosats/scripts/traditional/README.md

# Robosats traditional environment

Robosats backend development and testing without docker and containers.

Binaries needed:
* postgresql (`postgres`, `initdb`, `psql`)
* redis (`redis-server`)
* bitcoin (`bitcoind`, `bitcoin-cli`)
* cln (`lightningd`, `lightning-cli`, `holdinvoice`)
* lnd (`lnd`, `lncli`)

## Preparation

Postgresql and redis can be found in all linux distros and bsds.

Some distros do not put postgresql binaries in `PATH`, if this is the
case simply link them somewhere in your `PATH`.

Example on debian:
```
ln -sf /usr/lib/postgresql/16/bin/postgres ~/.local/bin/
ln -sf /usr/lib/postgresql/16/bin/initdb ~/.local/bin/
ln -sf /usr/lib/postgresql/16/bin/psql ~/.local/bin/
```

Bitcoin nodes if not already installed need to be manually downloaded.
* bitcoin core binaries can be found here: https://bitcoincore.org/en/download
* cln binaries can be found here: https://github.com/ElementsProject/lightning/releases
* holdinvoice binary can be found here: https://github.com/daywalker90/holdinvoice/releases
* lnd binaries can be found here: https://github.com/lightningnetwork/lnd/releases

Example preparation:
```
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install -r requirements_dev.txt
$ pip install -r requirements.txt

$ mkdir regtest
$ mkdir regtest/programs
$ cd regtest/programs
$ mkdir bitcoin cln lnd
# download bitcoin, cln (and holdinvoice) and lnd binaries

# if you want to use roboauto
# still in regtest/programs
$ git clone https://github.com/jerryfletcher21/roboauto
$ cd roboauto
$ pip install -r requirements.txt
$ pip install .
```

## env file

```
$ cp .env-sample .env
```
Edit `.env`, both robosats and regtest scripts will read from there.

Variables to change:
```
LNVENDOR = "CLN"
# LNVENDOR = "LND"

LND_DIR = "regtest/nodes/lnd-coord/"
MACAROON_PATH = "data/chain/bitcoin/regtest/admin.macaroon"
CLN_DIR = "regtest/nodes/cln-coord/regtest/"
BITCOIND_RPCPORT = "18443"
POSTGRES_DB = "robosats"
POSTGRES_USER = "robosats"
POSTGRES_PASSWORD = "robosats"
USE_TOR = False
LOG_TO_CONSOLE = True
LOGGER_LEVEL = "INFO"
```

Variables to add:
```
DEVELOPMENT = True
TESTING = True

LNVENDOR_ROBOT = "LND"
# LNVENDOR_ROBOT = "CLN"

BITCOIND_BIN = "regtest/programs/bitcoin/bitcoin-27.1/bin/bitcoind"
BITCOIN_CLI_BIN = "regtest/programs/bitcoin/bitcoin-27.1/bin/bitcoin-cli"
LIGHTNINGD_BIN = "regtest/programs/cln/usr/bin/lightningd"
LIGHTNING_CLI_BIN = "regtest/programs/cln/usr/bin/lightning-cli"
HOLDINVOICE_PLUGIN_BIN = "regtest/programs/cln/holdinvoice"
LND_BIN = "regtest/programs/lnd/lnd-linux-amd64-v0.18.1-beta/lnd"
LNCLI_BIN = "regtest/programs/lnd/lnd-linux-amd64-v0.18.1-beta/lncli"
ROBOAUTO_GIT_DIR = "regtest/programs/roboauto"

REGTEST_NODES_DIR = "regtest/nodes"
REGTEST_SERVICES_DIR = "regtest/services"
REGTEST_LOGS_DIR = "regtest/logs"
GNUPG_DIR = "regtest/services/gnupg"

BITCOIND_TEST_RPCUSER = "robodev"
BITCOIND_TEST_RPCPASSWORD = "robodev"
CLN_TEST_COORD_LISTEN_PORT = 9785
LND_TEST_COORD_LISTEN_PORT = 9885
LND_TEST_COORD_MACAROON_PATH = "regtest/nodes/lnd-coord/data/chain/bitcoin/regtest/admin.macaroon"
LND_TEST_ROBOT_MACAROON_PATH = "regtest/nodes/lnd-user/data/chain/bitcoin/regtest/admin.macaroon"
LND_TEST_ROBOT_REST_PORT = 8180
LND_TEST_COORD_REST_PORT = 8181
```

Paths can be relative or absolute. Binaries should be paths, they are
not resolved with `PATH`.

Roboauto can be disabled by not setting `ROBOAUTO_GIT_DIR` or setting it
to `false`.

If some ports are already in use, change their value.

To check which port are already in use, `netstat -tulnp` with root
privileges can be used.

For example if there is alread an instance of postgresql running on the
default port, change `POSTGRES_PORT = "5433"`.

## Usage

For development and testing two scripts are provided:
* `regtest-services` for non bitcoin related services
* `regtest-nodes` for bitcoin and lightning nodes

`regtest-services` sets up the database and manages the services.

Everything is done locally, so no root privileges/service managers are
needed.

`regtest-nodes` is a script that should be sourced, it sets up a regtest
environment, with bitcoin core, cln, lnd and roboauto, connecting them
and creating channels. It then exposes the functions `btc_reg`,
`cln_coord`, `cln_user`, `lnd_coord`, `lnd_user`, `ra_reg` to interact
with the nodes and roboauto.

If the script is sourced in a `bash` shell, it will also source
completions for all the functions.

`regtest-nodes` can also be run without arguments to simply expose the
functions to start and remove the nodes and to create the channels
between them, without setting up a specific environment.

Setup:
```
# after having changed .env file

$ . venv/bin/activate

# generate cln and lnd grpc
$ scripts/generate_grpc.sh

$ scripts/traditional/regtest-services postgres-setup

$ scripts/traditional/regtest-services postgres-database

# postgres-database will create the database specified by
# POSTGRES_DB in .env, it can be run multiple times with
# different values of POSTGRES_DB for different databases
```

Testing:
```
# edit .env setting LNVENDOR to either "CLN" or "LND"
# LNVENDOR_ROBOT while running tests should be set to "LND"

# in the main window
$ . venv/bin/activate
$ . scripts/traditional/regtest-nodes test

# in a secondary window
$ . venv/bin/activate
# can be stopped with Control-C
$ scripts/traditional/regtest-services test

# back in the main window
$ python3 manage.py test

# after having run the tests run
$ robosats_regtest_stop_and_remove_all
# to remove the nodes, they will be recreated when
# running the tests again

# the tests should be run with a clean database so if you have already run
# the server and want to keep the database, either use a different value
# of POSTGRES_DB or use a different directory by moving
# regtest/services/postgres somewhere and the moving it back when you want
# to run the server again
```

Development:
```
# edit .env setting LNVENDOR to either "CLN" or "LND"
# and LNVENDOR_ROBOT to either "CLN" or "LND"

# in the main window
$ . venv/bin/activate
$ . scripts/traditional/regtest-nodes server

# in a secondary window
$ . venv/bin/activate
# can be stopped with Control-C
$ scripts/traditional/regtest-services server

# to see the output of the django runserver command
# in a third window
$ tail -f regtest/logs/runserver

# if roboauto is active, use it to test the backend
# back in the main window
$ ra_reg --help
...
$ ra_reg create-order "$(ra_reg generate-robot --loc)" type=buy currency=btc min_amount=0.01 max_amount=0.02 payment_method="On-Chain BTC" premium=-1.5
...
$ ra_reg take-order $(ra_reg generate-robot --loc) order-id
...
$ ra_reg escrow-pay RobotName
...
```