robosats/setup.md

Set up

Attention: to use RoboSats you do not need to run the stack, simply visit the website and that's it! This setup guide is intended for developer contributors and platform operators.

Frontend Development Only

Running the full stack is not easy, since RoboSats needs of many services. However, contributing to the frontend development can be done with a minimal setup!

Set up time ~10 min. Tested in Chromium and Firefox in Ubuntu.

---

0 - git clone {robosats-project}

1 - cd robosats/frontend

2 - npm install

3 - npm run dev (leave it running)

4 - On another terminal npm install -g http-server

5 - Then run http-server "robosats/frontend/static/frontend/

6 - Install Requestly extension in your browser, it's a lightweight proxy. We want to use it so our browser grabs our local main.js instead of the remote. There are many alternatives to Requestly (be aware that Requestly might not respect your privacy. Didn't research it).

7 - Pick a RoboSats backend to test the new frontend: e.g. "robosats.onion.moe", or "unsafe.testnet.robosats.com". You can also use the onion services also if you are using Brave or Tor Browser (untested!)

8 - Open Requestly extension and add a new redirect rule. Make "{robosats-site}/static/frontend/main.js" redirect to "127.0.0.1:8080/main.js" and save the changes.

---

You are ready to go! Edit the frontend code in /frontend/src/ to make the changes you want. Within a few seconds, the npm run dev process will pack the code into the local main.js. Visit your selected {robosats-site} and you will see your new awesome frontend! :)

Every time you save changes to files in /frontend/src you will be able to see them in your browser after a few seconds using force refresh (Ctrl+Shift+R). If you need to edit CSS or other static files, simply add them to Requestly in the same way.

Full Stack Development

The Easy Way: Docker-compose (-dev containers running on testnet)

*Set up time, anywhere between ~45 min and 1 full day (depending on experience, whether you have a copy of the testnet blockchain, etc). Tested in Ubuntu.

Spinning up docker for the first time

docker-compose build --no-cache
docker-compose up -d
docker exec -it django-dev python3 manage.py makemigrations
docker exec -it django-dev python3 manage.py migrate
docker exec -it django-dev python3 manage.py createsuperuser
docker-compose restart

Copy the .env-sample file into .env and check the environmental variables are right for your development.

(optional, if error) Install LND python dependencies

Depending on your system your dev setup might already be good to start working. However, it might happen that when mounting "." into "/src/usr/robosats" in the docker-compose.yml, it deletes the lnd grpc files that where previously generated during the docker build. If this is the case, run the following:

cd api/lightning
pip install grpcio grpcio-tools googleapis-common-protos
git clone https://github.com/googleapis/googleapis.git
curl -o lightning.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/lightning.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. lightning.proto

We also use the Invoices and Router subservices for invoice validation and payment routing.

curl -o invoices.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/invoicesrpc/invoices.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. invoices.proto
curl -o router.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/routerrpc/router.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. router.proto

Generated files can be automatically patched for relative imports like this:

sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/router_pb2.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/invoices_pb2.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/router_pb2_grpc.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/lightning_pb2_grpc.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/invoices_pb2_grpc.py

All set!

Commands you will need to startup:

• Spinning up the docker orchestration: docker-compose up -d

• Then monitor in a terminal the Django dev docker service docker attach django-dev

• And the NPM dev docker service docker attach npm-dev

• You could also just check all services logs docker-compose logs -f

You will need these commands also often or eventually:

• Unlock or 'create' the lnd node docker exec -it lnd-dev lncli unlock

• Create p2wkh addresses docker exec -it lnd-dev lncli --network=testnet newaddress p2wkh

• Wallet balance docker exec -it lnd-dev lncli --network=testnet walletbalance

• Connect to peer docker exec -it lnd-dev lncli --network=testnet connect node_id@ip:9735

• Open channel docker exec -it lnd-dev lncli --network=testnet openchannel node_id --local_amt LOCAL_AMT --push_amt PUSH_AMT

RoboSats development site should be accessible on localhost:8000

The harder way (deprecated)

Setting up the stack on your base system is not supported by this guide anymore. Without dockerization of all services there is a thin chance you manage to have the platform running in less than 2 full days of work. These are legacy instructions, avoid them.

Django development environment

Install Python and pip

sudo apt install python3 python3 pip

Install virtual environments

pip install virtualenvwrapper

Add to .bashrc

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS=' -p /usr/bin/python3 '
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh

Reload startup file

source ~/.bashrc

Create virtual environment

mkvirtualenv robosats_env

Activate environment

workon robosats_env

Install Django and Restframework

pip3 install django djangorestframework

Install Django admin relational links

pip install django-admin-relation-links

Install dependencies for websockets

Install Redis apt-get install redis-server Test redis-server redis-cli ping

Install python dependencies

pip install channels
pip install django-redis
pip install channels-redis

Install Celery for Django tasks

pip install celery
pip install django-celery-beat
pip install django-celery-results

Start up celery worker celery -A robosats worker --beat -l info -S django

Django 3.2.11 at the time of writting Celery 5.2.3

Launch the local development node

python3 manage.py makemigrations
python3 manage.py migrate
python3 manage.py runserver

Install other python dependencies

pip install robohash
pip install python-decouple
pip install ring

Install LND python dependencies

cd api/lightning
pip install grpcio grpcio-tools googleapis-common-protos
git clone https://github.com/googleapis/googleapis.git
curl -o lightning.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/lightning.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. lightning.proto

We also use the Invoices and Router subservices for invoice validation and payment routing.

curl -o invoices.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/invoicesrpc/invoices.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. invoices.proto
curl -o router.proto -s https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/routerrpc/router.proto
python3 -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. router.proto

Relative imports are not working at the moment, so some editing is needed in api/lightning files lightning_pb2_grpc.py, invoices_pb2_grpc.py, invoices_pb2.py, router_pb2_grpc.py and router_pb2.py.

For example in lightning_pb2_grpc.py , add "from . " :

import lightning_pb2 as lightning__pb2

to

from . import lightning_pb2 as lightning__pb2

Same for every other file.

Generated files can be automatically patched like this:

sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/router_pb2.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/invoices_pb2.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/router_pb2_grpc.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/lightning_pb2_grpc.py
sed -i 's/^import .*_pb2 as/from . \0/' api/lightning/invoices_pb2_grpc.py

React development environment

Install npm

sudo apt install npm

npm packages we use

cd frontend
npm init -y
npm i webpack webpack-cli --save-dev
npm i @babel/core babel-loader @babel/preset-env @babel/preset-react --save-dev
npm i react react-dom --save-dev
npm install @material-ui/core
npm install @babel/plugin-proposal-class-properties
npm install react-router-dom@5.2.0
npm install @material-ui/icons
npm install material-ui-image
npm install @mui/system @emotion/react @emotion/styled
npm install react-native
npm install react-native-svg
npm install react-qr-code
npm install @mui/material
npm install websocket
npm install react-countdown
npm install @mui/icons-material
npm install @mui/x-data-grid
npm install react-responsive
npm install react-qr-reader

Note we are using mostly MaterialUI V5 (@mui/material) but Image loading from V4 (@material-ui/core) extentions (so both V4 and V5 are needed)

Launch

from frontend/ directory npm run dev

Robosats background threads.

There is 3 processes that run asynchronously: two admin commands and a celery beat scheduler. The celery worker will run the task of caching external API market prices and cleaning(deleting) the generated robots that were never used. celery -A robosats worker --beat -l debug -S django

The admin commands are used to keep an eye on the state of LND hold invoices and check whether orders have expired

python3 manage.py follow_invoices
python3 manage.py clean_order

It might be best to set up system services to continuously run these background processes.

Follow invoices admin command as system service

Create /etc/systemd/system/follow_invoices.service and edit with:

[Unit]
Description=RoboSats Follow LND Invoices
After=lnd.service
StartLimitIntervalSec=0

[Service]
WorkingDirectory=/home/<USER>/robosats/
StandardOutput=file:/home/<USER>/robosats/follow_invoices.log
StandardError=file:/home/<USER>/robosats/follow_invoices.log
Type=simple
Restart=always
RestartSec=1
User=<USER>
ExecStart=python3 manage.py follow_invoices

[Install]
WantedBy=multi-user.target

Then launch it with

systemctl start follow_invoices
systemctl enable follow_invoices

Clean orders admin command as system service

Create /etc/systemd/system/clean_orders.service and edit with (replace for your username):

[Unit]
Description=RoboSats Clean Orders
After=lnd.service
StartLimitIntervalSec=0

[Service]
WorkingDirectory=/home/<USER>/robosats/
StandardOutput=file:/home/<USER>/robosats/clean_orders.log
StandardError=file:/home/<USER>/robosats/clean_orders.log
Type=simple
Restart=always
RestartSec=1
User=<USER>
ExecStart=python3 manage.py clean_orders

[Install]
WantedBy=multi-user.target

Then launch it with

systemctl start clean_orders
systemctl enable clean_orders