Guide to Staking on Ethereum 2.0 (Ubuntu/Medalla/Nimbus)

Somer Esat

Sep 12, 2020·25 min read

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

This is a step-by-step guide to staking on the Ethereum 2.0 Medalla multi-client testnet via the Status Nimbus client. It is based on the following technologies:

• Ubuntu v20.04 (LTS) x64 server
• Go Ethereum Node (code branch)
• Status’ Ethereum 2.0 client, Nimbus (code branch)
• Official multi-client testnet public network, Medalla
• MetaMask crypto wallet browser extension
• Prometheus metrics
• Grafana dashboard

This guide includes instructions on how to:

• Configure a newly running Ubuntu server instance.
• Configure and run an Ethereum 1.0 node as a service.
• Compile and configure the Nimbus client software for Ethereum 2.0, Phase 0 (Medalla testnet) and run as a service.
• Install and configure Prometheus metrics and set up a Grafana dashboard.

WARNING

This guide is for the Medalla testnet. DO NOT, under any circumstances, send real ETH to this testnet. You will lose it.

Acknowledgements and Disclaimer

This guide is based on information I pulled together from various on-line resources and this guide wouldn’t exist without them. Thank you, all!

And a special Thank You to the client team and the EF researchers. Your tireless efforts over the past few years have brought us to the cusp of an incredible moment in history — the launch of Ethereum 2.0.

This is for educational purposes only. I’m not an expert in any of the technologies listed in this guide. I got it working and it’s a lot of fun, so I wanted to share it with others. Please forgive any errors or ill-informed choices. The accuracy of this guide is not guaranteed. Feedback is always welcome!

Support

This stuff can be tricky. If you need help there are two great resources you can reach out to:

• EthStaker community on Reddit or Discord. Great and friendly bunch of folks that are passionate about staking on Ethereum 2.0.
• Nimbus client team Discord. The client software engineering team. Experts on Nimbus and its usage.

Prerequisites

This guide is not intended for absolute beginners. It assumes some knowledge of Ethereum, ETH, staking, Linux, and MetaMask. Before you get started you will need to have your Ubuntu server instance up and running. It will help to have the MetaMask browser extension installed and configured somewhere. The rest we will do along the way.

Note for Raspberry Pi Users

I haven’t tested this guide on a Rpi. If you want to try, just swap out the software listed below for the ARM version, where available. No guarantee it will work!

Requirements

• Ubuntu server instance. I used v20.04 (LTS) amd64 server VM.
• MetaMask crypto wallet browser extension, installed and configured.
• Hardware requirements are currently undefined. I used a VM with the following specs:
  - Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
  - 4 GB RAM
  - 80GB SSD

Overview

This is a long and detailed guide. Here’s a super simplified diagram to help you conceptualize what we are going to do. The yellow boxes are the parts this guide mostly covers.

The conceptual flow is:

• Set up a Eth1 node and sync it with the Eth1 Goerli testnet
• Generate and activate validator keys
• Configure the Beacon Node (includes the validator)
• The Beacon Node makes the magic happen (blocks, attestations, etc.) with the help of the validator (signing)

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 0 — Connect to the Server

Using a SSH client, connect to your Ubuntu server. The root user account on Ubuntu server is normally disabled by default, however some cloud providers enable it. If you are logged in as root then let’s create a user-level account with admin privileges instead, since using the root user to log in is risky.

NOTE: If you are not logged in as root then skip this section and go to Step 1.

# adduser <yourusername>

You will asked to create a password and some other information.

Grant admin rights to the new user by adding it to the sudo group.

# usermod -aG sudo <yourusername>

When you log in as <yourusername> you can type sudo before commands to perform actions with superuser privileges.

Optional: If you used SSH keys to connect to your Ubuntu instance via the root user you will need to associate the new user with the root user’s SSH key data.

# rsync --archive --chown=<yourusername>:<yourusername> ~/.ssh /home/<yourusername>

Finally, log out of root and log in as <yourusername>.

Step 1 — Update Your System

Make sure your system is up to date with the latest software and security updates. Reboot if necessary.

$ sudo apt update && sudo apt upgrade
$ sudo apt dist-upgrade && sudo apt autoremove

Step 2 — Secure Your System

Security is important. This is not a comprehensive security guide, rather just some basic settings.

Configure the firewall

Ubuntu 20.04 servers can use the default UFW firewall to restrict traffic to the server. We need to allow inbound traffic for SSH, Go Ethereum, Grafana, and Nimbus.

Allow SSH
Allows connection to the server over SSH. For security reasons we are going to modify the default port of 22 because it is a common attack vector.

Note: If you would rather not change the default SSH port (not recommended), then include a rule allowing the default SSH port $ sudo ufw allow 22/tcp and move onto the “Allow Go Ethereum” section.

Choose a port number between 1024–49151 and run the following command to make sure your selection is not already in use on the server. If it is (red text), choose a different port.

$ sudo ss -tulpn | grep ':<yourSSHportnumber>'

Update the firewall to allow inbound traffic on <yourSSHportnumber>. SSH requires TCP.

$ sudo ufw allow <yourSSHportnumber>/tcp

Next change the default SSH port.

$ sudo nano /etc/ssh/sshd_config

Find the line with # Port 22 or Port 22 and change it to Port <yourSSHportnumber>. Remove the # if it was present.

Check the screen shot below for reference. Your file should look similar to that (but with the port number you chose). Exit and save.

Restart the SSH service.

$ sudo systemctl restart ssh

Next time you log in via SSH use <yourSSHportnumber> for the port.

Optional: If you were already using UFW with port 22/TCP allowed then update the firewall to deny inbound traffic on that port.

$ sudo ufw deny 22/tcp

Allow Go Ethereum
Allows incoming requests from Go Ethereum peers (port 30303/TPC and 30303/UDP). If you’d rather use a node hosted by a 3rd party (Infura, etc.) then skip this step.

Note: If you are hosting your Ubuntu instance locally your internet router and/or firewall will need to be configured to allow incoming traffic on these ports as well.

$ sudo ufw allow 30303/tcp
$ sudo ufw allow 30303/udp

Allow Nimbus
Allows P2P connections with peers for actions on the beacon node (ports 19000/TCP and 19000/UDP).

Note: If you are hosting your Ubuntu instance locally your internet router and/or firewall will need to be configured to allow incoming traffic on these ports as well.

$ sudo ufw allow 19000/tcp
$ sudo ufw allow 19000/udp

Allow Grafana
Allows incoming requests to the Grafana web server (port 3000/TCP).

$ sudo ufw allow 3000/tcp

Allow Prometheus (Optional)
If you want direct access to the Prometheus data service you can open up port 9090/TCP as well. This is not necessary if you are solely using Grafana to view the data. I did not open this port.

$ sudo ufw allow 9090/tcp

Now enable the firewall and check to verify the rules have been correctly configured.

$ sudo ufw enable
$ sudo ufw status numbered

Output should look something like this.

Step 3 — Configure ntpd Timekeeping (Optional)

Ubuntu has time synchronization built in and activated by default using systemd’s timesyncd service. However, for applications that require more precise timekeeping (like a blockchain) we will use ntpd.

First, we must disable timesyncd and verify it is off. Output should show NTP service: inactive.

$ sudo timedatectl set-ntp no
$ timedatectl

Next, install the ntp package. It will be started automatically after install. Verify the package.

$ sudo apt install ntp
$ ntpq -p

Output should look like this:

We can further verify the service is running as expected.

$ ntpq -c rv

Results in:

ethstaker@ETH-STAKER-001:~$ ntpq -c rv
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12@1.3728-o (1)", processor="x86_64",
system="Linux/5.4.0-48-generic", leap=00, stratum=2, precision=-24,
rootdelay=1.453, rootdisp=8.653, refid=209.51.161.238,
reftime=e315548b.d020811b  Wed, Sep 23 2020  4:58:51.812,
clock=e31554cf.99ddcf68  Wed, Sep 23 2020  4:59:59.601, peer=5978, tc=6,
mintc=3, offset=3.040122, frequency=6.453, sys_jitter=2.164815,
clk_jitter=2.062, clk_wander=0.539, tai=37, leapsec=201701010000,
expire=202012280000

In the output (2nd line) we have leap_none. This is the Leap Field and the value (leap_none) is a code that indicates we have a normal synchronized state.

In the output (2nd line) we have sync_ntp. This is the Source Field and the value (sync_ntp) is a code that indicates we are using NTP.

In the output (2nd line) we have clock_sync. This is the Event Field and the value (clock_sync) is a code that indicates the clock is synchronized.

If you have different codes to these check here to see a list of possible values. Resolve any issues as necessary.

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 4— Install and Run Go Ethereum Node

Install and configure an Ethereum 1.0 node that the Nimbus beacon node will connect to. If you’d rather use a node hosted by a 3rd party (Infura, etc.) then skip this step.

Install Go Ethereum

Go Ethereum recommends using PPA’s (Personal Package Archives).

$ sudo add-apt-repository -y ppa:ethereum/ethereum

Update the packages and install the latest stable version.

$ sudo apt update
$ sudo apt install geth

Run Go Ethereum as a Background Service

Create an account for the service to run under. This type of account can’t log into the server.

$ sudo useradd --no-create-home --shell /bin/false goeth

Create the data directory for the Eth1 chain. This is required for storing the Eth1 node data. Use the -p option to create the full path.

$ sudo mkdir -p /var/lib/goethereum

Set directory permissions. The goeth account needs permission to modify the data directory.

$ sudo chown -R goeth:goeth /var/lib/goethereum

Create a systemd service file to store the service config. We will use the config file to tell systemd to run the geth process.

$ sudo nano /etc/systemd/system/geth.service

Paste the following service configuration into the file.

[Unit]
Description=Ethereum go client
After=network.target 
Wants=network.target[Service]
User=goeth 
Group=goeth
Type=simple
Restart=always
RestartSec=5
ExecStart=geth --goerli --ws --datadir /var/lib/goethereum[Install]
WantedBy=default.target

The --goerli flag is used to target the Göerli test network and the --ws flag is to expose a web socket endpoint (ws://127.0.0.1:8546) that the beacon node will connect to.

Check the screen shot below for reference. Your file should look like that. Exit and save.

Reload systemd to reflect the changes.

$ sudo systemctl daemon-reload

Start the service and check to make sure it’s running correctly.

$ sudo systemctl start geth
$ sudo systemctl status geth

Output should look like this.

If you did everything right, it should say active (running) in green text. If not then go back and repeat the steps to fix the problem. Press Q to quit.

Enable the geth service to automatically start on reboot.

$ sudo systemctl enable geth

The Go Ethereum node will begin to sync. You can follow the progress by running the journal command. Press Ctrl+C to quit.

$ sudo journalctl -f -u geth.service

Sometimes it can take a while to find peers to sync. If so, you can add some peers to help things along. Go here for the latest list and modify the geth service as follows:

$ sudo systemctl stop geth
$ sudo nano /etc/systemd/system/geth.service

Modify the ExecStart line and add the --bootnodes flag with a few of the latest peers (comma separated).

ExecStart=geth --goerli --http --datadir /var/lib/goethereum --bootnodes "enode://46add44b9f13965f7b9875ac6b85f016f341012d84f975377573800a863526f4da19ae2c620ec73d11591fa9510e992ecc03ad0751f53cc02f7c7ed6d55c7291@94.237.54.114:30313,enode://119f66b04772e8d2e9d352b81a15aa49d565590bfc9a80fe732706919f8ccd00a471cf8433e398c55c4862aadb4aadf3a010201483b87e8358951698aa0b6f07@13.250.50.139:30303"

Save the file and exit. Restart the service and observe.

$ sudo systemctl daemon-reload
$ sudo systemctl start geth
$ sudo journalctl -f -u geth.service

Once it gets started the output should look like this.

Note: You should wait for the node sync to complete before you run the beacon node. You can see the latest block here.

For example, the screen shot above shows the node is processing block number=498880 and looking at the screen shot below (from here), we can see the latest block number is 3391213. So based on that we still have a while to go before completing the sync.

Next we will clone and build the Nimbus software. Consider opening a new terminal window here so you can continue to observe the Eth1 node sync.

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 5 — Install Dependencies

A number of dependencies Developer tools (C compiler, Make, Bash, Git, PCRE) are required to build the Nimbus client.

$ sudo apt install build-essential git libpcre3-dev

Step 6— Install and Build Nimbus

Now we’re ready to build Nimbus. The Nimbus build produces a beacon_node binary. We execute the binary with different subcommands or flags to get the functionality we need. E.g.:

beacon_node deposits import will import validator keys into the client wallet.

beacon_node --network=medalla will run a beacon node instance connected to the Medalla network.

First let’s get the correct version of Nimbus for the Medalla testnet.

$ cd ~
$ git clone https://github.com/status-im/nimbus-eth2
$ cd nimbus-eth2

Use Make to compile the Nimbus binary. We include the -d:insecure flag to compile in the metrics functionality.

$ make NIMFLAGS="-d:insecure" medalla-build

Output looks like this.

Depending on your hardware it can take a while to build. Good time to get a fresh beverage and re-hydrate. Maybe check out a few of my other articles.

If the build succeeds then continue. If not get help on the Nimbus Discord.

Step 7— Complete the Medalla On-boarding Process

In order to run a validator on the Eth2.0 Medalla testnet we will need to sign up for one or more validator accounts.

NOTE: If you have already generated your deposit data and submitted your staking deposits you can skip this step. This guide assumes the validator key files are stored on the Ubuntu server here: $HOME/eth2.0-deposit-cli/validator_keys.

The steps to sign-up are:

• Get Göerli ETH
• Generate the validator keys. Each key is a validator account
• Fund the validator account(s) with 32 Göerli ETH per account
• Wait for your validator account(s) to become active

Let’s get started.

Get Goerli ETH

1. Go to a computer with the MetaMask browser extension installed.
2. Click on MetaMask and log in.
3. Using the drop-down at the top, select the Göerli Test Network.
4. Click on the account name to copy your Göerli Test Network wallet address.
5. Using your address, get Göerli ETH from the authenticated faucet or via the #request-goerli-eth channel on the ethstaker Discord using the bot command: !goerliEth <yourwalletaddress>.

NOTE: Each validator requires a 32 ETH deposit. You should have sufficient Göerli ETH in your MetaMask wallet to fund each validator. For example, if you want 10 validators you need to have 320 Göerli ETH plus some extra (e.g. 1 Göerli ETH) to pay for the gas fees.

Generate Validator Keys

Next we will generate the validator keys. The validator client supports multiple validator keys. Each validator key is basically a “validator account” on the Medalla testnet.

Clone the deposit generator client.

$ cd ~
$ git clone https://github.com/ethereum/eth2.0-deposit-cli.git
$ cd eth2.0-deposit-cli

Check the Python version (Python 3.7 and above is required).

$ python3 -V

If you don’t have Python (or have a version below 3.7), follow these steps.

$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt update
$ sudo apt install python3.7

Install the deposit script prerequisites.

$ sudo apt install python3-pip

Install the application to generate the validator keys.

Change <numberofvalidators> to the number of validator keys you want to create. E.g. --num_validators 5.

$ sudo ./deposit.sh install
$ ./deposit.sh new-mnemonic --num_validators <numberofvalidators> --mnemonic_language=english --chain medalla

After running this step you should have a deposit JSON file, some validator key JSON files, a password, and a mnemonic seed phrase. You should back up all of these somewhere safe.

Success!
Your keys can be found at: /home/<yourusername>/eth2.0-deposit-cli/validator_keys

Copy Deposit Data File

In the validator_keys directory there will be a deposit_data-[timestamp].json file. You will need to upload this via a website in the next step. Since we are on a server we don’t have a web browser so secure FTP (SFTP) the file to a desktop computer that does.

Fund the Validator Keys

This step involves depositing the required amount of Göerli ETH to the Medalla testnet staking contract. This is done on the Eth2.0 Lauchpad website.

WARNING: DO NOT send real ETH to the Medalla testnet. You will lose your ETH.

Go here: https://medalla.launchpad.ethereum.org/

Click through the warning steps then select the number of validators you are going to run. Scroll down and click continue.

You will be asked to upload the deposit_data-[timestamp].json file. Browse or drag the file and click continue.

Connect your wallet. Choose MetaMask, log in, select the Göerli Test Network and click Continue.

WARNING: Be absolutely 100% sure you have selected the Göerli Test Network in MetaMask. DO NOT sent real ETH to the Medalla testnet.

Your MetaMask balance will be displayed. The site will allow you to continue if you have sufficient Göerli ETH balance.

A summary shows the number of validators and total amount of Göerli ETH required. Tick the boxes if you agree and click continue.

Click on initiate all transactions. This will pop open multiple instances of MetaMask, each with a 32 Göerli ETH transaction request to the Medalla testnet deposit contract. Confirm each transaction.

Once all the transactions have successfully completed you are done!

Check the Status of Your Validators

Newly added validators can take a while (hours/days/weeks) to activate. You can check the status of your keys with these steps:

1. Copy your Göerli Test Network wallet address
2. Go here: https://medalla.beaconcha.in/
3. Search your wallet address. Your keys will be shown.

Click on a key to see the Estimated Activation information.

That’s it! Now let’s create the validator wallet.

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 8— Create the Validator Wallet

The validator wallet is created by importing the keystore-m JSON files from the previous step.

First, create a directory to store the validator wallet and give the current user permissions to access it. Change <yourusername> to your logged in username.

$ sudo mkdir -p /var/lib/nimbus
$ sudo chown -R <yourusername>:<yourusername> /var/lib/nimbus

Set up the correct permissions on the folder. Enforced by the wallet creation process. Applies permissions 0750 (rwxr-x — -) to the data folder.

$ sudo chmod 0750 /var/lib/nimbus

Next, run the wallet creation process. The nimbus_beacon_node_spec_0_12_3 deposits import function expects the keystore-m JSON files to be in the directory $HOME/eth2.0-deposit-cli/validator_keys.

$ cd ~
$ cd nimbus-eth2
$ build/nimbus_beacon_node_spec_0_12_3 deposits import --data-dir=/var/lib/nimbus $HOME/eth2.0-deposit-cli/validator_keys

The --data-dir flag specifies the location to output the wallet data.

You will be asked to provide the password for the validator keys. This is the password you set when you created the keys during the previous step.

Output should look like:

That’s it! Now that the validator wallet is configured we will set up the beacon node (and validator) to run as a service.

Step 9— Configure the Beacon Node

We will run the beacon node as a service so if the system restarts the process will automatically start back up again.

Setup Accounts and Directories

Create an account for the beacon node to run under. This type of account can’t log into the server.

$ sudo useradd --no-create-home --shell /bin/false nimbus

Set directory permissions. We created a data directory (/var/lib/nimbus) in the previous step. The nimbus account needs permission to modify that data directory.

$ sudo chown -R nimbus:nimbus /var/lib/nimbus

Next, copy the nimbus_beacon_node_spec_0_12_3 binary we built in Step 6 to the /usr/local/bin directory. The nimbus.service will run it from there.

Note: You will need to do this step each time you pull/build a new version of the binary. See Appendix — Updating Nimbus at the end of this guide

$ sudo cp /$HOME/nimbus-eth2/build/nimbus_beacon_node_spec_0_12_3 /usr/local/bin

Create and Configure the Service

Create a systemd service file to store the service config.

$ sudo nano /etc/systemd/system/nimbus.service

Paste the following into the file.

[Unit]
Description=Nimbus Beacon Node
Wants=network-online.target
After=network-online.target[Service]
Type=simple
User=nimbus
Group=nimbus
Restart=always
WorkingDirectory=/var/lib/nimbus
Environment="ClientIP=$(curl -s v4.ident.me)"
ExecStart=/bin/bash -c '/usr/local/bin/nimbus_beacon_node_spec_0_12_3 --network=medalla --data-dir=/var/lib/nimbus --web3-url=ws://127.0.0.1:8546 --metrics --metrics-port=8008 --rpc --rpc-port=9091 --max-peers=250 --nat=extip:${ClientIP} --validators-dir=/var/lib/nimbus/validators --secrets-dir=/var/lib/nimbus/secrets --log-level=INFO --log-file=/var/lib/nimbus/beacon_node.log'[Install]
WantedBy=multi-user.target

Replace the --web3-url if you are using a remote or third party Eth1 node. If you created the Eth1 node locally (via Step 4), you don’t need to change this value.

The flags supplied are:

--network — the network the client is connecting to. In this case, Medalla.

--data-dir — the directory to store the beacon node and validator data.

--web3-url — the URL to the Eth1 node (local or remote). If you are using a remote Eth1 node it is preferred to use the web socket (ws) URL.

--metrics — toggles the metrics server on.

--metrics-port — redundant since it’s set to the default, but included for convenience. Must match the Nimbus metrics port in the Prometheus configuration file.

--rpc — toggles the RPC API on.

--rpc-port — required to avoid conflicting with Prometheus.

--max-peers — the maximum number of peers for the beacon node.

--nat — the external IP address of your server. We use an Environment variable Environment="ClientIP=$(curl -s v4.ident.me)" to get the client IP address because ExecStart doesn’t allow the call in-line. Using --p2p-host-ip=${ClientIP} is the work-around.

--validators-dir — the directory where the validator key data is located.

--secrets-dir — the directory where the validator key secrets are located.

--log-level — the logging verbosity level. INFO is used for performance reasons.

--log-file — required so the service can create the log-file. Path must match the service WorkingDirectory.

Check the screen shot below for reference. Your file should look like that. Exit and save.

Reload systemd to reflect the changes.

$ sudo systemctl daemon-reload

Note: If you are running a local Eth1 node (see Step 4) you should wait until it fully syncs before starting the Nimbus service. Check progress here: sudo journalctl -fu geth.service

Start the service and check to make sure it’s running correctly.

$ sudo systemctl start nimbus
$ sudo systemctl status nimbus

Output should look like this.

If you did everything right, it should say active (running) in green text. If not then go back and repeat the steps to fix the problem. Press Q to quit.

Enable the service to automatically start on reboot.

$ sudo systemctl enable nimbus

The beacon-chain will begin to sync. It may take several hours for the node to fully sync. You can check the progress by running the journal command. Press Ctrl+C to quit.

$ sudo journalctl -fu nimbus.service

The journal output is similar to this:

Nov 17 08:28:08 ETH-STAKER-001 bash[38646]: NOT 2020-11-17 08:27:56.010+00:00 Syncing in progress; skipping validator duties for now topics="beacval" tid=38646 file=validator_duties.nim:500 slot=754639 headSlot=4148
Nov 17 08:28:08 ETH-STAKER-001 bash[38646]: INF 2020-11-17 08:27:56.010+00:00 Slot end                                   topics="beacnde" tid=38646 file=nimbus_beacon_node.nim:551 slot=754639 nextSlot=754640 head=e26f2de5:4148 headEpoch=129 finalizedHead=9f8fdba9:4064 finalizedEpoch=127

To determine the distance between current and head, you can compare the headEpoch to the current epoch on the medalla.beaconcha.in site.

For example, the journal output above shows the node is processing headEpoch: 129 and looking at the screen shot below, we can see the latest epoch is 23582. So based on that we still have a while to go before completing the sync.

Now your beacon chain is running as a service. Congratulations!

NOTE: The validator(s) will automatically start attesting/proposing once the beacon node has fully synced.

You can check the status of your validator(s) via medalla.beaconcha.in. Simply do a search for your validator public key(s) or search using your MetaMask wallet address. It may be a while before they appear on the site.

That’s it. We have a functioning beacon chain and (eventually) validator(s). Congratulations: you are awesome!

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 10— Install Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. It runs as a service on your Ubuntu server and its job is to capture metrics. More information here.

We are going to use Prometheus to expose runtime data from the beacon-chain and validator as well as instance specific metrics.

Create User Accounts

Accounts for the services to run under. These accounts can’t log into the server.

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter

Create Directories

Program and data directories.

$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus

Set directory ownership. The prometheus account will manage these.

$ sudo chown -R prometheus:prometheus /etc/prometheus
$ sudo chown -R prometheus:prometheus /var/lib/prometheus

Download Prometheus software

Adjust the version number to the latest version from the Prometheus download page. Rpi users be sure to get the ARM binary.

$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.20.1/prometheus-2.20.1.linux-amd64.tar.gz

Unpack the archive. It contains two binaries and some content files.

$ tar xvf prometheus-2.20.1.linux-amd64.tar.gz

Copy the binaries to the following locations.

$ sudo cp prometheus-2.20.1.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.20.1.linux-amd64/promtool /usr/local/bin/

Set directory ownership. The prometheus account will manage these.

$ sudo chown -R prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown -R prometheus:prometheus /usr/local/bin/promtool

Copy the content files to the following locations.

$ sudo cp -r prometheus-2.20.1.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.20.1.linux-amd64/console_libraries /etc/prometheus

Set directory and file (-R) ownership. The prometheus account will manage these.

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Remove the downloaded archive.

$ rm -rf prometheus-2.20.1.linux-amd64.tar.gz prometheus-2.20.1.linux-amd64

Edit the Configuration File

Prometheus uses a configuration file so it knows where to scrape the data from. We will set this up here.

Open the YAML config file for editing.

$ sudo nano /etc/prometheus/prometheus.yml

Paste the following into the file taking care not to make any additional edits and exit and save the file.

global:
  scrape_interval:     15s 
  evaluation_interval: 15s scrape_configs:
  - job_name: 'nimbus'
    static_configs:      
      - targets: ['localhost:8008']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

The scrape_configs define the output target for the different job names. The node_exporter is for metrics related to the server instance itself (memory, CPU, disk, network etc.). We will install and configure node_exporter below.

Set ownership for the config file. The prometheus account will own this.

$ sudo chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

Finally, let’s test the service is running correctly.

$ sudo -u prometheus /usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

Output should look something like this. Press Ctrl + C to exit.

level=info ts=2020-09-12T22:16:21.179Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-09-12T22:16:21.181Z caller=main.go:700 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-09-12T22:16:21.181Z caller=main.go:701 msg="TSDB started"
level=info ts=2020-09-12T22:16:21.182Z caller=main.go:805 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-12T22:16:21.182Z caller=main.go:833 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-12T22:16:21.183Z caller=main.go:652 msg="Server is ready to receive web requests."

Set Prometheus to Auto-Start as a Service

Create a systemd service file to store the service config which tells systemd to run Prometheus as the prometheus user, with the configuration file located in the /etc/prometheus/prometheus.yml directory, and to store its data in the /var/lib/prometheus directory.

$ sudo nano /etc/systemd/system/prometheus.service

Paste the following into the file. Exit and save.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries[Install]
WantedBy=multi-user.target

Reload systemd to reflect the changes.

$ sudo systemctl daemon-reload

And then start the service with the following command and check the status to make sure it’s running correctly.

$ sudo systemctl start prometheus
$ sudo systemctl status prometheus

Output should look something like this.

If you did everything right, it should say active (running) in green. If not then go back and repeat the steps to fix the problem. Press Q to quit.

Lastly, enable Prometheus to start on boot.

$ sudo systemctl enable prometheus

Step 11 — Install Node Exporter

Prometheus will provide metrics about the beacon chain and validators. If we want metrics about our Ubuntu instance, we’ll need an extension called Node_Exporter. You can find the latest stable version here if you want to specify a different version below. Rpi users remember to get the ARM binary.

$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz

Unpack the downloaded software.

$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz

Copy the binary to the /usr/local/bin directory and set the user and group ownership to the node_exporter user we created above.

$ sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown -R node_exporter:node_exporter /usr/local/bin/node_exporter

Remove the downloaded archive.

$ rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64

Set Node Exporter to Auto-Start as a Service

Create a systemd service file to store the service config which tells systemd to run Node_Exporter as the node_exporter user.

$ sudo nano /etc/systemd/system/node_exporter.service

Paste the following into the file. Exit and save.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter[Install]
WantedBy=multi-user.target

Reload systemd to reflect the changes.

$ sudo systemctl daemon-reload

And then start the service with the following command and check the status to make sure it’s running correctly.

$ sudo systemctl start node_exporter
$ sudo systemctl status node_exporter

Output should look something like this.

If you did everything right, it should say active (running) in green. If not then go back and repeat the steps to fix the problem. Press Q to quit.

Finally, enable Node Exporter to start on boot.

$ sudo systemctl enable node_exporter

Test Prometheus and Node Exporter (Optional)

Everything should be ready to go. You may optionally test the functionality by opening a port in the firewall (see Step 1) and browsing to http://<yourserverip>:9090. From there you can run queries to view different metrics. For example try this query to see how much memory is free in bytes:

http://<yourserverip>:9090/new/graph?g0.expr=node_memory_MemFree_bytes&g0.tab=1&g0.stacked=0&g0.range_input=1h

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Step 12— Install Grafana

While Prometheus is our data source, Grafana is going provide our reporting dashboard capability. Let’s install it and configure a dashboard.

We will install using an APT repository because it is easier to install and update. Grafana is available in the official Ubuntu packages repository, however the version of Grafana there may not be the latest, so we will use Grafana’s official repository.

Download the Grafana GPG key with wget, then pipe the output to apt-key. This will add the key to your APT installation’s list of trusted keys.

$ wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add the Grafana repository to the APT sources.

$ sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Refresh the apt cache.

$ sudo apt update

Make sure Grafana is installed from the repository.

$ apt-cache policy grafana

Output should look like this.

grafana:
  Installed: (none)
  Candidate: 7.1.5
  Version table:
     7.1.5 500
        500 https://packages.grafana.com/oss/deb stable/main amd64 Packages
     7.1.4 500
        500 https://packages.grafana.com/oss/deb stable/main amd64 Packages
     7.1.3 500
        500 https://packages.grafana.com/oss/deb stable/main amd64
     ...

Verify the version at the top matches the latest version shown here. Then proceed with the installation.

$ sudo apt install grafana

Start the Grafana server and check the status to make sure it’s running correctly.

$ sudo systemctl start grafana-server
$ sudo systemctl status grafana-server

Output should look something like this.

If you did everything right, it should say active (running) in green. If not then go back and repeat the steps to fix the problem. Press Q to quit.

Enable Grafana to start on boot.

$ sudo systemctl enable grafana-server

Configure Grafana Login

Great job on getting this far! Now that you have everything up and running you can go to http://<yourserverip>:3000/ in a browser and the Grafana login screen should come up.

Enter admin for the username and password. It will prompt you to change your password and you should definitely do that.

Configure the Grafana Data Source

Let’s configure a datasource. Move your mouse over the gear icon on the left menu bar. A menu will pop-up — choose Data Sources.

Click on Add Data Source and then choose Prometheus. Enter http://localhost:9090 for the URL then click on Save and Test.

Import a Grafana Dashboard

Now let’s import a dashboard. Move your mouse over the + icon on the left menu bar. A menu will pop-up - choose Import.

Paste the JSON from here (copy from raw) and click Load then Import. You should be able to view the dashboard.

NOTE: This is obviously a very basic dashboard. The team is working on exposing more metrics (e.g. number of validators, attestations per validator, proposals per validator, balance per validator, etc.). The current official dashboard is here, and live here.

Alerts are also available through Telegram and Discord. See here for instructions.

Final Remarks

Okay… That’s it! We are done! I hope you enjoyed this guide.

• If you have feedback you can reach me on Twitter or Reddit.
• If you liked this guide and think others would benefit from it then please share it using the friends link!
• Tips appreciated: somer.eth

***
NOTE: This guide is now deprecated. Please use the newer version targeting the Pyrmont testnet.

*** located here.

Appendix — Updating Nimbus

If you need to update the code due to changes in the Git repository follow these steps to get the latest files and build your binaries:

$ cd ~
$ cd nimbus-eth2
$ git pull
$ make update
$ make NIMFLAGS="-d:insecure" medalla-build

Next we stop the beacon chain service and copy the binaries over to the /usr/local/bin directory and then start the services again.

$ sudo systemctl stop nimbus
$ sudo cp /$HOME/nimbus-eth2/build/nimbus_beacon_node_spec_0_12_3 /usr/local/bin
$ sudo systemctl start nimbus
$ sudo systemctl status nimbus # <-- Check for errors
$ sudo journalctl -fu nimbus.service # <-- Monitor

That’s it, the services have been updated.

Appendix — Updating Geth

If you need to update to the latest version of Geth follow these steps:

$ sudo systemctl stop geth
$ sudo systemctl stop nimbus
$ sudo apt update && upgrade
$ sudo systemctl start geth
$ sudo systemctl status geth # <-- Check for errors
$ sudo journalctl -f -u geth # <-- Monitor
$ sudo systemctl start nimbus
$ sudo systemctl status nimbus # <-- Check for errors
$ sudo journalctl -f -u nimbus.service # <-- Monitor