The following commands can be run on a node to aid in daily operation.

Check if the stack is running

Run this command to see the status of all containers:

Restart the whole stack

Restarting after encountering problems

After encountering problems e.g. with the parity client, it is better to run

If after restarting the client you are still encountering problems, you can try to delete the database and re-sync the node (please only do this after trying to restart without deleting the database first; if you have any questions or need assistance please post in the Validator Knowledge Base):

Update a component

Set the desired version in

save it and run

Update configurations

Make the configuration change in either .env or docker-compose.yml

and run

Show logfiles

In docker-stack run

to e.g. show the last 100 lines of the parity client logs run

The System architecture of a validator node on the Energy Web Chain is made up of two main components. (Note: the original node architecture featured a node control component, but it was never used and formally deprecated in 2020).

Client: The OpenEthereum or Nethermind client that connects the validator to the Energy Web Chain, collects transactions and proposes blocks according to the AuRa consensus algorithm. Read the documentation of the OpenEthereum client here, and the Nethermind client here.

Telemetry: There is a monitoring system on the validator node that uses Telegraf to securely send telemetry to an InfluxDB that is connected to Grafana and the EWC Validator Data Warehouse & Dashboard. The use of telemetry is opt-in. Validators can disable it if they have their own monitoring system in place that allows for real time tracking of all relevant metrics.

Telemetry data includes:

• CPU usage of the host machine

• Memory usage of the host machine

• Disk usage of the host machine

• Number of connected peers

• List of visible P2P peers

• Current block (latest block synced by the node)

• Network latency to 3 different and major locations (e.g. cloudflare, google, amazon)

• Network throughput

• Network error rate

• Number of SSH keys for the host machine

• Service status for SSH, docker and the parity container

• SHA256 hashes of key system components/binaries

• Current chain specification file (or hash)

As this is sensitive data, we rely on well established industry solutions to transfer these metrics off the validator node. Telegraf is a server agent that plugs into many different services to collect data. Telegraf collects relevant metrics from the host machine and the custom-built parity metrics collector which allows Telegraf to receive the metrics from the parity client. For more information on Telegraf visit: https://docs.influxdata.com/telegraf/v1.10/

The collected metrics are then stored in an InfluxDB database and can be visualized using the monitoring tool Grafana. For more information on influxDB visit: https://docs.influxdata.com/influxdb/v1.7/

For more information on Grafana visit: https://grafana.com/docs/

The use of telemetry is opt-in. Validators can disable it if they have their own monitoring system in place that allows for real time tracking of all relevant metrics.

All components are run in separate docker containers managed by docker compose. For additional information on docker visit: https://docs.docker.com/ and https://docs.docker.com/compose/

Wait a little bit longer

If you are not connected to any peers right away, often it can help to just wait a little longer and give the node a little time to connect to other peers. If the node is still not connected after a couple of minutes, there is probably a problem.

Sync your local time

If you have trouble connecting to peers you should first check if the local time on your device is exact. Go to and ensure it says: “Your time is exact”. If your time is not exact, go to System Preferences and make sure it is synchronized.

To make sure that your clock is always synchronized you can also use the ntp-pool-project, a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients:

Basic instructions on how to set up ntp-pool can be found here:

More detailed instructions for different operating systems can be found here:

Mac:

Linux:

Windows:

Delete nodes.json

If your node cannot connect to other peers, it might help to remove your nodes.json file to reestablish a connection. You can find the nodes.js file at the same directory where you store your keys under chain/Volta/network. If you don't know where that is, you can look at the path specified at "Keys Path" when you start the client in the terminal.

Usually it sits at:

$HOME/Library/Application\ Support/io.parity.ethereum/chains/ewc/network/nodes.json

or

$HOME/.local/share/io.parity.ethereum/chains/ewc/network/nodes.json

or

$HOME\AppData\Local\Parity\Ethereum\chains\ewc\network\nodes.json

Make sure the client is not running. Delete the file and than restart the client. A new nodes.js file will be created and you will hopefully connect to peers again.

Manually add peers when starting the client

Or follow the below steps (recommended)

2. Edit the parity-signing.toml & parity-non-signing.toml file

   1. Add the following line in [network] section

   2. reserved_peers = "/parity/config/bootnodes"

   3. Recreate the containers. docker-compose up -d --force-recreate

Manually add peers

If you are not able to find any peers or you want to connect to a specific peer you can add peers manually.

Make sure that the parity API's are enabled by starting the client with the flag

The Peer ID that you need for this function are called enodes and have this format (this is one of our bootnodes, it should already be on your nodes list by default): enode://59c9250cb805409e84c9cd0038e97d8e5e4605b928663675869ebdfd4c251d80ccad76267a5eb2f4362ddceb5ec671f7595463adfc0a12e9f68dbf233072db41@54.70.158.106:30303

At the very top it will give you the number oft active, connected and max peers. Active means, they are actively synchronizing the chain, a 0 basically means you are already fully synchronized or have no peers at all. Connected are all peers, which you are listening to for new blocks and transactions, ideally this should not be 0. Max is just the maximum number of peers you can connect to.

Below that you will see a list of peers your node is trying to connect to. Peers with non-empty protocols have completed handshake and are currently connected to you. You can find them quickly by searching for "difficulty" or "head".

Allow UDP traffic

If all this still does not resolve the problem, you should check if there is some firewall in place that causes the problem and make sure your network does not block UDP traffic.

Report Problem

Check the block explorers!

The block explorers allow you to search the specific addresses of your Volta and EWC nodes to see how recently they validated a block.

Once you’ve searched for your address in the block explorers (make sure to search for your EWC address in the EWC explorer and vice versa), navigate to the “Blocks Validated” page to see how recently it validated a block (ex: ). With ~30 validators and a 5 second block time, a functioning node validates a block once every ~2:30 minutes. Bookmark the “Block Validated” pages for both your Volta and EWC nodes to easily check if they’re validating.

Using the steps listed below, you can also view your validator node’s “Company Name” in each block it has sealed. The “Company Name” shows up in the extraData field of a block, but it's not shown directly in the explorer.

1. Determine the block number or block hash you want to check from the explorer

2. You can use the following commands to fetch block data:

curl --data '{"method":"eth_getBlockByNumber","params":["0x_blocknumber_in_hex_here",true],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://volta-rpc.energyweb.org

curl --data '{"method":"eth_getBlockByHash","params":["0x_blockhash_here",true],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://volta-rpc.energyweb.org

1. Find the extraData field and copy the value,

   • e.g: "extraData":"0x76616c696461746f722d35342e3135342e3130312e353454454f". [This is the company name (or whatever info) you set during the installation encoded as hex]

2. Paste the value here to convert to text:

SSH into your node and check the containers

Once you SSH into your node, you can use the following commands to check the status of your server and docker containers running validator services.

Use the following command to make sure you have admin rights before running other commands

Run this command to list all running docker containers:

You should see these three containers with STATUS “Up X hours”

• docker-stack_parity_1

• docker-stack_parity-telemetry_1

Check your node’s resources

If your node is hosted by a cloud provider, you will likely have access to a dashboard from that provider with information on the amount of disk space your node has available and other relevant info.

If you are SSH’d into your server, you can also run this command to check if your disk is nearing capacity:

And to see how much disk space your docker containers are using, run:

Check your node’s logs

Logs can give you a view into

To save your logs and transfer them to your local machine use the following commands

docker-compose logs > ~/YOURCOMPANY_logs.txt

You will likely need to stop (ctrl + c) this after a little while as it doesn’t seem to auto-complete.

Make sure the logs contain whatever timeframe you’re interested in (e.g., when your node stopped syncing).

Now transfer them to the Downloads folder on your personal computer using this command in a fresh terminal:

scp -P 2222 -i YOURKEYS.pem ubuntu@YOUR_INSTANCE:/home/ubuntu/YOURCOMPANY_logs.txt ~/Downloads/

If you were successful, you should see a download bar complete and have the file in your Downloads folder.

Validator Dashboard

[Coming Soon]

• Public-facing dash that only links node attributes to a public address

• Address, # of peers, latest block, list of unsynced nodes

Note: As of October 27, 2021, all EWC nodes should run a London-compatible client: OpenEthereum ( or later) or Nethermind ( or later)

The update schedule

OpenEthereum and Nethermind clients compatible with AuRa are released in a regular rhythm. The process for implementing regular updates is described below. Emergency updates (i.e. in the event of a known security vulnerability with a specific client version) will be accelerated.

All EWC client updates are tested by the Energy Web Chain Technical Committee to ensure compatibility with the EW Chain AuRa consensus mechanism. The Technical Committee strongly recommends validators and other node (e.g. RPC) operators refrain from updating their clients until compatibility with the current version is confirmed on this page. Once an update is tested, the Technical Committee will communicate to validators and the broader community via Slack and Telegram that it is safe to install the new version; this page will also be updated regularly. For security/stability reasons, we recommend rolling out the update in waves, first on Volta and then the production EW Chain.

Updating the client

Manual Upgrade

As the upgrade to OpenEthereum v3.3.0-rc.11 does not involve any config changes or require db resync, it should be relatively simple.

1. Download and verify the new image:

Download the node software (Openethereum v3.3.0-rc.11 or later)

docker pull openethereum/openethereum:v3.3.0-rc.11

Verify the downloaded software to make sure no one has changed something during data transfer

docker image inspect openethereum/openethereum:v3.3.0-rc.11 | jq -r '.[0].Id'

The result should be:

sha256:58ef9c2b1c475fe875fed8d291978bbaac6b19951aa9e8a4686342bbed086fab

If that's the case, proceed to the next step. If not, repeat the process above to download and verify the new image.

2. Make a backups & update image version

• Stop the containers:

• Make a backup of docker-compose.yml:

cp $HOME/docker-stack/docker-compose.yml $HOME/docker-stack/docker-compose.yml_backup

• Make a backup of .env file:

cp $HOME/docker-stack/.env $HOME/docker-stack/.env_backup

• Make a backup of DB [Optional]:

cp -r chain-data/chains/$CHAIN_NAME/ ./$CHAIN_NAME_backup

• Modify and save .env file (lines 3 & 9):

vim $HOME/docker-stack/.env

Modify and save docker-compose.yml file (line no.4):

vi $HOME/docker-stack/docker-compose.yml

New docker-compose.yml

Restart the containers

Check whether the node is running & syncing

Validator maintenance documentation is organized as follows:

Prerequisites

1. Check Docker Image, NPM Package

   1. Docker Image:

   2. Github Repo of

   3. NPM Package

2. In the Validator Node - get the path of the following two files -

   1. keyfile - UTC--*

      1. In Parity/OpenEthereum Validator Node, the path of keyfile is -

         • $HOME/docker-stack/chain-data/keys/$CHAINNAME/UTC--* [CHAINNAME can be Volta or EnergyWeb]

   2. secret file - .secret

      1. In Parity/OpenEthereum Validator Node, the path of secret file is -

         • $HOME/docker-stack/.secret

Execution of ewf-validator-tool Using Docker

1. Put the the value before execution:

   1. KEYFILE_PATH : path of keyfile in old validator node

   2. SECRETFILE_PATH : path of secret file in old validator node

   3. NEW_ADDRESS : Address of wallet or new node address

   4. TOKEN_AMOUNT: Amount of token to be taken out

   5. HTTPS_RPC_URL :

      1. Volta HTTPS_RPC_URL: https://volta-rpc.energyweb.org

      2. EnergyWeb HTTPS_RPC_URL: https://rpc.energyweb.org

2. Command to transfer token

   1. Before execute take superuser privilege - sudo -s or run docker with sudo

docker run -it -v "$KEYFILE_PATH":/keyfile -v $SECRETFILE_PATH:/keypass aznagy/ewf-validator-tool:latest transferto $NEW_ADDRESS $TOKEN_AMOUNT -r $HTTPS_RPC_URL

Example:

Here is an example

(It is Nethermind validator node, that’s why the path of keyfile and secret file are different than Parity validator node)

1. Transfer token from Volta Validator Node to Hardware Wallet -

   1. KEYFILE_PATH : keyfile path of my Validator Node

      • /home/ubuntu/docker-stack/keystore/UTC--2020-12-21T16-47-06.412446000Z--5125254ec2e024d8226cf4c389512c43802a76a1

   2. SECRETFILE_PATH : secret file path of my Validator Node

      • /home/ubuntu/docker-stack/keystore/.secret

2. NEW_ADDRESS : Address of my wallet 0x4933915c40477Db3a9AccA3Fa12DaA8ba5D4fD35

3. TOKEN_AMOUNT: 0.01

4. HTTPS_RPC_URL : Volta HTTPS_RPC_URL: https://volta-rpc.energyweb.org

Command

Transfer Token

Output

Check in the Explorer

Check with validator-tool command: payout check and payout changeto

1. payout check command

Example:

Output:

2. payout changeto command

Example:

Output:

Run Validator Tool using npm package

If you want to execute validator tool on host machine or local machine without using docker, follow the following process.

Prerequisite:

1. Install node and npm

   • npm comes with nodejs installation.

   • check nodejs and npm version

     • node -v

     • npm -v

2. • npm install -g ewf-validator-tool

   • Check validator-tool is installed

validatortool --version 21.0.3

Execution of ewf-validator-tool

1. From the old (original) Validator Node - get the path or grab the value of the following two files - [If you want to run it on your local machine, just to collect the value of keyfile and .secret file.]

   1. keyfile - UTC--*

   2. In Parity/OpenEthereum Validator Node, the path of keyfile is -

      • $HOME/docker-stack/chain-data/keys/$CHAINNAME/UTC--* [CHAINNAME can be Volta or EnergyWeb]

   3. secret file - .secret

      1. In Parity/OpenEthereum Validator Node, the path of secret file is -

         • $HOME/docker-stack/.secret

   4. HTTPS_RPC_URL : Volta HTTPS_RPC_URL: https://volta-rpc.energyweb.org

2. Command

   1. Transfer Token

      1. You can directly give the path of the keyfile and .secret file Or

      2. the keyfile value is stored in UTC-keyfile and value of .secret is stored in Secretfile

validatortool transferto $NEW_ADDRESS -k "$PATH_OF_KEY_FILE" 0.05 -s $PATH_OF_SECRETFILE -r $HTTPS_RPC_URL

Example

validatortool transferto 0x4933915c40477Db3a9AccA3Fa12DaA8ba5D4fD35 -k "$HOME/docker-stack/chain-data/keys/$CHAINNAME/UTC--*" 0.05 -s $HOME/docker-stack/.secret -r https://volta-rpc.energyweb.org

or

validatortool transferto 0x4933915c40477Db3a9AccA3Fa12DaA8ba5D4fD35 -k "UTC-keyfile" 0.05 -s Secretfile -r https://volta-rpc.energyweb.org

b. payout check command

validatortool payout check $OLD_VALIDATOR_NODE -r $HTTPS_RPC_URL

Example

validatortool payout check 0x5125254Ec2E024D8226cf4c389512c43802a76A1 -r https://volta-rpc.energyweb.org

c. payout changeto command

validatortool payout changeto $NEW_ADDRESS -r $HTTPS_RPC_URL

Example

validatortool payout changeto $0x4933915c40477Db3a9AccA3Fa12DaA8ba5D4fD35 -r https://volta-rpc.energyweb.org

You should get similar output if you run the validator-tool using docker. To get a glimpse of output, please check the above section.

Sometimes an organization needs to migrate their validator node into a new environment. Common reasons for this include:

• Changing infrastructure providers (e.g. switching cloud or on-premise vendors)

• Wanting to upgrade host environment configuration

• Rotating access due to security procedures or policies.

If you are currently operating a validator node on Volta of the EW Chain and need to migrate it from its existing host environment to a new one, there are two options:

Install a new node and retire the existing one

• This option requires provisioning a new host environment and following the installation procedures to establish an entirely new node (note: this will result in a new address, private key, EWT balance, etc.). Once your new node is installed, the current one will be removed from the network and you will be able to retire / deprecate the legacy node and host environment.

• The benefits of this approach are:

  • It is easy to replicate the node installation process, so the overall migration is simpler;

  • You don't have to worry about independently modifying the installation script;

  • You can maintain continuity by simultaneously retiring the original node and adding the new one to the network;

• The drawbacks of this approach are:

  • You must determine a secure way to maintain access to the legacy block rewards held on the original node address;

  • Depending on your organization's IT policies and procedures, it may be more complex and costly to procure multiple host environments during the migration.

  • You may need to replicate or modify access management policies and procedures for the new environment.

Migrate your existing node to a new environment

• This option involves migrating the existing node (i.e. maintaining the same address and private key) to a new host environment. To do this, you must extract the private key from the existing environment and then re-run a modified installation script in the new environment, using the private key of the validator node.

• The benefits of this approach are:

  • You maintain continuity by keeping the same address and private key;

  • You don't need to operate two validator nodes (and thus host environments) simultaneously.

• The drawbacks of this approach are:

  • Depending on the parameters of the existing vs. new environment, you may need to modify configurations and access management.

  • You must securely extract the private key from the existing node, temporarily store it, and import it into the new environment.

  • You will experience a period of "downtime" throughout the migration, during which your organization's validator node will not be operational.

Follow the steps to migrate existing nodes to new node:

1. 1. Migrate existing nodes: Follow the steps to migrate existing nodes to new node:

      1. Collect and store the following necessary info from Old Instance -

         1. You can stop containers in old instance. docker-compose down

         2. Value of Validator Address, InfluxDB Username, InfluxDB Password . You’ll find this info in install-summary.txt file.

         3. Value of docker-stack/keystore/.secret To copy the .secret key to your local instance:

            scp -P 2222 -i YOURKEYS.pem user@YOUR_INSTANCE:/home/user/docker-stack/keystore/.secret ~/Downloads/

         5. [optional] keep a backup of docker-stack/ dir.

      2. Commented out the following lines from script

         1. Volta validator installation script

         2. EWC validator installation script

   [line from 180-213] -

   from echo "Creating Account..." [line 180] till docker rm -f nethermind [line 213] .

   1. Then run the script in the new instance.

      1. The End Result would be the the following . Validator Address, Enode, InfluxDB Username and InfluxDB Password field would be empty

         ==== EWF Affiliate Node Install Summary ==== Company: Company_Name Validator Address: Enode: null IP Address: New_IP_ADDESSS InfluxDB Username: InfluxDB Password:

      2. In the new Instance, There is the docker-stack/ dir. Enter to the directory and run docker-compose ps and run docker-compose down to kill node and telemetry containers.

      3. The .secret file in docker-stack/keystoredir needs to be updated with the older one. [ .secret file is in only readonly mode. Change mode before updating with old value. chmod 644 .secret ]

      4. The .env file in docker-stack/ dir needs to be updated.

         1. Update the VALIDATOR_ADDRESS= field with old value.

         2. Update the NETHERMIND_KEY_FILE=./keystore/UTC--* with correct name.

      5. In the docker-stack/configs/volta.cfg or docker-stack/configs/energyweb.cfg, the following 2 value needs to be updated. In the new instance, this value would be empty. Update the value with old Validator Address

         "KeyStore": { "PasswordFiles": ["keystore/.secret"], "UnlockAccounts": ["Old Validator Address goes here..."], "BlockAuthorAccount": "Old Validator Address goes here..." },

      6. Copy the Validator’s mining private key to your new instance

         • Copy the UTC--* file from old instance to docker-stack/keystore dir or create that file in docker-stack/keystore/ dir with exact same name and same value. To copy the file to your local instance:

           scp -P 2222 -i YOURKEYS.pem user@YOUR_INSTANCE:/home/user/docker-stack/keystore/UTC_PART ~/Downloads/

         • [Optional] If you want a exact sync of database, you can copy db files from the old instance to the new instance -

           • DB Path - docker-stack/database/volta or docker-stack/database/energyweb

         • Run docker-compose up -d

         • Check logs of parity container -

           • docker-compose logs -f nethermind

           • docker-compose logs -f nethermind_telemetry

   The Validator’s IP address and enode will change. Once you have completed the migration, fill out the installation summary form (for Volta here, and for Energy Web Chain here) to provide the new IP and enode information. You can also contact netops@energyweb.org to provide notification of the migration.

To change parameters of the validator client you have to change the config file of your validator node.

In the host, make sure you have root access and go into the config folder

Now you can edit the parity-signing.toml file that should look similar to this:

[parity]
chain = "/parity/config/chainspec.json"
no_persistent_txqueue = true

[rpc]
disable = false
port = 8545
interface = "0.0.0.0"
cors = []
apis = ["eth", "net", "parity", "web3"]

[websockets]
disable = false
interface = "0.0.0.0"
port = 8546

[ipc]
disable = true

[secretstore]
disable = true

[network]
port = 30303
min_peers = 25
max_peers = 50
discovery = true
warp = false
allow_ips = "all"
snapshot_peers = 0
max_pending_peers = 64

[footprint]
db_compaction = "ssd"

[snapshots]
enable = false

[mining]
force_sealing = true
usd_per_tx = "0.000000000000000001"
usd_per_eth = "1"
price_update_period = "hourly"
min_gas_price = 1
gas_cap = "4000000"
gas_floor_target = "4000000"
tx_gas_limit = "4000000"
extra_data = "validator-XXX"
engine_signer = "0xXXX"

[account]
password = ["XXX"]
keys_iterations = XXX

Most parameters should not be changed. But there are several parameters that can be adapted.

Warp Sync

If your node is having trouble syncing after a client update, restarting the node, or an unplanned outage, you can speed up the syncing process by enabling warp sync:

• Under [network] change: warp = true

• Restart docker container: docker-compose up -d

For more information, see https://openethereum.github.io/Warp-Sync

Minimum Gas-Price

The minimum gas price defines the lowest price that you allow to be paid for a transaction in Wei/Gas to still be included into a block by your validator node. Every transaction that has a gas price below that will be ignored. What price to set should depend largely on the transaction volume at the current moment. Every validator can choose this parameter individually. If it is too low, there is little spam protection and spammers can just fill up the blocks for little cost. If it is too high, it unjustifiably increases the price per transaction and might deter usage.

usd_per_tx defines the amount of USD to be paid for a basic transaction of sending tokens. The minimum gas price is set accordingly.

usd_per_eth defines the USD value of one token.

price_update_period defines the time that is allowed to pass between each gas price update. T may be daily, hourly, a number of seconds, or a time string of the form "2 days", "30 minutes" etc.

min_gas_price defines the minimum gas price in Wei/Gas and overrides usd_per_tx.

Block Gas Limit

Validators can choose the size of the blocks and thereby the number of transactions to include in a block. The bigger the blocks, the higher the transaction throughput. But if the block is too big, executing all transactions in that block might create problems with latency. Therefore the EWF will advise on a target blockgas limit. Once the limit is changed in the config file it will slowly increase until it has reached the defined limit. If validators do not agree on a limit, it will increase and decrease depending on who proposed the block.

gas_cap defines the absolute maximum amount of gas that will be allowed in a block depending on the transaction volume.

gas_floor_target defines the block size that the validator node targets when creating a new block.

tx_gas_limit defines the maximum gas amount that is allowed for a single transaction.

You can read about all possible configurations here: https://openethereum.github.io/Configuring-OpenEthereum.html

To restart the validator node, run

The parity client should now be running with the new configurations.