It’s finally here! The official vNode troubleshooting guide. No longer do you have to search around the forum for answers to your questions.
Use the Node Monitor to check for issues (monitor.incognito.org).
When you see an issue come here to fix it.
This guide assumes you have followed the official installation/setup method for your vNode(s) which can be found here: How to setup your own node in a blink of an eye (Now support multi-nodes per host)
If an alternative installation/setup method was used the only parts that would need modification are the node data locations found in steps 3, 4, and 5.
Step 1 - Check Ports
Please make sure your vNodes ports are open and accessible (please reference the setup guide used). Visit a port checking website (eg. https://www.yougetsignal.com/tools/open-ports/). If there is a green flag and the ports are open you’re good to move to Step Two.
If, however, you see a red flag and the ports are closed please open the port on your router/server (router guide: https://portforward.com/router.htm) (server guide: reference your firewall settings or ask your VPS provider)
Step 2 - Restart Docker
Use the following command via terminal on vNode:
sudo docker restart $(sudo docker ps -aq) && sudo rm -rfv /home/incognito/node_data_*/*.log
After 5 or so minutes check the node monitor again (monitor.incognito.org). If you continue to have issues work through the following steps until your issue(s) are resolved.
Step 3 - Force Check Update
Run the following code to stop all incognito docker containerssudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)
(If have more than one node you should change inc_mainnet_* to the specific node you're having issues with)
Wait for docker to list the containers as it closes them. After this finishes, you can check they have closed with
docker ps
Next, we will re-run the script with
./inc*
After a few minutes, you should see your nodes running under ‘docker ps’. Now check the Node Monitor and if issues still arise move on to the next step.
Step 4 - Send Logs to Devs
At this point, our devs need to take a more in-depth look into your logs to isolate the problem.Please cd into your node data folder. If you are using Rocky's Node Guide then the command should be:
cd /home/incognito/node_data_0
(if you have more than one node please change node_data_0 to the appropriate number. node counting starts with 0.)
Now run the following command:
apt install pastebinit -y && pastebinit -b sprunge.us -i *-`date +%Y`-`date +%m`-`date +%d`.log
The terminal will spit out a web URL. Please select the URL then ‘ctrl + right click’ then select copy and paste the URL into a message to the @support account. Please also include a brief response to the error(s) you’re seeing. After you send this report please continue on to step 5.
Step 5 - Remove Problematic Data
Please ensure all other steps have been completed prior to this step as this step deletes data that may take a while to redownload!This step has been broken up into two sections part 5a deals with removing and resyncing beacon data and part 5b deals with removing shard data.
In the event of both shard and beacon stalls please remove the beacon data first (5a) followed by the shard data removal (5b) after complete resync of beacon data.
Step 5a - Remove Problematic Beacon
In the event of beacon data that can not be fixed with the above methods, it is recommended to delete the beacons data to clear the stall.
First, we will stop our docker containers from running again with:
sudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)
Now run the following command to clear the beacon data (the node number has been intentionally replaced with # character to prevent someone from copying and pasting without reading):
cd /home/incognito/node_data_#/mainnet/block/beacon/ && rm -rfv * && cd
After this command has been completed please run the following to restart all nodes:
sudo bash inc*
Step 5b - Remove Problematic Shard
If you notice a stalled shard that does not fix itself in a reasonable amount of time it is best to delete that shard’s data and resync.
First, we will stop our docker containers from running again with:
sudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)
Next, the following command will cd into a specific shard and then wipe data (the node number and shard number have been intentionally replaced with # character to prevent someone from copying and pasting without reading):
cd /home/incognito/node_data_#/mainnet/block/shard#/ && sudo rm -rfv * && cd
Feel free to rerun this command for any shards that are stalling. After you have finished rerun the setup script with the following:
sudo bash inc*
Bonus Step
Bootstrap your node to quickly get to Sync State Latest
:
As always, if anyone has any questions or comments please feel free to drop them below or contact the support account.