Official vNode Troubleshooting Guide

It’s finally here! The official vNode troubleshooting guide. No longer do you have to search around the forum for answers to your questions.


Use the Node Monitor to check for issues (monitor.incognito.org).
When you see an issue come here to fix it.


This guide assumes you have followed the official installation/setup method for your vNode(s) which can be found here: How to setup your own node in a blink of an eye (Now support multi-nodes per host)

If an alternative installation/setup method was used the only parts that would need modification are the node data locations found in steps 2, 3, and 4.

Step 1 - Restart Docker

Use the following command via terminal on vNode:

sudo docker restart $(sudo docker ps -aq)

After 5 or so minutes check the node monitor again (monitor.incognito.org). If you continue to have issues work through the following steps until your issue(s) are resolved.

Step 2 - Force Check Update

Run the following code to stop all incognito docker containers
docker container stop $(docker container ls -q --filter name=inc_mainnet_*) && rm -rfv /home/incognito/node_data_*/*.log

(If have more than one node you should change inc_mainnet_* to the specific node you're having issues with)

Wait for docker to list the containers as it closes them. After this finishes, you can check they have closed with

docker ps

Next, we will re-run the script with

./inc*

After a few minutes, you should see your nodes running under ‘docker ps’. Now check the Node Monitor and if issues still arise move on to the next step.

Step 3 - Send Logs to Devs

At this point, our devs need to take a more in-depth look into your logs to isolate the problem.

Please cd into your node data folder. If you are using Rocky's Node Guide then the command should be:

cd /home/incognito/node_data_0

(if you have more than one node please change node_data_0 to the appropriate number. node counting starts with 0.)

Now run the following command:

apt install pastebinit -y && pastebinit -b sprunge.us -i *-`date +%Y`-`date +%m`-`date +%d`.log

The terminal will spit out a web URL. Please select the URL then ‘ctrl + right click’ then select copy and paste the URL into a message to the @support account. Please also include a brief response to the error(s) you’re seeing. After you send this report please continue on to step 4.

Step 4 - Remove Problematic Data

Please ensure all other steps have been completed prior to this step as this step deletes data that may take a while to redownload!

This step has been broken up into two sections part 4a deals with removing and resyncing beacon data and part 4b deals with removing shard data.

In the event of both shard and beacon stalls please remove the beacon data first (4a) followed by the shard data removal (4b) after complete resync of beacon data.

Step 4a - Remove Problematic Beacon

In the event of beacon data that can not be fixed with the above methods, it is recommended to delete the beacons data to clear the stall.

First, we will stop our docker containers from running again with:

docker container stop $(docker container ls -q --filter name=inc_mainnet_*)

Now run the following command to clear the beacon data (the node number has been intentionally replaced with # character to prevent someone from copying and pasting without reading):

cd /home/incognito/node_data_#/mainnet/block/beacon/ && rm -rfv * && cd

After this command has been completed please run the following to restart all nodes:

./inc*

Step 4b - Remove Problematic Shard

If you notice a stalled shard that does not fix itself in a reasonable amount of time it is best to delete that shard’s data and resync.

First, we will stop our docker containers from running again with:

docker container stop $(docker container ls -q --filter name=inc_mainnet_*)

Next, the following command will cd into a specific shard and then wipe data (the node number and shard number have been intentionally replaced with # character to prevent someone from copying and pasting without reading):

cd /home/incognito/node_data_#/mainnet/block/shard#/ && rm -rfv * && cd

Feel free to rerun this command for any shards that are stalling. After you have finished rerun the setup script with the following:

./inc*



As always, if anyone has any questions or comments please feel free to drop them below or contact the support account.
7 Likes

Thank you SOOOOO much for this.

2 Likes

Hey @Jared, can you please confirm that we need not one port, but in fact 2 ports open and accessible :-
Port #1: Node Port , and
Port #2: RPC Port

Also, do you think it would be useful to include a link to @Rocky 's vNode installation scripts - How to setup your own node in a blink of an eye (Now support multi-nodes per host) as I’m not unsure if this troubleshooting guide is based on Rocky’s scripts or the old vNode installation scripts, or does it not matter?

I have updated the guide to indicate ports* and also included text to reference the installation/setup guide used. The reason Rocky’s script does not mention opening ports is due to most VPS hosts already allowing all ports. This is step 1 to hopefully be an easy fix and something that should be checked at the beginning as a precautionary measure.

See the post below for updated response.

This troubleshooting guide can be used for all vNodes. I have updated the post mentioning areas where changes will need to be made if using other installation/setup methods.

1 Like

Hey @Linnovations,

I just got off a call with the devs. I asked them for clarification on this. In the past, the discovery method required the node port to be open. Now that has been changed to only require the RPC port to be opened if you require this functionality. As majority of users will not be using the RPC port of their node this does not matter if it is open or closed.

Removing this step from the guide and will add it back if the connection method changes.

1 Like

Looks like you mean step 4 here. My question is, after sending the logs in step 3, immediately go and dump the shard/beacon? Or is it wait to hear from support, then proceed as advised?

Sorry I had removed the original step 1 and thought I changed all correctly. I will update now.

Yes go ahead and pull logs and send them to the devs and then delete the data. The logs are for them to see what happened and try to prevent in the future. They will instruct you to delete the data anyways.

1 Like

Thanks @Jared for bringing this up with the devs on your call. So, just to be clear, the only port we need open is the RPC port(s) to our vNode(s), is that correct?

Actually no ports need to be opened now that nodes talk directly with the highway. The only reason to open the RPC port is if you plan on doing advance things like calls to your nodes RPC.

The average user will not have to open any ports at all, sorry for the confusion.