Official vNode Troubleshooting Guide

It’s finally here! The official vNode troubleshooting guide. No longer do you have to search around the forum for answers to your questions.


Use the Node Monitor to check for issues (monitor.incognito.org).
When you see an issue come here to fix it.


This guide assumes you have followed the official installation/setup method for your vNode(s) which can be found here: How to setup your own node in a blink of an eye (Now support multi-nodes per host)

If an alternative installation/setup method was used the only parts that would need modification are the node data locations found in steps 3, 4, and 5.

Step 1 - Check Ports

Please make sure your vNodes ports are open and accessible (please reference the setup guide used). Visit a port checking website (eg. https://www.yougetsignal.com/tools/open-ports/). If there is a green flag and the ports are open you’re good to move to Step Two.

If, however, you see a red flag and the ports are closed please open the port on your router/server (router guide: https://portforward.com/router.htm) (server guide: reference your firewall settings or ask your VPS provider)

Step 2 - Restart Docker

Use the following command via terminal on vNode:

sudo docker restart $(sudo docker ps -aq) && sudo rm -rfv /home/incognito/node_data_*/*.log

After 5 or so minutes check the node monitor again (monitor.incognito.org). If you continue to have issues work through the following steps until your issue(s) are resolved.

Step 3 - Force Check Update

Run the following code to stop all incognito docker containers

sudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)


(If have more than one node you should change inc_mainnet_* to the specific node you're having issues with)

Wait for docker to list the containers as it closes them. After this finishes, you can check they have closed with

docker ps

Next, we will re-run the script with

./inc*

After a few minutes, you should see your nodes running under ‘docker ps’. Now check the Node Monitor and if issues still arise move on to the next step.

Step 4 - Send Logs to Devs

At this point, our devs need to take a more in-depth look into your logs to isolate the problem.

Please cd into your node data folder. If you are using Rocky's Node Guide then the command should be:

cd /home/incognito/node_data_0

(if you have more than one node please change node_data_0 to the appropriate number. node counting starts with 0.)

Now run the following command:

apt install pastebinit -y && pastebinit -b sprunge.us -i *-`date +%Y`-`date +%m`-`date +%d`.log

The terminal will spit out a web URL. Please select the URL then ‘ctrl + right click’ then select copy and paste the URL into a message to the @support account. Please also include a brief response to the error(s) you’re seeing. After you send this report please continue on to step 5.

Step 5 - Remove Problematic Data

Please ensure all other steps have been completed prior to this step as this step deletes data that may take a while to redownload!

This step has been broken up into two sections part 5a deals with removing and resyncing beacon data and part 5b deals with removing shard data.

In the event of both shard and beacon stalls please remove the beacon data first (5a) followed by the shard data removal (5b) after complete resync of beacon data.

Step 5a - Remove Problematic Beacon

In the event of beacon data that can not be fixed with the above methods, it is recommended to delete the beacons data to clear the stall.

First, we will stop our docker containers from running again with:

sudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)

Now run the following command to clear the beacon data (the node number has been intentionally replaced with # character to prevent someone from copying and pasting without reading):

cd /home/incognito/node_data_#/mainnet/block/beacon/ && rm -rfv * && cd

After this command has been completed please run the following to restart all nodes:

sudo bash inc*

Step 5b - Remove Problematic Shard

If you notice a stalled shard that does not fix itself in a reasonable amount of time it is best to delete that shard’s data and resync.

First, we will stop our docker containers from running again with:

sudo docker container stop $(sudo docker container ls -q --filter name=inc_mainnet_*)

Next, the following command will cd into a specific shard and then wipe data (the node number and shard number have been intentionally replaced with # character to prevent someone from copying and pasting without reading):

cd /home/incognito/node_data_#/mainnet/block/shard#/ && sudo rm -rfv * && cd

Feel free to rerun this command for any shards that are stalling. After you have finished rerun the setup script with the following:

sudo bash inc*

Bonus Step

Bootstrap your node to quickly get to Sync State Latest:






As always, if anyone has any questions or comments please feel free to drop them below or contact the support account.

9 Likes

Thank you SOOOOO much for this.

2 Likes

Hey @Jared, can you please confirm that we need not one port, but in fact 2 ports open and accessible :-
Port #1: Node Port , and
Port #2: RPC Port

Also, do you think it would be useful to include a link to @Rocky 's vNode installation scripts - How to setup your own node in a blink of an eye (Now support multi-nodes per host) as I’m not unsure if this troubleshooting guide is based on Rocky’s scripts or the old vNode installation scripts, or does it not matter?

I have updated the guide to indicate ports* and also included text to reference the installation/setup guide used. The reason Rocky’s script does not mention opening ports is due to most VPS hosts already allowing all ports. This is step 1 to hopefully be an easy fix and something that should be checked at the beginning as a precautionary measure.

See the post below for updated response.

This troubleshooting guide can be used for all vNodes. I have updated the post mentioning areas where changes will need to be made if using other installation/setup methods.

1 Like

Hey @Linnovations,

I just got off a call with the devs. I asked them for clarification on this. In the past, the discovery method required the node port to be open. Now that has been changed to only require the RPC port to be opened if you require this functionality. As majority of users will not be using the RPC port of their node this does not matter if it is open or closed.

Removing this step from the guide and will add it back if the connection method changes.

1 Like

Looks like you mean step 4 here. My question is, after sending the logs in step 3, immediately go and dump the shard/beacon? Or is it wait to hear from support, then proceed as advised?

Sorry I had removed the original step 1 and thought I changed all correctly. I will update now.

Yes go ahead and pull logs and send them to the devs and then delete the data. The logs are for them to see what happened and try to prevent in the future. They will instruct you to delete the data anyways.

1 Like

Thanks @Jared for bringing this up with the devs on your call. So, just to be clear, the only port we need open is the RPC port(s) to our vNode(s), is that correct?

Actually no ports need to be opened now that nodes talk directly with the highway. The only reason to open the RPC port is if you plan on doing advance things like calls to your nodes RPC.

The average user will not have to open any ports at all, sorry for the confusion.

Hey @Jared, On a different forum thread I brought up a questions about open Ports and I got the following reply from @fredlee

I thought I’d share with you as you might want to mention this in your excellent troubleshooting guide above.

@Linnovations,

I originally got conflicting information from the devs regarding this. I then tried shutting down all ports on my VPS and started seeing stalling on all nodes. I have re-added the Step 1 for checking for open ports.

If you are using a VPS the ports should not be blocked so this should not be a problem for most users. Users who are running vNodes behind a firewall (ie home run servers, etc) will need to manually port forward their routers.

I fall into the above category. FYI, I modified my Home firewall/router to only have my RPC ports open (following the comment from @fredlee ). My 2 x vNode seem to be “online” as reported in the Incognito App and on the Node Monitor website.

Thanks for discussing this with the devs and for putting the Troubleshooting Guide together. This guide is essential for folks like myself as I’m new to linux but want to help support this decentralise privacy network which is one of the most important things for freedom.

2 Likes

I got error below after ran the command, any other option?

ms@sv:/home/incognito/node_data_0$ sudo apt install pastebinit -y && pastebinit -b sprunge.us -i *-date +%Y-date +%m-date +%d.log
[sudo] password for sky:
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following NEW packages will be installed:
pastebinit
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 14.2 kB of archives.
After this operation, 156 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu impish/main amd64 pastebinit all 1.5.1-1 [14.2 kB]
Fetched 14.2 kB in 0s (114 kB/s)
Selecting previously unselected package pastebinit.
(Reading database … 165914 files and directories currently installed.)
Preparing to unpack …/pastebinit_1.5.1-1_all.deb …
Unpacking pastebinit (1.5.1-1) …
Setting up pastebinit (1.5.1-1) …
Processing triggers for man-db (2.9.4-2) …
Failed to contact the server: <urlopen error [Errno 32] Broken pipe>
sky@DESKTOP:/home/incognito/node_data_0$

This is due to the file being too large. Try deleting prior to running the script.

what is the script by “sudo bash inc*” ?
when i try this, I got nothing.

Sorry about that, the script name has changed to blink.sh. I’ll update the guide soon.