Common fixes for stalled, non voting and offline nodes.

Hi everyone, I see that there’s a few people that are posting a few queries on why their nodes aren’t voting consistently after seeing their status on Node Monitor. I thought we could try centralize common fixes in this one thread.

Potential examples are:
How to fix a zero voting Node?

What to do if your Node shows a shard stall?

What to do when the beacon stalls?

What to do if your node is offline but is online using Ensi and Nito (Telegram)?

Anyone else feel free to add any other potential issues.

It would also help if the team highlights the actual commands you need to use as not all vnode users are the most savvy coders (I.e. me!) but can work out basic powershell commands.

@Support @duc

5 Likes

Could someone explain what "Vote Stats : vote percentages of the two most recent epochs where the node was in a shard committee. " means in the node monitor. More specifically, how can my node earn normally and show earnings along with a 0 vote count % for a specific block? How can a node earn with a vote count of zero ?

How do we fix a node with zero % vote counts (but is otherwise “online”, synced to “latest” and is earning)?

Hey @SPAddict25

Here is some tricks to resolve these issues:

  • method1: restart docker container, wait for 5 mins then check status on node monitor again
sudo docker restart $(sudo docker ps -aq)
  • method2: kill the run.sh process, stop docker cointainer, execute the run.sh again so that it pulling new docker tag
[email protected]:~$ ps -ef | grep run.sh
khanhlh  29208     1  0 09:01 pts/17   00:00:00 bash run.sh
khanhlh  29977 29068  0 09:04 pts/17   00:00:00 grep --color=auto run.sh
[email protected]:~$ kill -9 29208
[email protected]:~$ ps -ef | grep run.sh
khanhlh  30039 29068  0 09:05 pts/17   00:00:00 grep --color=auto run.sh
[email protected]:~$ bash run.sh

...
[email protected]:~$ 
[email protected]:~$ docker ps
CONTAINER ID        IMAGE                                         COMMAND                  CREATED             STATUS              PORTS                                                                  NAMES
6942e615b72a        incognitochain/incognito-mainnet:20210810_1   "/bin/bash run_incog…"   55 seconds ago      Up 54 seconds       0.0.0.0:9113->9334/tcp, 0.0.0.0:9313->9433/tcp                         inc_mainnet13
379688624d6e        incognitochain/incognito-mainnet:20210810_1   "/bin/bash run_incog…"   57 seconds ago      Up 55 seconds       0.0.0.0:9112->9334/tcp, 0.0.0.0:9312->9433/tcp                         inc_mainnet12
[email protected]:~$ ps -ef | grep run.sh
khanhlh  29208     1  0 09:01 pts/17   00:00:00 bash run.sh
khanhlh  29977 29068  0 09:04 pts/17   00:00:00 grep --color=auto run.sh
[email protected]:~$ kill -9 29208
[email protected]:~$ sudo docker stop $(sudo docker ps -aq)
[email protected]:~$ sudo docker system prune -a
[email protected]:~$ rm -rf data
  • Reading log: go to data folder, tail the log with keyword BEACON or SHARD, if it keep syncing blocks (compare to latest blocks on mainnet fullnode) then you node is good.

[email protected]:~/incognito-mainnet-data/data1$ ls
68.162-2021-08-15.log  68.162-2021-08-18.log  error.log  mainnet
[email protected]:~/incognito-mainnet-data/data1$ tail -f 68.162-2021-08-18.log | grep BEACON
2021-08-18 09:13:41.966 beaconprocess.go:187 [INF] BlockChain log: BEACON | Finish Insert new Beacon Block 30604, with hash f940f8c9e58625117b0e3bddfec489e4ba9281ded0b03571e3851645596dfc77
2021-08-18 09:13:41.966 beaconprocess.go:108 [INF] BlockChain log: BEACON | InsertBeaconBlock  30605 with hash dc70533dd67e696da17aa513d159be351dbb7bd6ae5b1598ff9ee9d902ada1c5
2021-08-18 09:13:41.968 beaconprocess.go:177 [INF] BlockChain log: BEACON | Update Committee State Block Height 30605 with hash dc70533dd67e696da17aa513d159be351dbb7bd6ae5b1598ff9ee9d902ada1c5
2021-08-18 09:13:41.968 beaconprocess.go:182 [INF] BlockChain log: BEACON | Process Store Beacon Block Height 30605 with hash dc70533dd67e696da17aa513d159be351dbb7bd6ae5b1598ff9ee9d902ada1c5
2021-08-18 09:13:41.970 beaconprocess.go:187 [INF] BlockChain log: BEACON | Finish Insert new Beacon Block 30605, with hash dc70533dd67e696da17aa513d159be351dbb7bd6ae5b1598ff9ee9d902ada1c5
2021-08-18 09:13:41.971 beaconprocess.go:108 [INF] BlockChain log: BEACON | InsertBeaconBlock  30606 with hash 1fa395d1ae2f4729eb39f804c1224ef1a8c9e6e0a968ac54ab5505b24708c66c
2021-08-18 09:13:41.973 beaconprocess.go:177 [INF] BlockChain log: BEACON | Update Committee State Block Height 30606 with hash 1fa395d1ae2f4729eb39f804c1224ef1a8c9e6e0a968ac54ab5505b24708c66c
2021-08-18 09:13:41.973 beaconprocess.go:182 [INF] BlockChain log: BEACON | Process Store Beacon Block Height 30606 with hash 1fa395d1ae2f4729eb39f804c1224ef1a8c9e6e0a968ac54ab5505b24708c66c
[email protected]:~$ tail -f incognito-mainnet-data/data1/68.162-2021-08-18.log | grep SHARD
2021-08-18 09:29:56.774 shardprocess.go:245 [INF] BlockChain log: SHARD 4 | SKIP Verify Post Processing, block height 2093 with hash 558f43bd39624a00586354ef090ae9f9e92df644439ee12085fdf830e61324ae 
2021-08-18 09:29:56.776 shardprocess.go:248 [INF] BlockChain log: SHARD 4 | Update Beacon Instruction, block height 2093 with hash 558f43bd39624a00586354ef090ae9f9e92df644439ee12085fdf830e61324ae 
2021-08-18 09:29:56.776 shardprocess.go:269 [INF] BlockChain log: SHARD 3 | Finish Insert new block 2946, with hash 1aa3ee4ae12370cf72fa5524c95db3e353f41ec66116263f94e2ca0516aa1865 🔗, Found 🔎 0 transactions, 0 cross shard transactions, 0 instruction
2021-08-18 09:29:56.776 shardprocess.go:254 [INF] BlockChain log: SHARD 4 | Update Committee State Block Height 2093 with hash 558f43bd39624a00586354ef090ae9f9e92df644439ee12085fdf830e61324ae
2021-08-18 09:29:56.776 shardprocess.go:140 [INF] BlockChain log: SHARD 3 | InsertShardBlock 2947 with hash d6f412fea2609ba163afc0c0805260777988f09576b652d9652552ce629bbeda Prev hash: 1aa3ee4ae12370cf72fa5524c95db3e353f41ec66116263f94e2ca0516aa1865
2021-08-18 09:29:56.777 shardprocess.go:201 [INF] BlockChain log: SHARD 0 | SKIP Verify Pre Processing, block height 2492 with hash c0980f284e8564a7a5b5b87c505ba343955a901352eeae7b6e02c415a69a842f
2021-08-18 09:29:56.776 shardprocess.go:260 [INF] BlockChain log: SHARD 4 | Store New Shard Block And Update Data, block height 2093 with hash 558f43bd39624a00586354ef090ae9f9e92df644439ee12085fdf830e61324ae 
2021-08-18 09:29:56.777 shardprocess.go:1048 [INF] BlockChain log: SHARD 4 | Process store block height 2093 at hash 558f43bd39624a00586354ef090ae9f9e92df644439ee12085fdf830e61324ae
2021-08-18 09:29:56.779 shardprocess.go:245 [INF] BlockChain log: SHARD 0 | SKIP Verify Post Processing, block height 2492 with hash c0980f284e8564a7a5b5b87c505ba343955a901352eeae7b6e02c415a69a842f 
2021-08-18 09:29:56.779 shardprocess.go:248 [INF] BlockChain log: SHARD 0 | Update Beacon Instruction, block height 2492 with hash c0980f284e8564a7a5b5b87c505ba343955a901352eeae7b6e02c415a69a842f 
2021-08-18 09:29:56.780 shardprocess.go:254 [INF] BlockChain log: SHARD 0 | Update Committee State Block Height 2492 with hash c0980f284e8564a7a5b5b87c505ba343955a901352eeae7b6e02c415a69a842f
2021-08-18 09:29:56.780 shardprocess.go:201 [INF] BlockChain log: SHARD 3 | SKIP Verify Pre Processing, block height 2947 with hash d6f412fea2609ba163afc0c0805260777988f09576b652d9652552ce629bbeda

  • grab the log and send it to @support: stop docker container and compress the log, use filezilla to do sftp, transfer the log-file from your vps to local
[email protected]:~/incognito-mainnet-data/data1$ docker stop inc_mainnet1
inc_mainnet1

[email protected]:~/incognito-mainnet-data/data1$ sudo tar -zcvf 20210818-log.tar.gz 68.162-2021-08-18.log 
68.162-2021-08-18.log
[email protected]:~/incognito-mainnet-data/data1$ ls 
20210818-log.tar.gz  68.162-2021-08-15.log  68.162-2021-08-18.log  error.log  mainnet

10 Likes

Wow…it’s been a while since I posted here on the forum but I keep informed and read the entries just about daily and well on this posting…well I was like holy smokes now talk about responsive, helpful, and professional demeanor and work by the dev team…excellent work guys and ladies…and yes a big shout out to @khanhj on his posting…thank you so much…awesome…ok Nuff said…lol… :blush: :sunglasses: :partying_face: :100: :+1:

2 Likes

Thank you so much and to the rest of the dev team. You’re doing fantastic work!

3 Likes

@khanhj Thank you.

2 Likes

post updated :point_up_2: :point_up_2: :point_up_2:

2 Likes