Problems running a fullnode

I’ve been running a full node (mainnet_20210106_1 branch) for the past couple weeks, and noticed some problems. Sometimes it appears that the node got stuck, or some part got stuck (probably a shard failed to sync?). Some of the symptoms are given below(not happening at the same time, but all happened at one point or another):

  1. Querying balance of an account that used to return the correct amount suddently started to return 0.
  2. Querying balance of an account would return old value, but from incognito app I could see the amount already been updated. It is not just a problem of temporary out-of-sync, it stayed that way for hours.
  3. Querying pdex exchange rate returned old rates compared to what was in the incognito app, and they do not change for hours.

Whenever something like the above happened, I had to restart the node manually. Usually the restart eventually fixed the problem, but sometimes I had to restart several times. It has been a frustrating experience having to constantly watch out for stuck nodes. Have anybody else noticed similar problems? What can I do to debug what’s going on? Is there anything like a health check UI (or API) that I could use?

@quinto The node you’re referring to, is it a stand alone full node? Or is it a Validator?

Standalone full node, not validator. I use it for relaying transactions and serving API.

Also, I do see errors like below by going through the logs (even when it appears to be running fine), is it something to worry about?

estimatefee.go:220 [ERR] Mempool log: RegisterBlock: intermediate block not recorded; current height is 919077; new height is 959188, shardID 7
shardprocess.go:1068 [DBG] BlockChain log: -1041: Register Fee Estimator Error
 RegisterBlock: intermediate block not recorded; current height is 919077; new height is 959188
Register Fee Estimator Error
github.com/incognitochain/incognito-chain/blockchain.NewBlockChainError
        /home/incognito-chain/blockchain/error.go:365
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).processStoreShardBlock
        /home/incognito-chain/blockchain/shardprocess.go:1068
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
        /home/incognito-chain/blockchain/shardprocess.go:191
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlk
        /home/incognito-chain/blockchain/shardchain.go:213
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).insertShardBlockFromPool
        /home/incognito-chain/syncker/shardsyncprocess.go:166
runtime.goexit

This error does look concerning, because it kept repeating and the “current height” never changes.

estimatefee.go:220 [ERR] Mempool log: RegisterBlock: intermediate block not recorded; current height is 919077; new height is 959188, shardID 7

I guess it means shardID 7 wasn’t fully synced? It actually happened to all shards, not just 7.

Is there a way to force syncing? I tried restart a couple times, still doesn’t work.

What does your startup script look like?

For example:

./incognito --discoverpeersaddress "mainnet-bootnode.incognito.org:9330" --testnet false --nodemode "relay" --relayshards "all" --datadir "data/full_node" --listen "0.0.0.0:9433" --externaladdress "0.0.0.0:9433" --norpcauth --rpclisten "0.0.0.0:9334" --txpoolmaxtx 100000 --loglevel debug

Except I also had --rpcwslisten 0.0.0.0:19334, the rest of the command is exactly as what you described.

I’m trying to run a second node on a different machine to see if it has similar problems. But it’ll take a while to sync…