[Solved] Shards sync stalling for multiple nodes

[adrian]

I’ve got a few vnodes pending for shard 4 right now and they’re all stalling because of sync errors in other shards:

2021-05-15 10:04:52.717 shardsyncprocess.go:173 [ERR] Syncker log : Insert shard block from pool fail 1207086 [19 243 147 61 144 79 43 142 248 49 14 252 184 24 85 106 82 29 167 70 181 133 240 226 169 102 228 68 2 228 216 4] -1000: Unexpected error 
 -1000: Unexpected error 
 Invalid Signature!
Unexpected error
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.NewConsensusError
        incognito-chain/consensus_v2/blsbftv2/error.go:56
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.validateBLSSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:164
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.ValidateCommitteeSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:94
github.com/incognitochain/incognito-chain/consensus_v2.(*Engine).ValidateBlockCommitteSig
        incognito-chain/consensus_v2/key.go:172
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
        incognito-chain/blockchain/shardprocess.go:181
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlock
        incognito-chain/blockchain/shardchain.go:237
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).insertShardBlockFromPool
        incognito-chain/syncker/shardsyncprocess.go:172
runtime.goexit
        go/src/runtime/asm_amd64.s:1357
Unexpected error

2021-05-15 10:37:15.129 shardsyncprocess.go:173 [ERR] Syncker log : Insert shard block from pool fail 1207749 [224 251 232 60 44 178 131 209 196 101 223 95 1 226 8 8 63 17 214 154 249 7 214 240 237 17 38 167 49 50 213 234] -1000: Unexpected error 
 -1000: Unexpected error 
 Invalid Signature!
Unexpected error
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.NewConsensusError
        incognito-chain/consensus_v2/blsbftv2/error.go:56
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.validateBLSSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:164
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.ValidateCommitteeSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:94
github.com/incognitochain/incognito-chain/consensus_v2.(*Engine).ValidateBlockCommitteSig
        incognito-chain/consensus_v2/key.go:172
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
        incognito-chain/blockchain/shardprocess.go:181
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlock
        incognito-chain/blockchain/shardchain.go:237
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).insertShardBlockFromPool
        incognito-chain/syncker/shardsyncprocess.go:172
runtime.goexit
        go/src/runtime/asm_amd64.s:1357
Unexpected error

Rebooting the nodes didn’t help. Running mainnet_20210426_1

[adrian]

@duc these are likely stuck from a previous earning cycle and now future iterations will be stuck as well as it will never pass the failing shard. We should probably stop syncing the shard if the node is no longer pending/earning, and send a report if it errors continuously like this.

[adrian]

And another one, currently active in committee

2021-05-15 10:28:13.836 utils.go:100 [ERR] Syncker log : Insert block 1207726 hash 10de29c0ed5d9da1d3f7073b22e2625c42f319c651baf98785d876fc530a3ded got error -1000: Unexpected error 
 -1000: Unexpected error 
 Invalid Signature!
Unexpected error
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.NewConsensusError
        incognito-chain/consensus_v2/blsbftv2/error.go:56
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.validateBLSSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:164
github.com/incognitochain/incognito-chain/consensus_v2/blsbftv2.ValidateCommitteeSig
        incognito-chain/consensus_v2/blsbftv2/validation.go:94
github.com/incognitochain/incognito-chain/consensus_v2.(*Engine).ValidateBlockCommitteSig
        incognito-chain/consensus_v2/key.go:172
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
        incognito-chain/blockchain/shardprocess.go:181
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlock
        incognito-chain/blockchain/shardchain.go:237
github.com/incognitochain/incognito-chain/syncker.InsertBatchBlock
        incognito-chain/syncker/utils.go:94
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).streamFromPeer
        incognito-chain/syncker/shardsyncprocess.go:300
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).syncShardProcess
        incognito-chain/syncker/shardsyncprocess.go:204
runtime.goexit
        go/src/runtime/asm_amd64.s:1357
Unexpected error

[adrian]

A friend is also experiencing this on his vnodes, this might be impacting validators globally @Support

[kabamaru]

I’ve been seeing this too on one node on shard 4 (?):

2021-05-15 10:54:03.036 beaconsyncprocess.go:252 [ERR] Syncker log : Insert beacon block from pool fail 1208369 [48 189 106 1 157 5 178 165 167 154 195 91 181 24 56 132 12 234 96 3 211 99 177 63 156 131 158 58 152 243 113 5] -1018: ShardState Hash Error 
 Expect shard state hash to be 43c88a4b3dad04913b567c7a819ad82f251d5c37cb813c1f27f126d5af4dcec2
ShardState Hash Error
github.com/incognitochain/incognito-chain/blockchain.NewBlockChainError
	/home/code/go/src/incognito-chain/blockchain/error.go:383
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingBeaconBlock
	/home/code/go/src/incognito-chain/blockchain/beaconprocess.go:249
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertBeaconBlock
	/home/code/go/src/incognito-chain/blockchain/beaconprocess.go:138
github.com/incognitochain/incognito-chain/blockchain.(*BeaconChain).InsertBlock
	/home/code/go/src/incognito-chain/blockchain/beaconchain.go:209
github.com/incognitochain/incognito-chain/syncker.(*BeaconSyncProcess).insertBeaconBlockFromPool
	/home/code/go/src/incognito-chain/syncker/beaconsyncprocess.go:251
runtime.goexit
	/home/code/sdk/go1.13.15/src/runtime/asm_amd64.s:1357

[adrian]

pNode also stalling

[duc]

working on the issue now @adrian, will get you updated once it’s fixed.

[duc]

Hey guys, please follow the post to get the nodes’ issues fixed. If you have any other questions, feel free to reach out to @ support for help. Sorry again!