I’m having problem syncing vnode. This is a new vnode I started running from scratch about two days ago. I stopped it and restarted it at about 30 hours into the process. But afterwards I just keep getting these errors in the log:
2021-04-07 21:54:03.203 utils.go:89 [ERR] Syncker log : Insert block 169584 hash [5 87 172 29 243 65 60 79 155 134 4 105 62 27 107 244 77 51 112 13 237 248 214 206 31 177 120 217 165 193 118 185] got error -1051: Instruction Hash Error
Expect instruction hash to be de4c5dc7c63b9e3ceb8c0941dcd217a8f484aeb42008548cee25c4a77bf24b0a but get 0000000000000000000000000000000000000000000000000000000000000000 at block 169584 hash b976c1a5d978b11fced6f8ed0d70334df46b1b3e6904869b4f3c41f31dac5705
Instruction Hash Error
github.com/incognitochain/incognito-chain/blockchain.NewBlockChainError
/Users/autonomous/projects/incognito-chain/blockchain/error.go:365
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingShardBlock
/Users/autonomous/projects/incognito-chain/blockchain/shardprocess.go:325
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
/Users/autonomous/projects/incognito-chain/blockchain/shardprocess.go:147
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlk
/Users/autonomous/projects/incognito-chain/blockchain/shardchain.go:222
github.com/incognitochain/incognito-chain/syncker.InsertBatchBlock
/Users/autonomous/projects/incognito-chain/syncker/utils.go:83
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).streamFromPeer
/Users/autonomous/projects/incognito-chain/syncker/shardsyncprocess.go:286
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).syncShardProcess
/Users/autonomous/projects/incognito-chain/syncker/shardsyncprocess.go:190
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357, Committee of epoch [....]
It had similar errors for shard 0, 2 and 6, and the following also kept repeating without any progress:
2021-04-07 22:01:56.654 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 433077 to 1127179 fromShard 2
2021-04-07 22:01:56.654 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer from cID 2, [433077 1127179]
2021-04-07 22:01:56.654 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [433077..1127179] len 2, from 2 to 2, uuid = 4ce6e63a-d76c-4291-bb95-f5c0f7da2fb7
2021-04-07 22:01:56.769 connmanager.go:489 [ERR] Peerv2 log: [stream] rpc error: code = Unknown desc = Sync too fast, last time sync blocks from 909062 is 2021-04-07 22:01:54.406601635 +0000 UTC
2021-04-07 22:01:56.769 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 909062 to 1127246 fromShard 6
2021-04-07 22:01:56.769 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer from cID 6, [909062 1127246]
2021-04-07 22:01:56.769 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [909062..1127246] len 2, from 6 to 6, uuid = a5508c00-e78f-46bd-b66c-ece017ba7457
2021-04-07 22:01:57.280 connmanager.go:489 [ERR] Peerv2 log: [stream] rpc error: code = Unknown desc = Sync too fast, last time sync blocks from 169583 is 2021-04-07 22:01:54.96715273 +0000 UTC
2021-04-07 22:01:57.281 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 169583 to 1123960 fromShard 0
2021-04-07 22:01:57.281 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer from cID 0, [169583 1123960]
2021-04-07 22:01:57.281 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [169583..1123960] len 2, from 0 to 0, uuid = 710affcf-4ede-402c-bc43-611c06101997
My questions are:
- Is there a way to fix the problem without having to sync everything from scratch again?
- Why is block syncing so fragile?
- Is there a way of shutting down a running vnode without corrupting the database?
This is my 5th or 6th time of having to sync everything all over again (and I have a monthly internet bill to watch out for). Any help is greatly appreciated!