vnode syncing error

quinto · 7 April 2021 22:14

I’m having problem syncing vnode. This is a new vnode I started running from scratch about two days ago. I stopped it and restarted it at about 30 hours into the process. But afterwards I just keep getting these errors in the log:

2021-04-07 21:54:03.203 utils.go:89 [ERR] Syncker log : Insert block 169584 hash [5 87 172 29 243 65 60 79 155 134 4 105 62 27 107 244 77 51 112 13 237 248 214 206 31 177 120 217 165 193 118 185] got error -1051: Instruction Hash Error
 Expect instruction hash to be de4c5dc7c63b9e3ceb8c0941dcd217a8f484aeb42008548cee25c4a77bf24b0a but get 0000000000000000000000000000000000000000000000000000000000000000 at block 169584 hash b976c1a5d978b11fced6f8ed0d70334df46b1b3e6904869b4f3c41f31dac5705
Instruction Hash Error
github.com/incognitochain/incognito-chain/blockchain.NewBlockChainError
        /Users/autonomous/projects/incognito-chain/blockchain/error.go:365
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingShardBlock
        /Users/autonomous/projects/incognito-chain/blockchain/shardprocess.go:325
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertShardBlock
        /Users/autonomous/projects/incognito-chain/blockchain/shardprocess.go:147
github.com/incognitochain/incognito-chain/blockchain.(*ShardChain).InsertBlk
        /Users/autonomous/projects/incognito-chain/blockchain/shardchain.go:222
github.com/incognitochain/incognito-chain/syncker.InsertBatchBlock
        /Users/autonomous/projects/incognito-chain/syncker/utils.go:83
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).streamFromPeer
        /Users/autonomous/projects/incognito-chain/syncker/shardsyncprocess.go:286
github.com/incognitochain/incognito-chain/syncker.(*ShardSyncProcess).syncShardProcess
        /Users/autonomous/projects/incognito-chain/syncker/shardsyncprocess.go:190
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357, Committee of epoch [....]

It had similar errors for shard 0, 2 and 6, and the following also kept repeating without any progress:

2021-04-07 22:01:56.654 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 433077 to 1127179 fromShard 2
2021-04-07 22:01:56.654 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer  from cID 2, [433077 1127179]
2021-04-07 22:01:56.654 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [433077..1127179] len 2, from 2 to 2, uuid = 4ce6e63a-d76c-4291-bb95-f5c0f7da2fb7
2021-04-07 22:01:56.769 connmanager.go:489 [ERR] Peerv2 log: [stream] rpc error: code = Unknown desc = Sync too fast, last time sync blocks from 909062 is 2021-04-07 22:01:54.406601635 +0000 UTC
2021-04-07 22:01:56.769 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 909062 to 1127246 fromShard 6
2021-04-07 22:01:56.769 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer  from cID 6, [909062 1127246]
2021-04-07 22:01:56.769 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [909062..1127246] len 2, from 6 to 6, uuid = a5508c00-e78f-46bd-b66c-ece017ba7457
2021-04-07 22:01:57.280 connmanager.go:489 [ERR] Peerv2 log: [stream] rpc error: code = Unknown desc = Sync too fast, last time sync blocks from 169583 is 2021-04-07 22:01:54.96715273 +0000 UTC
2021-04-07 22:01:57.281 connmanager.go:410 [INF] Peerv2 log: [SyncShard] from 169583 to 1123960 fromShard 0
2021-04-07 22:01:57.281 connmanager.go:469 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer  from cID 0, [169583 1123960]
2021-04-07 22:01:57.281 blockrequester.go:235 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [169583..1123960] len 2, from 0 to 0, uuid = 710affcf-4ede-402c-bc43-611c06101997

My questions are:

Is there a way to fix the problem without having to sync everything from scratch again?
Why is block syncing so fragile?
Is there a way of shutting down a running vnode without corrupting the database?

This is my 5th or 6th time of having to sync everything all over again (and I have a monthly internet bill to watch out for). Any help is greatly appreciated!

0xkumi · 8 April 2021 04:10

Hello,

Sorry for this inconvenience. You only have to clear data if you get a corrupted database(the log should say so). Your log shows it is another problem.
The cause could be running the old version.
Did you run from our docker tag? What is your latest tag?

quinto · 8 April 2021 04:22

I’m running 20210406_1 docker image, which should be the latest.

Edit: now I’m running 20210406_1 , but a couple days ago it must be the version before it.

0xkumi · 8 April 2021 05:33

Do you run geth at the same machine?

quinto · 8 April 2021 05:41

Yes, I do

quinto · 10 April 2021 05:39

Also, while the syncing was incomplete for 3 shards, the incognito process kept consuming HUGE traffic. I ended up using up almost 1TB in just 2 days AFTER I noticed the syncing problem but left the process running. Something is seriously wrong here. I’ve seen similar reports that vnode or pnode acted like an infected botnet and took almost all available bandwidth. Believe the same is happening to me.

quinto · 12 April 2021 15:11

Now that I tried again with a fresh sync, and got exactly the same kind of error at height 169584:

Insert block 169584 hash [5 87 172 29 243 65 60 79 155 134 4 105 62 27 107 244 77 51 112 13 237 248 214 206 31 177 120 217 165 193 118 185] got error -1051: Instruction Hash Error

quinto · 13 April 2021 21:28

The problem seems to be geth as hinted by [Solved] Mainnet fullnode sync issue: Shard 6 blocks stopped to be imported - Community / Builders - Incognito

I’m now running incognito (without docker) with the following command:

GETH_NAME=mainnet.infura.io/v3/[infura project id] \
GETH_PROTOCOL=https \
GETH_PORT=""  \
./incognito [other parameters]

The “infura project id” can be obtained once you sign up a free tier infura service.

The previous way of running a geth lightclient in docker will not work with the incognito binary for syncing purpose.

Jared · 14 April 2021 03:33

As of when? Do you use the multi-vNode script?

quinto · 14 April 2021 03:41

Yes, that was what I used. Two docker containers, one incognito, the other geth. If anyone has recently successfully synced everything from scratch, I’d like to hear about it.

I only succeeded after switching to infura service.