vNode crashing after being selected to committee

[bjeung]

This is a long running node that has been running for months. Was selected for committee this morning and then started crashing repeatedly

2023-04-30 17:15:16.776 poolmanager.go:53 [INF] Txpool log : Received new role 7 committee

2023-04-30 17:15:16.777 pool.go:139 [INF] Txpool log : Start transaction pool v1
2023-04-30 17:15:16.793 payment_v2.go:211 [ERR] Privacy V2 log : aggregated range proof setbytes error: edwards25519: invalid point encoding

2023-04-30 17:15:16.793 payment_v2.go:129 [ERR] Privacy V2 log : PaymentProofV2 setbytes error: -9004: Set bytes payment proof error edwards25519: invalid point encoding
Set bytes payment proof error
github.com/incognitochain/incognito-chain/privacy/errorhandler.NewPrivacyErr
/Users/autonomous/projects/incognito-chain/privacy/errorhandler/error.go:98
github.com/incognitochain/incognito-chain/privacy/privacy_v2.(*PaymentProofV2).SetBytes
/Users/autonomous/projects/incognito-chain/privacy/privacy_v2/payment_v2.go:212
github.com/incognitochain/incognito-chain/privacy/privacy_v2.(*PaymentProofV2).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/privacy/privacy_v2/payment_v2.go:127
encoding/json.(*decodeState).literalStore
/usr/local/go/src/encoding/json/decode.go:871
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:387
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:774
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/transaction/tx_ver2.(*TxToken).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/transaction/tx_ver2/tx_token_ver2.go:1259
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/transaction.DeserializeTransactionJSON
/Users/autonomous/projects/incognito-chain/transaction/transaction.go:329
github.com/incognitochain/incognito-chain/blockchain/types.(*ShardBody).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/blockchain/types/shardblock.go:440
github.com/incognitochain/incognito-chain/blockchain/types.(*ShardBlock).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/blockchain/types/shardblock.go:276
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:774
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.(*ProposeBlockInfo).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/block_info.go:129
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.InitReceiveBlockByHash
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:159
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.newActorV2WithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:105
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.NewActorV2WithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:70
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.NewActorWithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_base.go:50

2023-04-30 17:15:16.793 tx_token_ver2.go:1261 [ERR] Transaction log: -9004: Set bytes payment proof error edwards25519: invalid point encoding
Set bytes payment proof error
github.com/incognitochain/incognito-chain/privacy/errorhandler.NewPrivacyErr
/Users/autonomous/projects/incognito-chain/privacy/errorhandler/error.go:98
github.com/incognitochain/incognito-chain/privacy/privacy_v2.(*PaymentProofV2).SetBytes
/Users/autonomous/projects/incognito-chain/privacy/privacy_v2/payment_v2.go:212
github.com/incognitochain/incognito-chain/privacy/privacy_v2.(*PaymentProofV2).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/privacy/privacy_v2/payment_v2.go:127
encoding/json.(*decodeState).literalStore
/usr/local/go/src/encoding/json/decode.go:871
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:387
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:774
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/transaction/tx_ver2.(*TxToken).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/transaction/tx_ver2/tx_token_ver2.go:1259
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/transaction.DeserializeTransactionJSON
/Users/autonomous/projects/incognito-chain/transaction/transaction.go:329
github.com/incognitochain/incognito-chain/blockchain/types.(*ShardBody).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/blockchain/types/shardblock.go:440
github.com/incognitochain/incognito-chain/blockchain/types.(*ShardBlock).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/blockchain/types/shardblock.go:276
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:774
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.(*ProposeBlockInfo).UnmarshalJSON
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/block_info.go:129
encoding/json.(*decodeState).object
/usr/local/go/src/encoding/json/decode.go:612
encoding/json.(*decodeState).value
/usr/local/go/src/encoding/json/decode.go:373
encoding/json.(*decodeState).unmarshal
/usr/local/go/src/encoding/json/decode.go:180
encoding/json.Unmarshal
/usr/local/go/src/encoding/json/decode.go:107
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.InitReceiveBlockByHash
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:159
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.newActorV2WithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:105
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.NewActorV2WithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_v2.go:70
github.com/incognitochain/incognito-chain/consensus_v2/blsbft.NewActorWithValue
/Users/autonomous/projects/incognito-chain/consensus_v2/blsbft/actor_base.go:50

[Jared]

Hello @bjeung,

What Docker tag are you running? You can check with docker ps.

How much Disk space and RAM are free?

[bjeung]

Docker tag 20230429_1
43GB disk space free

Ram
total used free shared buff/cache available
Mem: 15929 1578 157 0 14192 14029
Swap: 975 8 967
Total: 16905 1587 1125

16 Cores, 16GB Memory with 3 vnodes

And this is a different box in a different location/internet connection than my other one that is having the ‘sync too fast’ issue.

[Jared]

That’s very strange. Please use the newest version of the bootstrap script version 2.0. This updated version of the bootstrap script will help to remove any issues that prevent containers from running properly.

Node Operator Bootstrapping Introduction & Guide

Changelog: Version 2.0 - 4/20/2023 - Automatic downloading of dependencies. Will only run once per server. Automatic removal of other directories/files that occasionally get corrupted. - bnbrelayingv3, btcrelayingv8, coins, consensus, incognito, listenerpeer.json, mempool Version 1.5 - 6/9/2022 - Added fullnode support with option 55. Updated to the new bootstrap server. Optimized downloading of dependencies and will only check once per run. Bootstrap files will be updated daily to …

[bjeung]

I ran the new bootstrap script just to let it cleanup the files and that seemed to do the trick. It’s back syncing now so hopefully it doesn’t act up again next time it’s selected for committee

[Jared]

@bjeung

It’s back syncing now so hopefully it doesn’t act up again next time it’s selected for committee

What disk do your nodes write to?

[bjeung]

For this box it’s a SSD backed VM.

[Jared]

Cloud hosted or hosted on your own hardware? If cloud hosted, who is the host?

[bjeung]

This is on my own hardware.

Here are the speeds with the nodes running

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.16151 s, 340 MB/s

[Jared]

Hmmm, that seems to be fine.

I would suggest removing the container and then relaunching it.

sudo docker stop inc_mainnet_#

sudo docker rm inc_mainnet_#

sudo docker run --memory=2G --restart=unless-stopped --net inc_net -p ####:#### -p ####:#### -e NODE_PORT=#### -e RPC_PORT=#### -e BOOTNODE_IP=mainnet-bootnode.incognito.org:9330 -e GETH_NAME=https://mainnet.infura.io/v3/#### -e TESTNET=false -e MININGKEY=#### -v /home/incognito/node_data_#:/data -d --name inc_mainnet_# incognitochain/incognito-mainnet:20230429_1

Change the # in each of those commands. I limit my containers to 2GB of RAM and set them to restart unless I tell them to stop. Feel free to remove those 2 portions of the docker command.

Hopefully you won’t have any issues after that.

It wouldn’t hurt to bootstrap the nodes again in case there is some kind of file corruption:

Node Operator Bootstrapping Introduction & Guide

Changelog: Version 2.0 - 4/20/2023 - Automatic downloading of dependencies. Will only run once per server. Automatic removal of other directories/files that occasionally get corrupted. - bnbrelayingv3, btcrelayingv8, coins, consensus, incognito, listenerpeer.json, mempool Version 1.5 - 6/9/2022 - Added fullnode support with option 55. Updated to the new bootstrap server. Optimized downloading of dependencies and will only check once per run. Bootstrap files will be updated daily to …