Incognito chain’s code upgrade (tag: 20210824_1)

Hi folks,

We’ve just released a protocol upgrade for Bitcoin trustless non-custodial bridge, the latest code is published in 2 forms:

In order for your nodes to operate correctly, please update them to the latest code. If you run pNodes or set up your vNodes by following our instructions , they should pull the latest docker image automatically. In case you encounter any issues with the code update, feel free to contact @support for assistance.

Thanks!

6 Likes

I just rolled this update out to all of my nodes but now I am seeing they are all restarting. Is anyone seeing this issue?

Infinite restart loop on all nodes with the new update tag 20210824_1

Hey @Jared, my vnodes work flawlessly for +33 hours. FYI.

Crashing for me as well, segfault. cc @duc @Support

$ cat error.log 
2021/08/25 19:44:08 Using network param file for mainnet
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xc22444]

goroutine 254 [running]:
github.com/incognitochain/incognito-chain/dataaccessobject/statedb.(*ProcessedUnshieldRequestBatch).GetUnshieldRequests(...)
        /Users/autonomous/projects/incognito-chain/dataaccessobject/statedb/stateobject_portalv4_batchunshield.go:32
github.com/incognitochain/incognito-chain/portal/portalv4/portalprocess.(*PortalSubmitConfirmedTxProcessor).ProcessInsts(0x0, 0xc00923ad6c, 0x3, {0xc008ec4c80, 0x47a815, 0x40daf7}, 0xc009a89650, {0xc0002e8d50, 0x5, 0xc0002e8db0, ...}, ...)
        /Users/autonomous/projects/incognito-chain/portal/portalv4/portalprocess/unshieldbatch.go:772 +0x1a4
github.com/incognitochain/incognito-chain/portal/portalv4/portalprocess.ProcessPortalInstsV4(0xc0002e8d80, 0xc0002e8de0, {0xc0002e8d50, 0x5, 0xc0002e8db0, 0xc0002e8d80, 0xc0002e8de0, 0xc0002e8e10, 0xc0002e8e40, 0x2d, ...}, ...)
        /Users/autonomous/projects/incognito-chain/portal/portalv4/portalprocess/processingutils.go:180 +0x42d
github.com/incognitochain/incognito-chain/portal.ProcessPortalInsts({0x3, 0x0}, 0xc009b36600, 0xc00923a036, 0x3, {{{0x16c0f9f, 0x14}, {0x16b8ec0, 0xf}, {0x16b6ffc, ...}, ...}, ...}, ...)
        /Users/autonomous/projects/incognito-chain/portal/portalprocessing.go:111 +0x2f0
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).processPortalInstructions(0xc0000ec580, 0xc008eb9c50, 0xc009376a80)
        /Users/autonomous/projects/incognito-chain/blockchain/portalprocess.go:66 +0x245
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).processStoreBeaconBlock(0xc0000ec580, 0xc00002e380, 0xc0002bc700, 0xc009376a80, 0xc009844700)
        /Users/autonomous/projects/incognito-chain/blockchain/beaconprocess.go:871 +0xf66
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertBeaconBlock(0xc0000ec580, 0xc009376a80, 0x1)
        /Users/autonomous/projects/incognito-chain/blockchain/beaconprocess.go:185 +0xe89
github.com/incognitochain/incognito-chain/blockchain.(*BeaconChain).InsertBlock(0xc001098410, {0x1a8a890, 0xc009376a80}, 0xe0)
        /Users/autonomous/projects/incognito-chain/blockchain/beaconchain.go:210 +0x45
github.com/incognitochain/incognito-chain/syncker.InsertBatchBlock({0x7f9748b0f8e8, 0xc001098410}, {0xc007c4b200, 0xd, 0x10})
        /Users/autonomous/projects/incognito-chain/syncker/utils.go:112 +0x309
github.com/incognitochain/incognito-chain/syncker.(*BeaconSyncProcess).streamFromPeer(0xc001039d80, {0xc007829f20, 0x2e}, {0x61269d92, {0xc008136700, 0x40}, 0x15cece, 0x0})
        /Users/autonomous/projects/incognito-chain/syncker/beaconsyncprocess.go:365 +0x7f3
github.com/incognitochain/incognito-chain/syncker.(*BeaconSyncProcess).syncBeacon(0xc001039d80)
        /Users/autonomous/projects/incognito-chain/syncker/beaconsyncprocess.go:274 +0x15f
created by github.com/incognitochain/incognito-chain/syncker.NewBeaconSyncProcess
        /Users/autonomous/projects/incognito-chain/syncker/beaconsyncprocess.go:61 +0x247

I tried rolling back but just getting shard stall with 20210820_1.

Please confirm you’re running 20210824_1 and what your configuration is. What version and distro of Linux and how do you have docker setup?

Okay, we’re checking and get back to you guys asap.

1 Like

Hi @adrian,
I am investigating what the cause of the problem is. Help me check if you have any other log files. They are usually formatted as ipaddress-date.log. Thank you so much.

1 Like

DMed you the logs

1 Like

Hey @Jared,

I share my script. My VPS is Ubuntu 18.04.

#!/bin/sh bash

run()
{
  # ATTENTION: CHANGE THESE
  validator_key=VALIDATOR_KEY
  ####################################
  bootnode="mainnet-bootnode.incognito.org:9330"
  latest_tag=$1
  current_tag=$2

  node_index=1
  data_dir="data${node_index}"
  node_port=$((9432 + ${node_index}))
  rpc_port=$((9333 + ${node_index}))
  container_name="inc_mainnet${node_index}"
  script_name="[r]un${node_index}.sh"
  infura_api_key="API_KEY"
  backup_log=0

  docker -v || bash -c "wget -qO- https://get.docker.com/ | sh"

  # Remove old container 
  docker rm -f ${container_name}
  if [ "$current_tag" != "" ]
  then
    docker image rm -f incognitochain/incognito-mainnet:${current_tag}
  fi

  docker pull incognitochain/incognito-mainnet:${latest_tag}
  docker network create --driver bridge inc_net || true

  docker run --restart=always --net inc_net -p $node_port:$node_port -p $rpc_port:$rpc_port -e NODE_PORT=$node_port -e RPC_PORT=$rpc_port -e BOOTNODE_IP=$bootnode -e GETH_NAME=mainnet.infura.io/v3/${infura_api_key} -e GETH_PROTOCOL=https$

  if [ $backup_log -eq 1 ]; then
    mv $data_dir/log.txt $data_dir/log_$(date "+%Y%m%d_%H%M%S").txt
    mv $data_dir/error_log.txt $data_dir/error_log_$(date "+%Y%m%d_%H%M%S").txt
  fi
}

# kill existing run.sh processes
ps aux | grep $(basename $0) | awk '{ print $2}' | grep -v "^$$\$" | xargs kill -9

current_latest_tag=""
while [ 1 = 1 ]
do
  tags=`curl -X GET https://hub.docker.com/v1/repositories/incognitochain/incognito-mainnet/tags  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}' | sed -e 's/\n/;/g'`

  sorted_tags=($(echo ${tags[*]}| tr " " "\n" | sort -rn))
  latest_tag=${sorted_tags[0]}

  if [ "$current_latest_tag" != "$latest_tag" ]
  then
    run $latest_tag $current_latest_tag
    current_latest_tag=$latest_tag
  fi

  sleep 3600s

done &
2 Likes

I’ll shoot you a PM with my logs as well. Only error I’m seeing is the following:

“Cannot publish message peerstate”

Hey @Jared, @adrian, just wanted to have a quick confirm, dont your nodes pull new code (or new docker image) automatically once having a new image?

After investigating the logs, we guess the issue happened because you updated your nodes with new code after the point Bitcoin trustless bridge effected, there were some new shielding & unshielding requests along with their state that not existed in your nodes’ data due to late update, so these nodes couldn’t process for consequent requests because these requests need data of previous ones.

This is very bad and I hope the other nodes with an auto-update setup don’t face the issue. Really apologize for the issue. So I afraid you have to wipe these nodes’ data and re-sync from scratch or let me clone and zip beacon data from a fullnode, share with you two and then you can replace the damaged ones with ours to see if it can help.

1 Like

Hello @duc,

I use the following script to run my nodes:

I don’t recall ever having the script auto pull new updates. It has always been a manual thing that I check every few days.

Hi @Jared, as mentioned in the original post of the topic, you can follow the instruction to set up auto-update process for your nodes.
But let us fix the issue first, wait for me a bit, I’m cloning beacon’s data and sharing you then.

Unfortunately this script / install method doesn’t work for me as I have a really beefy VPS that I use to host several nodes.

Okay understood. I’ll wait for your files. Thanks a bunch.

In order for this script to automatically update you need to leave it running. It’s not flawless but it generally works.

1 Like