Vnode Issue - image stuck on 'restarting' again...

Hi @Support,

I recently changed a vnode’s server and when going through the setup the incognito image is stuck on ‘restarting’ status. I previously had this issue which was subsequently solved as per the below topic:

https://we.incognito.org/t/solved-vnode-issue-image-status-permanently-stuck-on-restarting/12607

Screenshot of the issue:
image

Container Output for your reference below. THANK YOU FOR YOUR ASSISTANCE!

[
  {
    "Id": "555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f",
    "Created": "2021-06-29T04:16:08.384862454Z",
    "Path": "/bin/bash",
    "Args": [
      "run_incognito.sh"
    ],
    "State": {
      "Status": "restarting",
      "Running": true,
      "Paused": false,
      "Restarting": true,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 3,
      "Error": "",
      "StartedAt": "2021-06-29T04:19:05.83705508Z",
      "FinishedAt": "2021-06-29T04:19:06.775030812Z"
    },
    "Image": "sha256:ef90cb023c9f3eec527acd8590f804e76e66c2be72450e75f3b8a18111ed6a83",
    "ResolvConfPath": "/var/lib/docker/containers/555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f/hostname",
    "HostsPath": "/var/lib/docker/containers/555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f/hosts",
    "LogPath": "/var/lib/docker/containers/555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f/555a2be5fade54351c20b2ed7daf085270dfac3ad84b5aa81d4405c4f9ef0c6f-json.log",
    "Name": "/inc_mainnet",
    "RestartCount": 12,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "docker-default",
    "ExecIDs": null,
    "HostConfig": {
      "Binds": [
        "/root/data:/data"
      ],
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "json-file",
        "Config": {}
      },
      "NetworkMode": "inc_net",
      "PortBindings": {
        "9334/tcp": [
          {
            "HostIp": "",
            "HostPort": "9334"
          }
        ],
        "9433/tcp": [
          {
            "HostIp": "",
            "HostPort": "9433"
          }
        ]
      },
      "RestartPolicy": {
        "Name": "always",
        "MaximumRetryCount": 0
      },
      "AutoRemove": false,
      "VolumeDriver": "",
      "VolumesFrom": null,
      "CapAdd": null,
      "CapDrop": null,
      "CgroupnsMode": "host",
      "Dns": [],
      "DnsOptions": [],
      "DnsSearch": [],
      "ExtraHosts": null,
      "GroupAdd": null,
      "IpcMode": "private",
      "Cgroup": "",
      "Links": null,
      "OomScoreAdj": 0,
      "PidMode": "",
      "Privileged": false,
      "PublishAllPorts": false,
      "ReadonlyRootfs": false,
      "SecurityOpt": null,
      "UTSMode": "",
      "UsernsMode": "",
      "ShmSize": 67108864,
      "Runtime": "runc",
      "ConsoleSize": [
        0,
        0
      ],
      "Isolation": "",
      "CpuShares": 0,
      "Memory": 0,
      "NanoCpus": 0,
      "CgroupParent": "",
      "BlkioWeight": 0,
      "BlkioWeightDevice": [],
      "BlkioDeviceReadBps": null,
      "BlkioDeviceWriteBps": null,
      "BlkioDeviceReadIOps": null,
      "BlkioDeviceWriteIOps": null,
      "CpuPeriod": 0,
      "CpuQuota": 0,
      "CpuRealtimePeriod": 0,
      "CpuRealtimeRuntime": 0,
      "CpusetCpus": "",
      "CpusetMems": "",
      "Devices": [],
      "DeviceCgroupRules": null,
      "DeviceRequests": null,
      "KernelMemory": 0,
      "KernelMemoryTCP": 0,
      "MemoryReservation": 0,
      "MemorySwap": 0,
      "MemorySwappiness": null,
      "OomKillDisable": false,
      "PidsLimit": null,
      "Ulimits": null,
      "CpuCount": 0,
      "CpuPercent": 0,
      "IOMaximumIOps": 0,
      "IOMaximumBandwidth": 0,
      "MaskedPaths": [
        "/proc/asound",
        "/proc/acpi",
        "/proc/kcore",
        "/proc/keys",
        "/proc/latency_stats",
        "/proc/timer_list",
        "/proc/timer_stats",
        "/proc/sched_debug",
        "/proc/scsi",
        "/sys/firmware"
      ],
      "ReadonlyPaths": [
        "/proc/bus",
        "/proc/fs",
        "/proc/irq",
        "/proc/sys",
        "/proc/sysrq-trigger"
      ]
    },
    "GraphDriver": {
      "Data": {
        "LowerDir": "/var/lib/docker/overlay2/2fbfcd6c3c0bd96e1e6dfff4a47a73581b12664f84d03642b8b41d3eed3a26a4-init/diff:/var/lib/docker/overlay2/277210d9347b68d6b83b84103ff881f0774cd92b75585f3b2fc0f3a765459922/diff:/var/lib/docker/overlay2/d0c39533244b53fb8b627b41c59c570e80ea9fb2b2b99698d8b5bdfed27f1556/diff:/var/lib/docker/overlay2/23037f1ae23c8ae65396cf62c43574e5484399b6d6bac9e0be130152ded44047/diff:/var/lib/docker/overlay2/a3b117a415f0882c32e36f5840e853b874a1cb5f2c777649c55706549ec0e8a0/diff:/var/lib/docker/overlay2/a1710695b1b12256f34e02786a771cd636b4b5d7e5219e74831c038f7a58bd28/diff:/var/lib/docker/overlay2/c3fd14bc91e7630f1b85ec9149568b26da31faf0fb193c56e0e522651074c7bd/diff:/var/lib/docker/overlay2/408f60253ea0fb76db4be02439988086d20ab35262d159431265a754230c18b3/diff:/var/lib/docker/overlay2/5cbd1df632cc516915995f2ff981f2e4ee88105b5c56ccc9aa37ea5263396a27/diff:/var/lib/docker/overlay2/1ca947405ade74124586e2644ae0e7e65cde65fba6cfdaf12180775b07232c96/diff:/var/lib/docker/overlay2/3245439feca1c11290387e23ecb6f2b85c8406b70ce0d6ccde3598b87bece7e6/diff:/var/lib/docker/overlay2/fdb2d862ebb3fb1231b3cda9960a60575935f479e7e5e3dfb22932221521c44b/diff:/var/lib/docker/overlay2/bbaade5ba2d9039a0daa9b212fa57fa7d97eb720ec6eaab995baffde29995181/diff:/var/lib/docker/overlay2/96bc1df148006bfbe2d6d228a9f83fba298d41db517fa36651bf57f104f1d7b2/diff:/var/lib/docker/overlay2/df10e4853d168898979f89b119a6f7045976f620223d633c359d14ca00062054/diff:/var/lib/docker/overlay2/70f37eee3ddec931a1bcf2963b7c87ad68491e667b474994fc6fef5af828c5da/diff:/var/lib/docker/overlay2/ae60ec77c74640814e501a8959c6846d03176ae2af3c552dc654c3b5e8d9fa2a/diff:/var/lib/docker/overlay2/b94b92c30bd495506af9fe433c9c09adf23e9539540df2884bb423738676889b/diff:/var/lib/docker/overlay2/c75d568bea893a74a398867c2faa8c98f67140714c854dcc923d44a089287b47/diff",
        "MergedDir": "/var/lib/docker/overlay2/2fbfcd6c3c0bd96e1e6dfff4a47a73581b12664f84d03642b8b41d3eed3a26a4/merged",
        "UpperDir": "/var/lib/docker/overlay2/2fbfcd6c3c0bd96e1e6dfff4a47a73581b12664f84d03642b8b41d3eed3a26a4/diff",
        "WorkDir": "/var/lib/docker/overlay2/2fbfcd6c3c0bd96e1e6dfff4a47a73581b12664f84d03642b8b41d3eed3a26a4/work"
      },
      "Name": "overlay2"
    },
    "Mounts": [
      {
        "Type": "bind",
        "Source": "/root/data",
        "Destination": "/data",
        "Mode": "",
        "RW": true,
        "Propagation": "rprivate"
      }
    ],
    "Config": {
      "Hostname": "555a2be5fade",
      "Domainname": "",
      "User": "",
      "AttachStdin": false,
      "AttachStdout": false,
      "AttachStderr": false,
      "ExposedPorts": {
        "9334/tcp": {},
        "9433/tcp": {}
      },
      "Tty": false,
      "OpenStdin": false,
      "StdinOnce": false,
      "Env": [
        "GETH_PORT=",
        "MININGKEY=**VALIDATOR KEY**",
        "NODE_PORT=9433",
        "RPC_PORT=9334",
        "GETH_NAME=mainnet.infura.io/v3/**INFURA ACCOUNT**",
        "GETH_PROTOCOL=https",
        "BOOTNODE_IP=mainnet-bootnode.incognito.org:9330",
        "TESTNET=false",
        "LIMIT_FEE=1",
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      ],
      "Cmd": [
        "/bin/bash",
        "run_incognito.sh"
      ],
      "Image": "incognitochain/incognito-mainnet:20210626_1",
      "Volumes": null,
      "WorkingDir": "",
      "Entrypoint": null,
      "OnBuild": null,
      "Labels": {}
    },
    "NetworkSettings": {
      "Bridge": "",
      "SandboxID": "52f8de6275e0a68c2a06e517560edf9ac7d99f5290216d70f7a801ed4957070f",
      "HairpinMode": false,
      "LinkLocalIPv6Address": "",
      "LinkLocalIPv6PrefixLen": 0,
      "Ports": {},
      "SandboxKey": "/var/run/docker/netns/52f8de6275e0",
      "SecondaryIPAddresses": null,
      "SecondaryIPv6Addresses": null,
      "EndpointID": "",
      "Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "IPAddress": "",
      "IPPrefixLen": 0,
      "IPv6Gateway": "",
      "MacAddress": "",
      "Networks": {
        "inc_net": {
          "IPAMConfig": null,
          "Links": null,
          "Aliases": [
            "555a2be5fade"
          ],
          "NetworkID": "359f49d1240bab70874b7b6a82cd7e4b2f71b5f2504a744c25e8128252042f6f",
          "EndpointID": "",
          "Gateway": "",
          "IPAddress": "",
          "IPPrefixLen": 0,
          "IPv6Gateway": "",
          "GlobalIPv6Address": "",
          "GlobalIPv6PrefixLen": 0,
          "MacAddress": "",
          "DriverOpts": null
        }
      }
    }
  }
]

Normally on vnodes in a constant restart loop we delete the data directory reset the docker network and restart the node.
With the latest vnode code it usually works.

This happens, we understand, when the code changes when the node is in the process of processing records with the old code and restarts. The restart has old data format in the new code which causes it to crash and restart continually.

Hope that helps

1 Like

Hey @SPAddict25, can you please share with us the error.log file? The content of the file would show us the panic error that caused the crash.
Also, FYI the new code tag has been just released here, please try to run your node with the tag. Thanks!

1 Like

Here’s the error log

Node error log
panic: -1006: Load key error
checksum error
Load key error
github.com/incognitochain/incognito-chain/consensus_v2.NewConsensusError
/Users/autonomous/projects/incognito-chain/consensus_v2/error.go:55
github.com/incognitochain/incognito-chain/consensus_v2.GetMiningKeyFromPrivateSeed
/Users/autonomous/projects/incognito-chain/consensus_v2/key.go:26
github.com/incognitochain/incognito-chain/consensus_v2.(*Engine.Start
/Users/autonomous/projects/incognito-chain/consensus_v2/engine.go:284
main.Server.Start
/Users/autonomous/projects/incognito-chain/server.go:729
main.mainMaster
/Users/autonomous/projects/incognito-chain/incognito.go:193
main.main
/Users/autonomous/projects/incognito-chain/incognito.go:252
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

goroutine 1 [running]:
github.com/incognitochain/incognito-chain/consensus_v2.(*Engine.Start(0xc000445810, 0x0, 0x0)
/Users/autonomous/projects/incognito-chain/consensus_v2/engine.go:286 +0xa2b
main.Server.Start(0x0, 0x0, 0xc000ee2050, 0xb, 0x1, 0xc004e56000, 0xc0002952c0, 0xc000226f30, 0xc00009bdc0, 0xc00051f500, …)
/Users/autonomous/projects/incognito-chain/server.go:729 +0x313
main.mainMaster(0x0, 0x0, 0x0)
/Users/autonomous/projects/incognito-chain/incognito.go:193 +0xfbd
main.main()
/Users/autonomous/projects/incognito-chain/incognito.go:252 +0x241
root@vmi614920:~/data#