Fullnode stopped syncing

Hey @Support,

My fullnode stopped syncing for 2-3 hours. It sometimes syncs some blocks and then stops. I think there is a problem around bootnode. Could you fix it?

Mostly I’m getting this:

2023-04-02 18:47:16.300 outcoinIndexer.go:900 [INF] Transaction log: CoinIndexer is full or no OTA key is found in queue, numWorking 0, queueSize 0
2023-04-02 18:47:16.484 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:16.484 blockrequester.go:359 [INF] Peerv2 log: Dial to new HW QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:17.676 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:17.676 blockrequester.go:359 [INF] Peerv2 log: Dial to new HW QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:21.485 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:21.485 blockrequester.go:359 [INF] Peerv2 log: Dial to new HW QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:22.676 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:22.677 blockrequester.go:359 [INF] Peerv2 log: Dial to new HW QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:26.301 outcoinIndexer.go:900 [INF] Transaction log: CoinIndexer is full or no OTA key is found in queue, numWorking 0, queueSize 0
2023-04-02 18:47:26.485 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:26.485 blockrequester.go:381 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:27.677 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop
2023-04-02 18:47:27.677 blockrequester.go:359 [INF] Peerv2 log: Dial to new HW QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop

Some excerpts from the log:

2023-04-02 18:35:04.451 submanager.go:264 [INF] Peerv2 log: Process message txprivacytok and topic [{txprivacytok-0-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-1-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-2-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-3-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-4-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-5-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-6-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok-7-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub PUB} {txprivacytok–QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-0-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-1-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-2-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-3-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-4-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-5-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-6-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB} {txprivacytok-7-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodesub SUB}]
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-0-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-1-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-2-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-3-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-4-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-5-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-6-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub
2023-04-02 18:35:04.451 submanager.go:273 [INF] Peerv2 log: Skip subscribing to PUB topic: txprivacytok-7-QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop-nodepub

2023-04-02 18:35:30.452 blockrequester.go:370 [ERR] Peerv2 log: Could not dial to highway grpc server: context deadline exceeded QmSekgKqNtgvA9J9TVUrHj5q8vPh6KbJi7M1D8BakZ9fop

2023-04-02 18:35:41.151 addrkeeper.go:451 [ERR] Peerv2 log: Can not get HW from RPCUrl mainnet-bootnode.incognito.org:9330, error Connect to discover peer mainnet-bootnode.incognito.org:9330 return error dial tcp: lookup mainnet-bootnode.incognito.org on 127.0.0.11:53: read udp 127.0.0.1:50499->127.0.0.11:53: i/o timeout:

Syncker: create channel fail
2023-04-02 18:38:21.765 connmanager.go:445 [INF] Peerv2 log: [SyncShard] from 3159717 to 3159915 fromShard 5
2023-04-02 18:38:21.765 connmanager.go:504 [INF] Peerv2 log: [stream] Request Block type BlkShard from peer from cID 5, [3159717 3159915]
2023-04-02 18:38:21.765 blockrequester.go:251 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [3159717…3159915] len 2, from 5 to 5, uuid = a41f6193-5863-4fd2-97d9-d9a1cdd71191
2023-04-02 18:38:21.766 connmanager.go:508 [ERR] Peerv2 log: [stream] requester not ready
Syncker: create channel fail
2023-04-02 18:38:21.765 connmanager.go:504 [INF] Peerv2 log: [stream] Request Block type BlkBc from peer from cID 255, [3185954 3186154]
2023-04-02 18:38:21.765 blockrequester.go:251 [INF] Peerv2 log: [stream] Requesting stream block type BlkShard, spec false, height [3162079…3162278] len 2, from 6 to 6, uuid = a82f6131-413d-4466-80bc-c68087ac521e
2023-04-02 18:38:21.766 connmanager.go:508 [ERR] Peerv2 log: [stream] requester not ready
Syncker: create channel fail

Did you try to reboot. It shows that your node connection has problem.

Yes, I restarted the node, not the computer. You may be right about the connection. Let me watch it for some time.