What does status “offline” and role “waiting” mean?
Both Nito and the App say that I am waiting to be selected and I have a green circle.
Is the Node working properly or not?
How to use the Node monitor
1 Like
Doesn’t look healthy.
‘Waiting’ means that the node has been staked and is waiting to be selected for validation (flow is: waiting -> pending -> earning).
‘Offline’ means that other incognito nodes can’t do RPCs to your node, so make sure your node is actually running and the RPC port is open to the internet. It shows green in your app because the IP is 192.168… which is a local one. You can use https://www.yougetsignal.com/tools/open-ports/ with the node public IP (not 192.168…) and RPC port to check if its open to the internet.
3 Likes
@17Strife also make sure your using the new “validator public key” not the old public key that nito uses. Its confusing that they named it that, but they are two completely different keys. All mine were showing offline until I realized my mistake.
2 Likes
@adrian, I used your link and I just hit check with the info that was auto-generated, it said “port closed”. Then, checked the “Power” tab a couple minutes later and…
The link still shows my port as closed, but I think I’m good to go based on the photo. Thanks!
Yes this looks a lot better. Seems it was offline for a little bit and is now syncing with the network to get caught up again. Keep an eye on what’s going on with it but you should be good to go now.
Beacon height is currently at 1165889. As of my post.
3 Likes
Is this good? anything to do?
shows a few rows of info as not syncing. anything to be concerned about?
1 Like
Your node is not selected in a committee yet, so it’s only syncing beacon chain now. Nothing to be concerned.
1 Like
Validator public key is highlighted in image with red box. That is the proper key.
1 Like
Is there information on the next steps when issued are found?
- If your status is shard stall should you reboot the node? Reboot the docker instance?
- If your status is beacon stall should you stop everything and repull the image?
3 Likes
Is there a way to get validator public key from a public key or bls key?
Validator Public key is the BLS Mining key. Not sure why the terminology changed. It seems unnecessarily confusing for users who are more familiar with the RPC calls.
1 Like
Using the Incognito Monitor and I have two nodes (one pnode and one vnode both behind the same router on my network). My pnode has been earning and has been running along smoothly. But for last epochs 3346, 3347, 3348, the monitor indicates 0% vote count though it otherwise looks to be functioning with status:online, role: waiting, sync state: latest (syncing with beacon fine) and Ensi and Nito are reporting node is waiting. Prior to epoch 3346, I had a vote count of 100. Is this an issue at my pnode and what should I do at this point since all my other diagnostic checks appear to indicate my pnode is fine?
My vnode is new and has not earned yet and has completed full sync 2 days ago. The incognito monitor indicates status: online, role:waiting, sync state: latest and vote stats (%) with a dash (no data, not even a 0 or epoch count). Nito and Ensi report this vnode is waiting and looks fine. Ports are checked and open and the docker data looks good and it is reporting correct latest beacon height. So, why is the monitor not reporting VOTE STATS at all? Is Incognitor monitor working or is something wrong with my node ? I can’t tell since everything else looks good.
I have 0 vote count for pnode (for last three epochs) and No vote count for vnode, but all other indictors look fine.
What should I test or check now on my vnode? Or should I assume issue with monitor? Is there an announcement or thread I have missed discussing this?
2 Likes
Hi @0xkumi…Question… @sid asked about this and I was wondering if any progress had been accomplished on this matter…
- Every time a member of the dev team makes an update to the Node Monitor or if I refresh my page, it looks like it loses all saved Validator Public Keys, and it requires me to add it it all again. This is very inconvenient and annoying, it would greatly help if you could save this in a cookie or some browser based session.
Look forward to hearing from you…thank you… 
1 Like
Did this ever get solved? Since the update to 1.0.6, I’ve noticed some weird issues with Monitor. My node shows as offline via Monitor, but in the app, it shows online and waiting. During the time Monitor has shown me as offline, I’ve validated 2 blocks and received rewards. For the last few epochs that I’ve validated, I show a 0 vote count.
The only metric that matters, or rather that WILL matter going forward, is vote count.
Let’s back up a moment. The current state of community Nodes (Nodes other than the 176 Incognito-run Nodes) is the temporary solution to a decision back in 2019. That temporary solution was only supposed to last ~3 months. Briefly – while the network was bootstrapping from the genesis block, the Incognito team operated 176 Nodes that ensured network consensus across the 8 shards, while community Nodes came online in increasing numbers. These Nodes represent 2/3rds of the voting power, which is enough to achieve consensus for any transaction. Community Nodes fill the other 1/3rd of voting power. The intent was (and is) to ensure that unstable community resources (be it server specs, network infrastructure, the internet itself, etc) wouldn’t affect consensus during these early days. There is more to unpack, however I’m attempting to be brief here.
This means that for every validated block, all 350 of them per shard per epoch, any each shard’s committee will be awarded block rewards because consensus will always be achieved due to those team-run nodes. Each of the ten community Nodes per shard could be offline, have a flapping internet connection, run on VPS without enough resources to validate blocks, etc. Block rewards are awarded to the entire shard, then divided equally between the committee members. It is not apportioned out based on actual block voting (or lack thereof).
Thus right now, block rewards for community Nodes basically means your Node was successfully staked and indexed by the Incognito blockchain. That’s it. Your Node is not awarded block rewards because it was online during its assigned committee or that it voted during same. It is was awarded block rewards because it was successfully staked when you first brought it online and has been selected for committee. Participation in committee is not currently a factor.
The voting metric will be used by the soon-to-be-implemented slashing, which will begin removing Nodes that are not maintaining a vote percentage of at least 50%. A Node can only vote when both the beacon shard and the assigned committee shard are fully synced before committee participation. This is the reason once Nodes are selected for committee, they remain in a Pending role: they are syncing the assigned shard’s blockchain.
There are a number of reasons a Node may not be able to successfully vote.
- A Node may be powered off or fully disconnected from the internet.
- These nodes would have a 0 vote percentage.
- A Node may reside on a server than cannot validate blocks in the appropriate amount of time (~40 seconds).
- These nodes would have a zero to low vote percentage.
- A Node may be connected to the internet with an unstable connection that makes that Node unavailable at times.
- These nodes would have a vote percentage that would be less than 100%. It would be possible for one epoch to have a vote percentage above 50% and the next below 50%.
- A Node may not have fully synced the shard blockchain, despite being online with a solid internet connection.
- These nodes would have a vote percentage of zero, as they cannot vote until the shard blockchain is fully synced.
- Other reasons including but not limited to acts of war, deity, man and beast, etc.
Under the current “temporary” solution, these Nodes are each awarded block rewards when selected to committee. However under the forthcoming slashing rules, these Nodes would each be slashed – have their 1750 PRV stake forcibly unstaked, removing them from the large pool of waiting substitute candidates.
The first three reasons are easily within an operator’s control.
- Turn on your Node if it’s off.
- Bump up the host server specs if your vNode can’t verifying transactions in ~40 seconds. (pNodes shouldn’t have this issue)
- Change your internet provider or fix your network issues to improve network availability.
The team is currently working hard to resolve issues creating sync stalls between Nodes. Again there are several reasons for a stall. A Node server could have a failing or failed HD/SSD. The Node’s HD/SSD may be write locked due to running out of space. The internet connection may be flapping. There may be other issues with the software components (Docker, etc). Thus a sync issue may be up to the operator to resolve or it may be also be a more widespread network issue requiring intervention from the team.
The takeaway should be that a Node that has been earning block rewards is NOT guaranteed to earn block rewards once slashing is enabled. Pay attention the Voting metric on the Node Monitor. If it is less than 50% contact support. The more Node operators that take an active role in resolving sync stalls with support, the stronger the network will become.
7 Likes
Thank you for the reply. I will open up a new thread for this issue so it can be resolved. Thank you.
Hi @fiend138 , I will distinguish the difference between status on app and on node monitor web
- App : show online it mean the power is plugged and node online.
- Node monitor : status on there is sync chain data status, it’s relate docker file run.
There may be cases where the node is online but not syncing the data.
You’ve highlighted a section, what are you trying to point out?