How to use the Node monitor

The node monitor is officially live to prepare all validators for the upcoming slashing release. It’s integral for releasing the network because nodes must be healthy for the network to be stable.

For more details, a node will be slashed (a.k.a swapping out of a shard committee at the end of epoch) in case the node does not contribute sufficient required votes on blocks committed in an epoch. The required votes are 50% currently.

This is how to monitor your node and avoid slashing.

1. Access the Node Monitor

Navigate to https://monitor.incognito.org. You will see the following interface:


Figure 1 - The node monitor page

The tool allows you to check your nodes’ statuses (My nodes) as well as the whole network’s inactive nodes (Inactive nodes).

2. Add nodes to your list

Obtain a node’s validator key from the keychain tab in your Incognito app.

Note: the next release will include a more convenient way for you to obtain the Validator Public Key.


Figure 2 - Obtain Validator public key from Incognito app

Input node name (arbitrary), and associated validator key.

Then click ‘Check’. This will add the node to your list so you can monitor important information. To add another Node, just repeat the process.


Figure 3 - Add a node to monitor list

3. Review health of your nodes

Here’s what the information in each column means:

Name - the node name. This does not need to line up with the name of your node in the app.

Validator Public key - the key associated with this particular node.

Role:

  • Waiting: waiting to be selected into a shard
  • Pending Shard i: where i is from 0 to 7, being selected into shard i, waiting to get into a committee
  • Committee Shard i: where i is from 0 to 7, being in a shard committee and earning rewards
  • Not staked - the key is not yet staked

Status:

  • Online: syncing data
  • Offline: not syncing data and no votes (starting from the date the node was added to the monitor)
  • Unknown: running outdated code (i.e., an old docker tag) so the tool does not have appropriate data to show.

Sync State:

  • Node stall: syncing is disabled
  • Beacon syncing: syncing beacon blocks
  • Beacon stall: syncing beacon blocks but the node cannot insert the blocks. Issues may be different among nodes so please reach out to @Support for further assistance
  • Shard syncing - syncing shard blocks
  • Shard stall - syncing shard blocks but the node cannot insert the blocks
  • Latest: syncing the up-to-date beacon block. If the node is in a shard committee, it also shows the node is caught up with shard chain
  • Unknown: no data

Vote Stats: vote percentages of the two most recent epochs where the node was in a shard committee.

4. See historical activity

Just click on the node. There will be a pop-up detailing vote count (%), as well as earning history for each epoch. Please note that all data applies only from the date the Node was added to the monitor.


Figure 5 - Node details

5. See health of a node right on Incognito app

In addition to using the Node monitor’s web version, you can also see health as well as historical activity of your node right on Incognito app.

If you have any questions, please let us know in the comments below. @0xkumi is the responsible dev for this initiative, but feel free to reach out to any of us if you need assistance.

Looking forward to seeing an entire network of healthy nodes!

Inactive nodes

The Inactive nodes display all nodes which may get slashed mostly because of Offline status once the Slashing is enabled.


Figure 6 - Inactive nodes

Similarly, it’s possible to view a node details when clicking on a node from the table.


Figure 7 - Inactive node details

That’s it for now!

Last but not least, please start using the tool to see how your nodes operate and then take into account of getting issues resolved as soon as possible if any. Again, it’s surely our responsibility to help you out with these node issues.

We still keep working on the tool to add more insight to it such as node contribution statistics, average vote latency, histogram of vote counter and number of active/inactive nodes. Always happy to hear your feedback!

16 Likes

Each time you update the monitor software, the list of my nodes is lost. Could you please save them as a cookie or provide us with exporting/importing our node list in one click? Thanks.

These instructions say use the validator key, but the site shows use public key.

image

1 Like

Wow this tool is both great and depressing at the same time. Glad I can fix the nodes that need fixing. Also confirms that the one pnode I knew wasn’t working out of the box for the last 12 months really wasn’t working. I feel vindicated! I can understand now why the team didn’t want to release the fix nodes, probably too many of our nodes have only had the illusion of working.

1 Like

One of my pNodes shows up fine in the wallet app, but as not staked in the monitor.

The pNode is selfstaked since March 2020

The other one shows up fine.

The screenshots also guide the user to use the Public Validator Key from the Anon tab instead of correctly using the Public Validator Key from the node tab, as far as I can tell

@0xkumi Please clarify this as it is confusing.

Post and screenshots need updated as well.

The screen only has one account, which in theory would mean that is the account the node is attacheded to, which in turn means the screen is correct.

The text should mention “Get the validator key from the keychain your node is attached to” to make it clearer.

Could it be that pNodes that have been setup the “manual Ether cable” way, show the wrong status? That is just about the only difference between my pNodes.

Hey guys, we just updated the tutorial a bit, you can also see health as well as historical activity of your node right on Incognito app alongside the web version. (see section 5 above for more details)

2 Likes

I’m dig’n this node monitor, wish we had this early on. Got all my nodes Online and Waiting now…except for one that I’ll message support about. Well done.

Yup one has to admit…this Node monitor tool ROCKS!!!.. :100: :+1: :sunglasses:

1 Like

I found for every single pnode I own that they were offline and had contributed zero votes for every selection so I went to unstake. I think its a bug because as soon one was selected, it’s showing an online status and contributing perfectly. It seems vnodes do not have these same issues.

1 Like

Sorry I don’t understand what you are saying.

Good stuff! My Pnodes checked out.
At first I entered the PUBLIC KEY by mistake and got “ not online” message!
Reread the instructions and then entered the VALIDATOR PUBLIC KEY and bingo the proper info came up.
This is a great interface that will provide great feedback to the pnode owner.

1 Like

Make sure you use the VALIDATOR PUBLIC KEY. I entered the public key by mistake and it showed off line.

1 Like

That’s not the problem. The valiator public key has been used. I have a stack of pnodes, all showed up as offline with no historical votes.

Once the pnode got selected, the status for syncing and voting was all completely fine. My point is some people may think they have a pnode not working when in reality it does.

Noooo pnodes look like they ain’t working
Screenshot_20210424-140444_Brave

Oh wait they do work once selected…
Screenshot_20210424-140740_Brave

For all vnodes I haven’t had the same issue so it appears to only be for pnodes.

@0xkumi and @Support - I noticed 2 issues, one is probably already reported on this thread by another community member.

  1. Every time a member of the dev team makes an update to the Node Monitor or if I refresh my page, it looks like it loses all saved Validator Public Keys, and it requires me to add it it all again. This is very inconvenient and annoying, it would greatly help if you could save this in a cookie or some browser based session.

  2. Right now, there is no way to export the validator public key in bulk. If I have several vnodes, I have to manually copy each validator public key to a notepad or email, and then copy it over the the desktop browser if I am trying to look at all of it together on my computer. This is extremely annoying! Is there any way we could bulk copy the validator public keys just like the backup function works for the private keys under keychains?

Finally - is there any info @duc or the team can share on the plan and timing for implementing slashing? I know you plan on implementing dynamic committee next week (from the most recent newsletter), but are you still planning on implementing slashing in June or will that be sooner?

Thanks,
Sid

3 Likes

Hey there, please take a look at our timeline
Dynamic committee size is an initiative consisting staking flow v2 and slashing. Its development has been done, we plan to deploy it onto mainnet in 1 or 2 days, but not yet enabled.
Slashing will be enabled in June, in other words, we will have over 1 month left to work together on node issues if any.

8 Likes