Introducing Slashing V2

Slashing is one of the most desired features of Incognito’s core protocol where it comes to help increase reliability and validity for the network.

How does slashing help the network?

The network will be able to detect inoperative and misbehaving validators and replace them with better validators. A node is called inoperative and gets slashed when it doesn’t contribute 50% of required votes in an epoch where “required votes” is defined as the average number of blocks produced in a particular shard. If a node is slashed, the 1750 staked PRV would be returned back to the node operator and they may re-stake at a later time. After slashing has been active for some time, this will increase the stability of the network as the network will utilize “healthy” validators.

A different thing from Slashing V1 is that rewards from slashed nodes will be distributed evenly to remaining validators in a committee. This will help increase fairness among validators and motivate node owners to maintain their nodes’ operation.

When will slashing be effective?

Currently, there are about 2,500 nodes operating the network, according to the latest statistics at the time of writing, about 40% of total nodes are inoperative, they may be offline, stalled, or just not contributed during committee duty for some reason. So if we trigger slashing immediately in Sept’s 1st week as planned, the network will lose many pending slots, and a number of validators assigned to each shard will become unbalanced that will cause insufficient syncing time for even “healthy” nodes so these nodes will get unexpectedly slashed.

Fixed nodes vNodes pNodes All nodes
Number of nodes staked to the network 183 1444 879 2506
Number of nodes is offline 0 272 455 727
Number of nodes is stalled 0 87 36 123
Number of nodes is syncing 0 132 99 231
Number of nodes is running old code version 0 4 108 112
Number of nodes may be slashed when entering a committee 0 450 557 1007

To be able to get rid of the problem, we’re aggressively pushing Staking Flow V3 to get it released sooner than the end of Oct as planned. For more information, in Staking Flow V3, we’re introducing a new phase in the staking cycle called Sync phase and once a node is assigned to a shard committee, it has to sync sufficiently required data of that shard in the Sync phase prior to entering the waiting list so a node will always be ready to contribute to the network when being in a committee.

For that reason, we would change the plan a bit, the Slashing will still be released and run on mainnet since Sept’s 1st week but only actually slash inoperative nodes from 4th week because Staking Flow V3 will also be ready for release a week after. This is a strict deadline and we will try our best to meet it after several delays.

What should community validators do to avoid getting slashed?

In order for nodes to avoid getting slashed, a node owner needs to keep an eye on nodes’ status and contact the Support team for assistance right away if they are not working well. Please don’t ignore it since your nodes won’t be rewarded anymore if they don’t contribute to the network.

For pNode, we’re going to send an email to all pNode owners to inform you about the Slashing so that you can be aware of the change and take necessary actions in time. We’ve also implemented a way to distinguish between a node unstaked proactively by a node owner and a slashed node so that we can help you re-stake for the latter case.

Thank you for your support!

21 Likes

Hi,
I sent a message to the Team about 4 weeks ago for this problem pNode’s showing “offline” . I have yet to receive a reply .
Disappointed pNode investor !

Hey @ForestCity,

Sorry you’ve had this experience. The dev team is extremely busy and might have missed your message.

I sent you a PM. I’m sure we can figure out how to get your node running again.

Is there a care and feeding guide for vNodes? Or when we encounter stalls, we should reach out to support each time?

If a vNode is syncing the shard and not at latest will it get to vote?

My understanding is with the new script using Infura there should not be stalling anymore or at least very reduced.

There will be a sync phase and if your node does not sync within this window your node will be slashed. will wait in this phase until it is brought to latest.

Not quite. With the new phase and rules, a node remains in the Sync phase until it is fully synced. This is specifically meant to keep nodes from being slashed due to slow syncs. A slow sync will not slash a node because it can’t enter committee under the new rules, unlike the current rules. This ensures only nodes with up-to-date shard date can participate in committee. Thus slashing will only be based only on the voting performance metric.

3 Likes

Ah, good call! Some how I missed that part (must have been tired when I read the post).

I agree it’s getting better, but I’m only voting about 50% of the time right now. Still not a playbook for what to do.

Under the node monitor how often are your nodes showing up as latest or syncing?

Sounds like sync speed (either computational, harddrive or internet speed) is your issue. If your nodes are not syncing fast enough your vote will be low (at least how I understand it). Does your node setup meet minimum specs?

2 Likes

Hey @Jared, I’m running a vNode. Is the “node monitor” you mention above an application or is it a website?

I’d like to learn how I can check whether my node is likely to be slashed or not.

Any help on how I check on my vNode would be most appreciated. Thank you.

You want to check monitor.incognito.org

There you should insert a node name and the validator public key (not the validator key). You’ll notice the column to the far right says “Vote Stats (%)” ideally you want this to be 100%.

You can click on each node to see more information about beacon and shard sync as well as previous vote stat.

I’ve also been having this problem. I’ve had my pnode i bought from you guys for about a year and my app always shows it offline but when i use 3rd party bots to monitor my pnode it shows its online. No one has ever offered to help me other than to tell me to unplug it and plug it back in. Spending $400 on something that always tells me its not working is pretty annoying but not as annoying as the “solution” i was given.

Hey @Jared, thanks for sharing the website.

I have checked my vNode and the Ports 9334 and 9433 are open to my server and when I look at my “docker stats” everything looks fine (see screenshot below)

vNode-docker-stats

However, when I log on to the website you mentioned: monitor.incognito.org and enter in my “Validator public key” I get the following “Offline” message:

vNode-offline-

Can someone please help me determine whether my vNode is offline or not?

Thanks in advance.

1 Like

Hello @Linnovations,

I’m sure we can get your vNode working 100% correctly. In an effort to not derail this thread for slashing I have sent you a PM to continue troubleshooting.

1 Like

Hello @keyglock,

I have also sent you a PM to help troubleshoot your issue. I’m confident we can get your node working and operational.

1 Like

What is the minimum spec?

In the past it was said to be:

4 CPUs, 4 GB Ram and 50 GB SSD

However, I would recommend a much higher storage size due to the expanding size of the blockchain.

1 Like

I think someone said the blockchain was approaching 150 gb.

Eventually (my guess is ~6 months after the staking, unless you sync the node from scratch) all vNodes will converge to the size of the full node data which is currently around 150gb.

In fact, there is a solution to this but after Sync pool is enabled, that solution may cause you to earn the rewards less frequently.

The Solution: Deleting all shards’ data except the syncing one :slight_smile: In this case, each time a validator enters to Pending role, it will sync the corresponding shard from scratch.