Shard stall

[Mike_Wagner]

@Silvercap718

What are my options on dealing with the stalled status

You could try deleting the shard data on the pNode. You’ll need to know the ID of your pNode (it’s the text of QR code sticker on the bottom of the pNode) and the IP address of the pNode on your home network.

You would then edit the following URL where noted:

http://<YOUR NODE IP>:5000/restart-node?delete-data=1&qrcode=<QR CODE OF NODE>

This will remove all synced data and your pNode will sync from scratch. This is highly dependent on the speed of your internet link, but it will take a while for the pNode to fully sync the beacon chain. My pNode that is syncing from scratch right now, looks to complete the beacon sync in probably another 6 hours or so. Then it will need another 8? 9? 10? hours to (hopefully) fully sync the chain for shard 0 (that’s the shard it’s currently Pending for).

As I noted above, your pNode will be active in committee during Epochs 3476 and 3477. The current epoch is 3469. That’s a difference of 6 epochs or ~24 hours. That should be just enough time for your pNode to fully sync both the beacon chain and assigned shard chain if you delete the chain info right now. Even if it doesn’t fully sync by the time it enters committee, you’ll still be awarded block rewards. And if it completes the sync during the first epoch, you should see successful votes tallied in the second epoch.

[Mike_Wagner]

@fitz_fiat

Next question is if the node is in pending but stalled will the node fix the stall issue and go into committee or just stay stalled?

Right now, stalled nodes still enter committee. They just do not tally votes in the Node Monitor tool. When slashing is enabled, this will be important because Nodes that don’t maintain a vote percentage of 50% during committee will be slashed. But for now, stalled/not-stalled do not affect the ability of Node to enter committee. I’ve personally had several stalled Nodes in committee without issue.

@fitz_fiat

Which then brings up the question, if it’s stalled in pending state, if the vnode is stopped the data cleared and restarted will that cause it to fall out of pending?

The node wouldn’t fall out of pending. The pNode I used for this post has been in Pending since early this morning. I deleted the shard chain info this afternoon and started a new sync from scratch. It’s still syncing and still in Pending.

[Thriftinkid]

Hey @duc as @fitz_fiat said, I now have 4 nodes pending that are all stalling. I’m still a bit confused as to why resetting them would fix the stalling issue if he reset them all after you guys pushed the update. Did you push another one after that? They should all be running on the newest code, and there should be no issue then.

Shard 6 909062 16 hours ago (stalling)

Shard 6 909062 11 hours ago (stalling)

Shard 6 1003289 stalling

Shard 2 433077 a day ago (stalling)

[Mike_Wagner]

@Mike_Wagner

Anyone have a Node in Shard 0 also stuck on block 169583 ? Asking for a friend.

@duc

The shard’s stall is because it ran the old code and accidentally produced data that was not compatible with Incognito chain’s data. That’s why I recommended node owners who have sync issues to clear up data and then re-sync with the latest code.

Well … some 20 hours later and my pNode has (again) fully synced the beacon chain and has (again) stalled on block 169583 in Shard 0.

This shard stall is not due to a mix of old shard data and the new flow code, at least in my case. Something else is preventing the sync or block insert at that height for Shard 0’s chain.

[JG20]

I have multiple nodes stalled on block 169583 in Shard 0. All data was recently refreshed.

[JG20]

Also, on the Infura dashboard, it still says zero requests but I have the key attached to multiple nodes. Any ideas?

[Mike_Wagner]

@Thriftinkid

Shard 2 433077 a day ago (stalling)

Check your Node. My stall … unstalled for Shard 2 sometime in the last few hours. I did NOT delete chain info on this Node either.

[doc]

I want to get see if I can get clarification, just to make sure I am understanding and I didn’t miss this comment on one of the many post about the updates for slashing and pNodes.

My current understanding is that slashing goes live in June, and if your node (p or v) doesn’t meaningfully contribute to committee it will get slashed and unstaked.

Does this mean, that pNodes will get unstaked and not have the ability to get the funded stake back?

I definitely understand the need for slashing, but I am a bit concerned that with the random selection process, there really isn’t enough time to test (and retest) if a stall/sync issue is fixed/resolved/still exist on pNodes… especially if you have one of those 30+ day dry spells waiting for selection…

This isnt a huge deal for vNodes, as you can just restake… and probably not a huge deal for pNodes if you can get the funded stake back, but I just wanted to get clarity on this… as I havent found the specific answer on this… which I could have definitely overlooked, through the multiple post I haver read about getting pNodes set up with new firmware, node monitor, etc.

[JG20]

Great question. I have been wondering this myself.

[Tempestblack]

Thank you @doc for asking the question…I have 3 pnodes myself all with funded stake at this time and well I wonder if they were to get slashed what will happen to the funded stake on them…will it be reinstated since I took no action to unstake them, to begin with?..

[Mike_Wagner]

Corey – great questions. I’ve been asking some of the same things for a while now, more so in the past few weeks. This is a topic you’ve been on since last fall.

@doc

My current understanding is that slashing goes live in June,

@hungngo

We will be based on the community nodes’ performance to decide when Slashing is enabled. That said, the Slashing release date is not 100% finalized.

June was clarified to be a tentative date; not a hard deadline thankfully. However there has not been any further timeline update since, but I do hope the issues arising from this weekend’s changes will help push the date back until they are fully resolved.

@doc

Does this mean, that pNodes will get unstaked and not have the ability to get the funded stake back?

That’s more or less the reality that, IMHO, has not truly sunk in for many pNode operators.

@duc

Stop supporting a pNode to go back to funded staking if it DID unstake. You can still stake using your own funds at any time. Let me give clarity to the decision, for every bulk of pNode sold, we prepared an amount of PRV as a rented funds. But once a pNode owner unstakes his/her node and restake with his/her own funds in order to take 100% earnings, it’s supposed that he/she did not need rented funds anymore so the funds would be moved and used for other purposes. Please note that the rented funds program was about to support a pNode owners in the initial setup, how can he/she demand another round of this sort of support if he/she could stake with his/her own funds and earn 100% rewards?

The stated position appears to be that every pNode operator must have enough PRV or enough funds to buy the requisite PRV to privately stake their pNode.

Of course that’s certainly not the case for any solo pNode operators. I estimate it would take 4-6 pNodes earning since the Genesis block to produce ONE stake of 1750 PRV. The move over the weekend to a Pending phase that will eventually reach 1 week seriously compromises any kind of meaningful reward potential going forward.

With the new approach, it is quite conceivable that unlucky Nodes may go 2-4 months without earning. The protocol adjustment has swung wildly out into left field, far overcompensating for slow syncs. An adjustment to 1-2 days max is all that was needed.

[Tempestblack]

@Mike_Wagner so you are telling me that those of us whom have Funded pNodes at this time that were to suddenly be defunded due to slashing then would be unable to recover that funded status for the slashed pNode even though we had no doing in unstaking or unfunding the pNode to begin with and it happened due to slashing…am I reading you correctly on this then?

[doc]

Thanks for the reply, I do understand their policy change and the thought process around it… but this does feel a bit different. IMHO, it is different if I physically unstake my node, or am unable to keep my pNode is a proper environment to allow it to meaningfully contribute to network/committee… it is something completely different if my pNode gets slashed because we are working through bugs. Which I also completely understand happen and have to be worked through, but its different when on a testnet or no financial penalty is being felt while working through those bugs. Having a user now need to pony up 1750PRV (~$3,500 at time of writing), seems very different… and if a validator has to wait for a random selection process to verify their pNode is functioning properly… it seem pretty difficult to meaningfully test and validate.

Hopefully, as you suggested the timeline for June will not be hard and fast, and more consideration will be given on providing ample time to work through these issues.

[Tempestblack]

Once more may I state that the price of PRV has gone up since the genesis block was created…and well since the inception and creation of the pNodes…I mean 1750 in PRV a year ago was not what it would cost at today’s prices…so yea…I can say that staking a PRV back then was a much more easier and feasible thing to do as compared to today

[duc]

Hey guys, I do really understand your concern about a pNode may not participate in funded staking aid if it gets un-staked for some reasons. There are two cases of un-staking:

1. A pNode owner un-stakes physically and then re-stakes by his/her own funds in order to earn 100% of rewards. We would not support re-staking with rented funds for this case if he/she wants to go back as noticed earlier.

2. A pNode gets slashed. We’re working on a way to distinguish this from the (1). If a pNode falls into this case (a.k.a unstaked because of slashing), it can still re-stake with rented funds as usual.

[Mike_Wagner]

@duc

2. A pNode gets slashed. We’re working on a way to distinguish this from the (1). If a pNode falls into this case, it can still re-stake with rented funds as usual.

@duc Thank you for the much appreciated clarification!

[doc]

Thanks @duc ! That is very good to know.

[Tempestblack]

Apology for delayed response @duc…but indeed a very very grateful and thankful and enthusiastic response on my part to your post…

[Thriftinkid]

@Mike_Wagner

The move over the weekend to a Pending phase that will eventually reach 1 week seriously compromises any kind of meaningful reward potential going forward.

With the new approach, it is quite conceivable that unlucky Nodes may go 2-4 months without earning. The protocol adjustment has swung wildly out into left field, far overcompensating for slow syncs. An adjustment to 1-2 days max is all that was needed.

@Mike_Wagner, I’m looking for @duc to confirm this, but I think the move for more pending slots is with the idea of releasing all the slots. Currently I see anywhere between 128-152 nodes in pending slots on each shard. That is an incredibly long time to wait at the moment since they are only swapping out 5 at a time. When they release all the slots, they will be swapping out 32 at a time. Bringing the epoch pending time down to 4-5 epochs (less than a day) depending on how many nodes in front of you dont sync properly. Am I right here @duc?

Another weird thing I just encountered was I had a pending node that went into committee. During that, it stated 2 epochs before it went back into pending (instead of back to waiting). It just finished committee and indeed went back to pending. I’ve never seen that before.

[Mike_Wagner]

It is true the pending pool will need to be much larger to support all 32 slots as fixed slots are released. However I based my statement on this private chat with support earlier this month about fixing pNodes with Vote Stats of 0%: