For reward, the monitor just show what it see when beacon chain distribute reward to a shard. Shard5 recently is unstable, the number of created block is less than other shard, so the reward is also less than the others.
The vote stat has certain problem, but if you are not getting slashed or get 0 reward, everything is fine. We are looking at vote stat problem.
missing node monitor statistics
It seems Shard6 is getting unstable as well
To make things more interesting I have 2 validators in shard 6 getting different rewards for epoch 5902
@Support what is going on ?
1 Like
@Support here we go again, 5967 stats missing on my nodes in shards 3, 4, 5, and 6. Got two nodes in shard 2 that is showing epoch 5967 though.
Is it possible, for us whiny bitches, to run the node-monitor locally? I only found the front-end code on github, Is the back-end code closed source?
Chain 3,4,5 and 6 is stuck, so there is no vote count currently. Chain will recover soon
2 Likes
Can you ping me your public mining key?
My validators were updated to 20220627_1 but all of them have stalled. So they seem offline. FYI. @0xkumi
@abduraman @fredlee, please help check your nodes again, make sure they are on the tag: 20220627_1 and syncing the latest blocks. The network was back up a couple of hours ago.
@duc They are on the latest tag but they donāt sync, unfortunately. The state of my fullnode has gone bad too. It receives blocks but cannot receive/transmit any transaction. So strange 
Ah, Iām on 10220621_newdb_1, Iāll switch to 10220627_newdb_2 if that makes a difference. My nodes shows as synced, shard 2 is getting 100% votes. But 3,4,5,6 is still not showing the epoch stats. Itās not showing 0%, itās just blank. So I donāt think Iām getting slashed.
All of my nodes are updated to the latest tag and fully synced 
an offtopic question tho. Why is my node spamming the error.log file with:
2022/06/27 12:37:23 validator key 1B1Qy2sGBAVey...Dwpk, shardID 5, role committee
My nodes are also messed up after last nights situation. 1 got slashed, several off-line, most not having the up to do date code.
Doing Triage this morning.
Why exactly is this happening to all of us?
I updated to the latest tag a while ago and nodes sync works fine. I got no stats and reduced rewards
1 Like
One of my nodes is slashed up to now. I will recover soon ( I hope ).
Btw, while I examine the reason of slashing and fault in my fullnode, Iāve found an issue. Probably incognito-cli or go-incognito-sdk does not work fully or partially since the faucet gives the tx hash but the tx does not enter into the mempool. This is similar to what I face in my fullnode. @Support
Haha, in an odd turn of events, stats for epoch 5968 shows up on 3,4,5, and 6. But now shard 2 does not update vote stats instead. 
99% of my nodes are fine. Odd issue here and there thatās solved with a reboot or resyncing the data.
1 Like
Final situation: One node was slashed and restaked via @fredleeās approach (funder and validator accounts are different). Others were fixed on their own after a while. I think Contabo 
has started to decrease my VPSās resources again on the sly. The extension still cannot import the masterless accounts 
1 Like
I can confirm. They have done the same to all of my servers. Iām assuming youāre seeing slow disk IO?
1 Like
Unfortunately, yes. I run dstat -dcnm --output dstat20220628.csv 10 1000 yesterday and I will run it each day. It gives statistics about disk, CPU, network and memory. To the numbers, they impose increasing restriction on disk I/O speed. In fact, there is no need to these stats. While the restriction level increases, all basic commands (even ls, bash completion) respond really slowly.