Slashed nodes

fwiw, restaking is much faster and simpler now than in the old days.

Do you have automatic updates turned on? I believe there was a docker upgrade rolled out on Ubuntu.

Yes, automatic updates are turned on.

1 Like

I’ve had a few slashes occur which are a bit unusual since the hardware is more than capable and I’ve been running nodes for a while. I need to investigate further.

I just had a healthy node get slashed. It started earning and then got slashed. How do we get them back online?

Please always check the Node Monitor page (monitor.incognito.org) as this will give you the best and most accurate information.

For pNodes:

For vNodes:

If your only issue is stuck beacon/shard data then bootstrap with the following:

1 Like

Thanks @Jared, i’ll give that a try.

2 Likes

Houston, we still have a problem. Just had another vnode slashed. Was showing online, fully synced, etc. Then I get a notice a few hours later via Nito Bot that it’s been slashed, but it’s still showing online and fully synced. This is starting to happen more frequently. Either the node monitor is missing something or there’s a slashing issue.

Bonus Question: Can we get a built-in system or app notification telling us when a node goes offline so that we can fix the problem before it gets slashed?

@Jared @Support

1 Like

Are you sure you only have this node on one server?

What host is your server running on?

This functionality existed with the old app. Due to possible security/privacy issues, the app notifications were removed. The devs have decided to allow the community to develop these systems (like Nito bot) so they can focus on other functionality.

I use Contabo. There are other nodes on the same server, but server performance doesn’t appear to be an issue (via webmin). Been running multiple nodes on servers for a long time but the sudden slashing issue is new.

1 Like

:warning::warning::warning::warning:
This is your issue right here. If it is a VPS they throttle your disk I/O to the point where it is worse than a HDD speeds.

Stop all of your nodes and run:

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync

Let me know the output.

1 Like

@Jared
While bootstrapping the slashed pnode, I got an error with the beacon, (picture attached), shows a corrupted block. that was the first part, i have not moved on to the shard yet.

Sorry about that @VINWOSNI, the devs are currently in the process of upgrading the bootstrap files to include state pruning as it can be CPU intensive and will help to reduce blockchain storage space.

1 Like

@Jared,
Ok, thank you. Is there a timeframe of when I should try again or any other steps I can take to reverse the slash?

We currently do not have a timeframe. I would just let it resync for now and I’ll let you know when the bootstrap is ready.

1 Like

@Jared,
The pnode shows offline now, i’m assuming because of the deletion process at the start of the bootstrapping but not able to copy the files back? honestly, i don’t know.
Am we able to chat on telegram to get it online and syncing? I don’t know how to make it start up again.
(when i closed out the bootstrapping, i used command 88 as instructed, but as i noted before, the process didn’t finish, including the shard part. should i run it again for the shard?)

For now run the following:

sudo RM -rf /home/nuc/aos/inco-data

1 Like

I’m doing something wrong, I’m getting “command not found” when I try to run it.
Am i trying to run the command from the wrong prompt?

@JG20, this keeps happening to me too. about the same time as your first post in this thread. contabo as well. i wrote to them about this sudden dip in performance, and they told me to get a VDS :smiley:

i’ve reduced the number of nodes active on my VPS XL at any one time to just 4, and there are still issues. tearing my hair out!

sorry I was on mobile. rm not RM.

1 Like