Earlier, some of you were unable to load the Incognito app. The problem was caused by an issue with the app fullnode, not the network itself. Blocks continued to be produced, validators continued to earn, and so on.
As of 4 hours ago, the app is fully functional.
Here’s a little more information about what the fullnode does, and what happened to it.
The fullnode helps to receive and transmit transactions on the Incognito network. This process is passive and relies on Nodes for updates. Issues arose from a conflicted database when an update was deployed chain-side this week. It was unable to sync fully with current blocks and heights.
We restarted the fullnode and fixed the issue. It was a fairly simple fix, but the issue caused considerable disruption.
To avoid similar situations in the future and preempt possible scenarios, here’s what we will do going forward.
- Build a monitoring tool for the fullnode so we can react quickly.
- Set up filters/barriers to block potentially malicious behaviors.
- Install a 24/7 ‘watch’ of core dev members.
- Maintain multiple fullnodes to ensure they can cover for each other.
Thank you to all in the community who stayed calm and helped others. To all those who were inconvenienced, please accept our apologies. Timezone issues meant we were not able to respond in time to deliver what should have been a quick fix.
We will continue to install technical safeguards, and are currentIy working on improving the ‘human’ layer – 24/7 support to keep pace with how fast Incognito is growing.
As always, thank you for being patient with us.