About today's app issue

Earlier, some of you were unable to load the Incognito app. The problem was caused by an issue with the app fullnode, not the network itself. Blocks continued to be produced, validators continued to earn, and so on.

As of 4 hours ago, the app is fully functional.

Here’s a little more information about what the fullnode does, and what happened to it.

Image from iOS (5)

The fullnode helps to receive and transmit transactions on the Incognito network. This process is passive and relies on Nodes for updates. Issues arose from a conflicted database when an update was deployed chain-side this week. It was unable to sync fully with current blocks and heights.

We restarted the fullnode and fixed the issue. It was a fairly simple fix, but the issue caused considerable disruption.

To avoid similar situations in the future and preempt possible scenarios, here’s what we will do going forward.

  1. Build a monitoring tool for the fullnode so we can react quickly.
  2. Set up filters/barriers to block potentially malicious behaviors.
  3. Install a 24/7 ‘watch’ of core dev members.
  4. Maintain multiple fullnodes to ensure they can cover for each other.

Thank you to all in the community who stayed calm and helped others. To all those who were inconvenienced, please accept our apologies. Timezone issues meant we were not able to respond in time to deliver what should have been a quick fix.

We will continue to install technical safeguards, and are currentIy working on improving the ‘human’ layer – 24/7 support to keep pace with how fast Incognito is growing.

As always, thank you for being patient with us.

30 Likes

Hi @Peter

Very much appreciated for the response to the situation regarding today’s case, the full node issue.

Precautions are very necessary to avoid the slightest problem that arises, will greatly affect the comfort and trust of users.

User trust and comfort are very important. If the user feels comfortable. Then they will be happy to be part of the community, otherwise they will leave.

First impressions are very important.

Can you imagine, if I were a new user and had just deposited thousands of dollars. Then suddenly the wallet cannot be accessed, so naturally I will panic and my first impression will be very bad.

Maybe I’ll leave after everything gets better. This means that the project has lost one potential investor or user.

Or even if an old user, if often have problems. Then it will cause frustration and disappointment.

Finally, thank you to the project team who quickly responded to each case that occurred and resolved it quickly.

8 Likes

Thanks Peter!

Could you add the feature below to the todo list please? :pray: :pray: :pray: :pray: :pray:

2 Likes

Thanks Peter for the update. I think these all look like valuable changes.

3 Likes

Thanks for the clear explanation as to what happened, with a list of things to hopefully prevent this in the future. Is the end goal to have the fullnodes decentralised as well? Or do I get that wrong?

1 Like

Hi @Trader3 ,

You’ve made some very good points here. We surely understand how important first impression and trust are for new users, especially with a product involving money. As Peter already stated, we have worked out a plan to avoid similar situations. The monitoring tool will be built soon and we’re trying to set up a 24/7 user support so at least there’re someone user can talk to right away!

1 Like

The end goal won’t be to have the fullnode decentralized, but to make it work more stable, avoiding these similar issues. This fullnode is different from the nodes powering the network.

Hey, some of you may have noticed that the app had a rough day last Friday. While the monitoring tool is now live, it still needs to be improved to cover all scenarios.

Which scenarios? In the interests of transparency, here’s a quick update.

  1. The fullnode was unable to handle the requests from the app. Those requests are for creating the transactions. If they can’t reach the chain network, they cannot complete.

  2. Some of the APIs used to check fullnode information were unresponsive because they lacked resources to run. As a result, we didn’t receive the alert in time.

  3. Slow app loading due to UTXOs. Incognito uses a UTXO based ledger. In most cases, fewer than 32 existing UTXOs are used as inputs and 2 new UTXOs are produced as outputs. The more UTXOs your wallet has, the longer it takes for your app to load.

To do:

  1. Build a full monitoring system & alert system for all API/ status requests. We are focusing on both the fullnode performance and back-end API (Shield/ Unshield processes). Devs will receive them directly via internal channels. It will be done this week.
  1. Improve loading and caching for UTXOs. It will be done this month.

  2. Review the whole app to verify components. We will start work this month.

Please feel free to ask questions if you have any, and thanks again for your patience.

2 Likes

Can we look into including this?

1 Like

It sounds familiar with this: How to use my own fullnode on the mobile app?
Provide a way so you/ a user can switch to another fullnode through the app is possible. But I am not sure what will happen with the current transaction history (if any).
Let me check with the team.

1 Like

Awesome thank you @Peter!