Buy crypto anonymously on the Incognito pDEX   

[Resolved] Recent Network Outages

#1

Hello Incognito Community, :wave:

Update May 8th, 2023 - All Provide rewards and pNode rewards should process today by end of day. We are monitoring the backend systems to ensure everything is running smoothly for these rewards.

Update May 4th, 2023 - Good news! The network is now operating at 100%. :partying_face: :raised_hands:t2:
Stuck/pending TXs will automatically be retried by the backend. If you have previously opened a support ticket we will monitor all of them to ensure TXs are being processed.

Update May 3rd, 2023 - We have released a new Docker tag 20230503_1. This code goes live after 67% of nodes have updated. We will monitor the network after this update goes live and update the community accordingly.

Recently there have been some network service interruptions impacting shields, unshields and swaps. This post hopes to inform the community of what happened, what is happening and what will happen.

What Happened?

A few days ago, the group Trail of Bits (https://www.trailofbits.com/about/), a security research company, contacted us to inform they have identified an issue in our bulletproof implementation, that we use with our privacy layer.

We have patched the issue with Docker tag 20230429_1, the fix has been reviewed and confirmed by the Trail of Bits team. Unfortunately, after the patch was rolled out (and reached 70% majority), unforeseen bugs in this release impacted the other parts of transaction creation process that caused the lately unreliable transaction inclusion (as pointed out here - https://we.incognito.org/t/20441).

:warning: Node operators running Docker tags prior to 20230429_1 will not be able to earn rewards going forward. Please ensure your node(s) are up-to-date and running 20230429_1 to continue earning rewards.

What is Happening?

As of today, the dev team is hard at work to fix the network service interruptions and bring the network back to 100% functionality. Funds are 100% safe and the team is dedicated to the long-term success of the Incognito network. We have developed a queue to make sure we review all support tickets and process them accordingly.

What will Happen?

The dev team is working around the clock and overtime to ensure all bugs and issues with the network service interruptions are fixed. The team has laid out a timeline of ~48 hours until the network is operating fully at 100%. During that 48 hour window, more services will become available.

Community users who have opened a support ticket by contacting either myself (@Jared) or @Support directly will receive a follow up message once their TX has processed.

It is not necessary to open a support ticket during the next 48 hours as TXs that are stuck/pending will automatically process once the network is fully operational.

Moving Forward

On behalf of the entire Incognito team, I want to offer our sincerest apologizes for this network service interruption! :pray: The entire team has pulled together and worked overtime to rectify this as soon as possible. We understand that crypto network instability can be an annoyance/frightening experience to the end user and that is why we have put together this transparency report to help ensure the community is knowledgeable to the issues at hand.

Finally, thank you to you, the wonderful community members of Incognito! Users such as yourself help to show the Incognito team that privacy in the cryptocurrency space is much needed and sought after. :raised_hands: :prv: :incognito: :lock:

Thank you :heart:

20 Likes
Reward withdrawal stuck?
Shielded coins doesnot apper/ resolved
Consolidation of UTXOs
Awkward transaction
PRV Provide Reward withdrawal still pending
Provide Unlocking - Stuck in Pending
#3

Was there any notification sent out about this :point_up_2: upgrade prior to it being rolled out?

I realise it may have been sensitive in nature as it was to fix a security issue in the privacy layer, but still it would have been nice for Incognito Core Dev Team to inform Node operators that a new version was going to be rolled out.

AN IDEA TO PREVENT THIS IN THE FUTURE
I believe in the saying “Prevention is better than cure”.

To prevent transactions being stuck and inconvenience and worry caused in the future, perhaps this is what Incognito Core Team should do:

  1. Provide (at a mimum) 24 hours notice that a Network upgrade is being rolled out across all nodes via a Popup Message in the App that the user must acknowledge to have read before continuing to use the App.
  2. Have a countdown timer located in the bottom footer on every page of the App informing the user that

:warning: A Network upgrade will start in: XX Hours XX Minutes

  1. Have a Warning Message appear 1 hour prior to the upgrade starting and that it is advised not to:
  • Shield/Unshield
  • Swap
  • Withdraw from Provide, etc.
    until the network upgrade has been completed.

Whilst the upgrade is taking place, consider limiting the functionality within the App, E.g. Prevent certain functions such as swaps, but make is clear that it is due to a network upgrade and full functionality will return in XX hours once the upgrade has been completed successfully.

FINAL THOUGHTS
Do other community members think it would be nice to have improved communications in the App as to when upgrades are about to occur?

Please LIKE :+1: if you agree.

14 Likes
#4

Hi @Linnovations,

Previously we have put a notification in the app about upcoming updates and also frozen certain aspects of the app. There’s a history of all of these notices on the main screen of the app under the bell :bell: icon in the top right.

It was decided this was not the best decision given the potential security risk. Unforeseen bugs have surfaced with the patch integration causing the Network Service Interruption.

I like this, it could help to make sure all app users see the message. However, we would need to figure out what to do about users not using the mobile apps.

I just did a test myself and I was able to swap PRV > USDT and then unshield to my wallet without issue. The only problem I noticed was a lack of TX history within the app.

3 Likes
#5

Well said!

1 Like
#6

Hi @Jared, and how is it going with the repairs, what is the expected timeline?
Thanks for the update!

3 Likes
#7

Shielding must still be down? I have shielded some USDC from ETH everything has confirmed on ETH block explorer but no transaction history is showing up in the mobile app, website wallet or Chrome extension. Is there anything i can do on my end or is do I just need to continue waiting… Please let me know @Support

1 Like
#8

@Cob1a34d21 & @Crypto_0G,

Please see the update at the top of the main post.

2 Likes
#9

I think you mean update May 3 :wink:

1 Like
#10

Yes, thank you. Just fixed it :sweat_smile:

1 Like
#11

Many of my vNodes gone to “offline” mode since few days. I have been reading these messages in the forum and was hoping the nodes come back online after the network outage issue gets resolved. However many of my Vnodes are still offline. Any recommendation what commands to be run to bring these nodes online from being offline due to network outage?

#12

Please open a support ticket. I don’t want to derail this thread.

http://we.incognito.org/g/Support

#13

From my side, I’m not able to do any swap… I know there’s a waiting list to process the withdrawals from provide. Could you get us updated about resolving the issues on “Provide”. Thank you.

#14

For me its also not working, my shielding is still processing. I assume we have to wait until the new release is updated by 67% of the nodes, but would be great to know what the current status is of this update (are we at 10% or 59% or…)?

#15

Thank you for your patience.

I spoke with the dev team and we still have 4 shards operating without enough nodes running the latest release. Once all shards have enough nodes running the newest release then the network will be back to 100%.

1 Like
#16

Thank you for the information and updates.

And thank you Trail of Bits for their help! :clap:
Can you provide any more context around this? Were they performing an audit or did they happen to find Incognito by scanning tons of contracts searching for instances of this vulnerability?

Speaking of audits … I know Incognito has had some in the past, but things change and new bugs are sometimes introduced :grimacing:. When was the last audit and how does the team decide another audit is needed?

1 Like
#17

At this time, we can not offer much information as it could potentially impact other projects. Once we get the go ahead from Trail of Bits we will make a public post detailing more information.

Incognito smart contracts were audited by Coinspect and you can find their report here: https://www.coinspect.com/doc/Incognito%20-%20Smart%20Contract%20Audit.pdf

2 Likes
#18

Indeed so @Jared, for me its working smooth and fast again :smile:
Thanks for the reassurances while waiting and of course the hard work by the development team. Have a great day!

3 Likes
#19

Thanks @Jared for the update. I’m running a fullnode and I use the incognito-cli in order to consolidate my PRV earnings from vNodes. When I try to withdraw vNodes earnings from incognito-cli, I have this error:

“Cannot create reward withdrawal transaction: RPC returns an error: &{-6004 Pool reject invalid tx: signature, or proof or verify by itself fail -6004: -1001: Reject invalid tx -1025: Can not verify proof of tx [xxxxxxx] AggregatedRangeProofV2 of PaymentProofV2 must be version 2 or higher”

I’ve got almost the same issue while trying to do a normal TX. Could you confirmit is related to the fact that nodes aren’t,as of now, running the latest version of software? Thanks.

#20

From what I can tell, the cli project was not updated with the protocol changes. But it’s on its way.

https://we.incognito.org/t/started-having-problems-with-my-incognito-cli/20507/5?u=fredlee

2 Likes
#21

@support I’m not sure things are fixed. I still have a pending prv withdrawal from april

Incogo
Incogo1284Ă—2778 83.8 KB

1 Like