Hello Incognito Community,
Update May 8th, 2023 - All Provide rewards and pNode rewards should process today by end of day. We are monitoring the backend systems to ensure everything is running smoothly for these rewards.
Update May 4th, 2023 - Good news! The network is now operating at 100%.
Stuck/pending TXs will automatically be retried by the backend. If you have previously opened a support ticket we will monitor all of them to ensure TXs are being processed.
Update May 3rd, 2023 - We have released a new Docker tag 20230503_1. This code goes live after 67% of nodes have updated. We will monitor the network after this update goes live and update the community accordingly.
Recently there have been some network service interruptions impacting shields, unshields and swaps. This post hopes to inform the community of what happened, what is happening and what will happen.
A few days ago, the group Trail of Bits (https://www.trailofbits.com/about/), a security research company, contacted us to inform they have identified an issue in our bulletproof implementation, that we use with our privacy layer.
We have patched the issue with Docker tag 20230429_1, the fix has been reviewed and confirmed by the Trail of Bits team. Unfortunately, after the patch was rolled out (and reached 70% majority), unforeseen bugs in this release impacted the other parts of transaction creation process that caused the lately unreliable transaction inclusion (as pointed out here - https://we.incognito.org/t/20441).
Node operators running Docker tags prior to 20230429_1 will not be able to earn rewards going forward. Please ensure your node(s) are up-to-date and running 20230429_1 to continue earning rewards.
What is Happening?
As of today, the dev team is hard at work to fix the network service interruptions and bring the network back to 100% functionality. Funds are 100% safe and the team is dedicated to the long-term success of the Incognito network. We have developed a queue to make sure we review all support tickets and process them accordingly.
What will Happen?
The dev team is working around the clock and overtime to ensure all bugs and issues with the network service interruptions are fixed. The team has laid out a timeline of ~48 hours until the network is operating fully at 100%. During that 48 hour window, more services will become available.
Community users who have opened a support ticket by contacting either myself (@Jared) or @Support directly will receive a follow up message once their TX has processed.
It is not necessary to open a support ticket during the next 48 hours as TXs that are stuck/pending will automatically process once the network is fully operational.
On behalf of the entire Incognito team, I want to offer our sincerest apologizes for this network service interruption! The entire team has pulled together and worked overtime to rectify this as soon as possible. We understand that crypto network instability can be an annoyance/frightening experience to the end user and that is why we have put together this transparency report to help ensure the community is knowledgeable to the issues at hand.
Finally, thank you to you, the wonderful community members of Incognito! Users such as yourself help to show the Incognito team that privacy in the cryptocurrency space is much needed and sought after.