[Shipped] Reduce Blockchain size by 50%

so cool @Mike_Wagner, i love it

Awesome! I like the idea of the db icon either looking short in the icon or even crushed like an aluminum can.

By the power vested in me by the internet - I hereby dub @hungngo the DATABASE CRUSHER.

Screen Shot 2020-03-19 at 10.48.13 PM

10 Likes

thanks guy, we will bring more good news soon

1 Like

Hi @hungngo,

I tried to get your code and run to sync data from the mainnet. I saw that at beacon block 90496, I couldn’t sync it because of this error log:

“2020-03-17 22:46:09.652 beaconprocess.go:302 [CRT] BlockChain log: FORK BEACON DETECTED, New Beacon Block Height 90496, Hash 44d925ddeed6b8a39f186c9d93356d4900eba3a988229f5d9a0beb1000c91715, Expected Previous Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230, BUT Current Best State Height 90495 and Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230
2020-03-17 22:46:09.652 synker.go:976 [ERR] BlockChain log: -1048: Fetch Beacon Block Error
-1001: block 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230 not exist
Get Beacon Block By Hash Error
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.NewRawdbError
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/error.go:157
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.GetBeaconBlockByHash
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/accessor_beacon.go:66
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:300
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:102”

I needed to restart node, and it synced continuously again. I think this is because beacon block 90495 could not be inserted into DB, but we update the best state of beacon chain. So we need to review the reversion of this process. When I review this logging text, I think it doesn’t make sense in this scenario.

Please have a look.

1 Like

@thaibao i solved this problem. This problem happened because node didn’t revert current state to previous state when it failed to insert new block

REPORT 14 Mar - 20 Mar 2020
Current Step

Finish:

  • Fixed bug and passed all QC testcase in dev/net, local network
  • Finish code review by @thaibao (let’s give him a big clap)

In progress:

  • We are moving to testnet, we deployed source code with database version 2 to shard 5 in testnet
  • Add time log and monitoring node operation with database version 2

To Do Next:

  • If things are smooth, we will move on mass testnet committes deployment and mainnet fullnode deployment
  • Merge code and support other team with our new database infrastructure

I have all the confident that our blockchain size will be 10 time smaller soon

2 Likes

Wonderful news over the weekend!

  • We have deployed a mainnet fullnode with database v2. The blockchain size in this fullnode is only 12GB compared to the current 150GB.
    Log rotation in two days consumed storage more than 5-months blockchain size. lol

Screen Shot 2020-03-22 at 21.54.43

6 Likes

Yeah, so great. We’re trying to move some old mainnet node to this DB version and run a while before moving all nodes to this great version.
This week
All testnet node will be moved for this version
Few mainnet nodes will be moved later

4 Likes

@aaron @ning let’s create a long topic explaining this. this is a big improvement that is super helpful to our validators! from 150gb to 12gb, that is indeed a wonderful news @hungngo :slight_smile:

cc @thaibao @0xkumi @binh

2 Likes

REPORT 21 Mar - 29 Mar 2020
Current Step

Finish:

  • Pass all QC testcase on both dev/net and test/net, no bug was found this week.
  • Testnet: All node in testnet now using the new database version
  • Mainnet: One node in each shard and beacon run with database version 2

In progress:

  • Testnet: monitoring node behavior
  • Mainnet: monitoring node behavior and continuously update node by node to database version 2

This weeks we received NO BAD NEWS. As scheduled, that we will deliver this feature to all node in mainnet within next 2-3 weeks.

2 Likes

Unfortunately, we having a trouble with instruction hash of beacon validator that use db v2. Maybe issue is in reproducing hash from instructions of block reward that mean beacon producer create a block with instruction hash which is different from beacon validator’s hash who using db v2 . @hungngo and @hyng are going to try hard to fix this issue earlly and we can retry on beacon validator again.

Add log error for this case

3 Likes

All great things take more trial and error than most people will ever know.
I am sure you will fix it.

1 Like

Team fixed this problem and in middle of testing and re-deployment. This problem didn’t effect database v2. So every node will be deployed as schedule

7 Likes

Perfect

Thank @hungngo

REPORT 30 Mar - 4 Apr 2020
Current Step

Finish:

  • We successfully deployed dbv2 to half of shards (all shard) beacon committees
  • Basically everything is ready for mass deployment and official release

In progress:

  • Still in deployment phase. This phase will be closed when all node in shards, beacon committees run with new database version

This proposal can be closed this week, no problem arise last week. Deployment phase is on a good track.

5 Likes

This is great progress @hungngo ! Will our nodes automatically update or do we have to reinstall? The current blockchain size has pushed Digital Ocean vnodes under breakeven, but those can’t be downgraded, so new droplets will need to be setup. Any idea what day this might be occurring this week? Thanks!

1 Like

@JG20 For manual update, which works by now, we can just follow these simple task:

  1. Delete our current database (all of them) and start again from the beginning
  2. Run with our new docker tag

@thaibao Please help us answer these question, tks a lot

  1. Could you tell us when will be able to make an official release?
  2. Will node got automatically update?

Thanks, @hungngo. What is the new docker tag?

I will publish it today or tomorow. But if you guys want to run it now, please view docker tag dbv2_20200331_1

I used it for many nodes and fullnodes

1 Like