Instruction of running new validator's database modes - beta release

[khanhj]

Greeting Incognito Validators,

As you know, we are preparing to release new database modes, including Archive, Batch, and Lite. In addition, there will be another feature called FFStorage, which stores raw blocks in file-type system. For more details, please look at our new database mode introduction.

Note: only Archive/Batch and disabled FFStorage are compatible with old databases, otherwise, you have to resync the database. The bootstrap file will be provided later.

The beta production branch is at production-newdb

1. For the command-line users, build the incognito binary from the source code:

• Database mode can be configured by flag: --sync-mode [archive | batchcommit | lite ]

• FFstorage can be enabled with : --ffstorage

Example of using Archive and enabled FFStorage from the command line

./incognito --nodemode "auto" --sync-mode archive --ffstorage --discoverpeersaddress "mainnet-bootnode.incognito.org:9330" --miningkeys "your-validator-key" --datadir "./path/to/data" --listen "0.0.0.0:9334" --externaladdress "0.0.0.0:9334" --norpcauth --rpclisten "0.0.0.0:9335" --loglevel "debug"

Command to run Fullnode

./incognito --nodemode "relay" --relayshards "all" --sync-mode archive --ffstorage --listen "0.0.0.0:9334" --externaladdress "0.0.0.0:9433" --datadir "./data" --discoverpeersaddress "mainnet-bootnode.incognito.org:9330" --norpcauth --rpclisten "0.0.0.0:9334" --loglevel debug

2. For docker user:

• Add 2 environment vars DBMODE=[archive | batchcommit | lite] and FFSTORAGE=true

• Auto reload script can be download in inc_node_installer_beta.sh (modify variables DBMODE & FFSTORAGE prior to running).

Example of using Lite and enabled FFStorage from our docker repo

$ wget https://raw.githubusercontent.com/incognitochain/incognito-chain/dev/newdb-mode-prod/bin/inc_node_installer_beta.sh
$ nano inc_node_installer_beta.sh  
## Then changing the DBMODE value you want as mentioned above

$ chmod +x inc_node_installer_beta.sh
$ sudo ./inc_node_installer_beta.sh ## then follow the interactive guide
$ sudo docker ps
CONTAINER ID   IMAGE                                               COMMAND                  CREATED         STATUS         PORTS                                            NAMES
efae9b4c993b   incognitochain/incognito-mainnet:10220418_newdb_1   "/bin/bash run_incog…"   9 seconds ago   Up 8 seconds   0.0.0.0:8334->8334/tcp, 0.0.0.0:9433->9433/tcp   archive_fftrue_inc_mainnet_0

Check the running log (optional)

2022-04-19 04:28:33.622 statedb.go:106 [INF] StateDB log: Init database CONSENSUSDB mode: rebuild:21b463e3b52f6201c0ad6c991be0485b6ef8c092e64583ffa655cc1b171fe856-0 pivot:21b463e3b52f6201c0ad6c991be0485b6
ef8c092e64583ffa655cc1b171fe856--1
2022-04-19 04:28:33.624 statedb.go:106 [INF] StateDB log: Init database TXDB mode:archive rebuild:21b463e3b52f6201c0ad6c991be0485b6ef8c092e64583ffa655cc1b171fe856-0 pivot:21b463e3b52f6201c0ad6c991be0485b6
ef8c092e64583ffa655cc1b171fe856--1
2022-04-19 04:28:33.624 statedb.go:106 [INF] StateDB log: Init database FEATUREDB mode:archive rebuild:21b463e3b52f6201c0ad6c991be0485b6ef8c092e64583ffa655cc1b171fe856-0 pivot:21b463e3b52f6201c0ad6c991be0
485b6ef8c092e64583ffa655cc1b171fe856--1
2022-04-19 04:28:33.624 statedb.go:106 [INF] StateDB log: Init database REWARDDB mode:archive rebuild:21b463e3b52f6201c0ad6c991be0485b6ef8c092e64583ffa655cc1b171fe856-0 pivot:21b463e3b52f6201c0ad6c991be04
85b6ef8c092e64583ffa655cc1b171fe856--1
2022-04-19 04:28:33.624 statedb.go:106 [INF] StateDB log: Init database SLASHDB mode:archive rebuild:21b463e3b52f6201c0ad6c991be0485b6ef8c092e64583ffa655cc1b171fe856-0 pivot:21b463e3b52f6201c0ad6c991be048
5b6ef8c092e64583ffa655cc1b171fe856--1

---

Note on usage:

Although these features are already tested in several environments, it is not battle-tested yet.

We recommend that node operators should run at least 1 node with a production branch. In case of bug or db corruption of the beta branch, you can clone and revert to the production branch.

[khanhj]

It is worth mentioning that:

• we recommend to run batchcommit or archive for fullnode
• lite mode for fullnode still work to get account balance, may fail to create new transaction (but you can retry, as always)

[Josh_Hamon]

Using docker, I added

DBMODE="lite" 
FFSTORAGE="true"

In the config section (above VALIDATOR_K). Then I ran sudo ./script -y

It doesn’t seem to have updated the docker tags or container names. Per above is that expected behavior?

[Jared]

These variables will only work when these features are released into that docker tag. Right now you are good to go for when it is released, however, if you want to do beta testing (thank you in advance) you can do the following:

Grab and edit this script:
https://raw.githubusercontent.com/incognitochain/incognito-chain/dev/newdb-mode-prod/bin/inc_node_installer_beta.sh

The instructions for use are at the top. Notice if you already have a running node and would like that node to run beta then you would need to stop that node and then run this script.

[fredlee]

Did the developer team profile and find any memory leaks in docker tag 10220418_newdb_1?

When running --syncmode batchcommit (without ffstorage), first of all it eats a lot of memory, but I have also noticed that it’s not freeing up the allocated virtual memory. This is a screenshot from it running for a couple of days.

Then after restarting the container and waiting for it to spin up…

That looks like leaks or bad memory management to me.

[khanhj]

Hey @fredlee
The team acknowledge the issue with RAM and still implementing the fix.
I will let you know the new tag code to test later.

[fredlee]

I am now running a bunch of nodes on lite and lite_ff, but they require regular restarts because of the memory leaks. Does 10220502_newdb_2 address memory issues? Should I start testing it or wait for another release?

[duc]

Oh you realized the new docker tag released for the beta storage mode
We were monitoring on our nodes to see if the memory leak issue is resolved and will let the community validators know once we confirm that. But you can always run your couple of nodes with the new docker tag 10220502_newdb_2 to help the team with more testing environment.
Thanks again for your volunteer on the beta test!

[duc]

Hey @fredlee, just wanted to confirm that the new docker tag (10220502_newdb_2) doesn’t solve the high memory use issue, we still keep profiling and fixing it. Thanks for your patience.

[fredlee]

Ah, thank you. Good! Because after running it for a day I was not that impressed with the memory usage.

For hosting a couple of nodes, memory usage is not that important. Because the minimum memory you get with hardware is enough. But at a certain point, it’s harder and more expensive to get more RAM than to build a bigger RAID. Just my ¢2

[fredlee]

I’ve run the new database modes now for a couple of months. Because of the memory problems my nodes are doing rolling restarts when not in committee and it has been working without any issues. I’m currently running the 10220714_newdb_2 release as it’s the latest docker tag I could find.

1. Are the memory issues solved in the latest version, should I restart memory testing?
2. Was the goal to keep all these modes in the future or test out which would be most viable?
3. Is there any mode or anything special I should focus testing on at this point?
4. Are we getting a newdb release with pruning?

[Jared]

@fredlee

Was the goal to keep all these modes in the future or test out which would be most viable?

Actually, these will be discontinued in favor of state pruning.

@fredlee

1. Are we getting a newdb release with pruning?

See above.

[fredlee]

Interesting, then I’ll transition all my nodes into state pruning.

[Jared]

In order to help reduce the CPU burden for vNode operators, the devs are working on building bootstrap files with state pruning already implemented. We will notify the community when these are ready.

[fredlee]

Sounds good. No worries tho, I only need to do it once, all my nodes are clones.