State pruning

In this topic, we are presenting an improvement mode for Incognito stateDB that would help Incognito fullnode saves at least 60% data storage.

What is state pruning?

Incognito node uses stateDB for storing state data such as pDEX, tokens, and transactions. And each time one leaf node changes value. New state trie will need to build a new path and new leaf node for the changed value. Therefore, with the purpose of decreasing the storage of stateDB, we will try to remove old paths and old nodes.

Frame 216

ETH state pruning

ETH uses a bloom filter to filter trie nodes that do not belong to a specific state root and genesis root. For more information check out this article:

Incognito state pruning

Based on ETH state pruning, Incognito also uses a bloom filter to filter state trie nodes that do not belong to current final and best states.

How to prune data

We’ve supported the following two RPCs for triggering the pruning process and checking the pruning status. We will support a call to action button on the Incognito app’s Power tab so that you can trigger the process via a more user-friendly UI in the future.

Note that pruning is a CPU-intensive process so if you are running multiple nodes on the same server, please take CPU computation into account when triggering it.

For example, you are running 10 nodes on a single server but CPU capacity of the server can only handle pruning for a maximum of 4 nodes so if you trigger the pruning for more than 4 nodes, the server would likely be stressed and all 10 nodes on it would be affected.

Prune

Call the following RPC to activate the state pruning process in the background:

curl --location --request POST "node_IP:rpc_port" \
--header 'Content-Type: application/json' \
--data-raw '{
    "id": 1,
    "jsonrpc": "1.0",
    "method": "pruneState",
    "params": [
        {
            "Config": {
                "0": {
                    "ShouldPruneByHash": false
                },
                "1": {
                    "ShouldPruneByHash": false
                },
                ...
                "7": {
                   "ShouldPruneByHash": false
                }
            }
        }
    ]
}'

ShouldPruneByHash is a boolean param. If it is true, the process would traverse by rootHash. Otherwise, If it is false, the process would traverse by height (default and recommended)

Get prune status

curl --location --request POST "node_IP:rpc_port" \
--header 'Content-Type: application/json' \
--data-raw '{
    "id": 1,
    "jsonrpc": "1.0",
    "method": "getPruneState"
}'

Example response:

{
    "Id": 1,
    "Result": {
        "0": {
            "Status": 1,
            "PrunedHeight": 2934
        },
        "1": {
            "Status": 2,
            "PrunedHeight": 1111
        },
        ...
        "7": {
            "Status": 2,
            "PrunedHeight": 3456
        }
    },
    "Error": null,
    "Params": null,
    "Method": "getPruneState",
    "Jsonrpc": "1.0"
}

Status:

)
 0: IDLE  
 1: PRUNING  
 2: CHECKING
)
8 Likes

can you explain the difference between prune by rootHash and by Height?
CPU/RAM consume during pruning, Disk space released after that. For example, if i have 100Gb beacon data and 80Gb Shard-0 data?

1 Like

Prune process will prune transactionStateDB, It means transactionRootHash will be stored somewhere in levelDB by shardBlockHash will be the key and transactionRootHash will be value. In fork scenario, and shardHeight is the first height of epoch, next transactionRootHash will be replaced, prune process by rootHash will scan all theses scenarios and process height will not. But don’t worry this scenario rarely appears in Incognito chain network.
Therefore, CPU/RAM consume during pruning there is no difference between height and rootHash prune process. Disk space release with rootHash process will be greater but it is not worth for paying attention to.
For example, if I have 100Gb beacon data and 80Gb Shard-0 data? rootHash process will release 51.001 gb of 80 gb shard 0, and height process will release 51.000 gb of 80 gb shard 0

1 Like

So when you say node_ip:node_port, which port would that be ? Docker container has 2 ports forwarded/listening ?
3a69aa086958 incognitochain/incognito-mainnet:20220801_1 "/bin/bash run_incog…" 7 days ago Up 5 days 0.0.0.0:9334->9334/tcp, :::9334->9334/tcp, 0.0.0.0:9433->9433/tcp, :::9433->9433/tcp inc_mainnet_1

Updated to the topic, It’s RPC port, 9334 in your case.

1 Like

Another thought: with no authentication it would be possible to mount a denial of service attack by issuing prune requests on all ports listening on a particular IP. That would likely overwhelm the CPU and there would be trouble.

  1. If a single node is in the pruning process, the next pruning request for the current shard will be invalid.
  2. RPC port should not be leaked to trustless nodes or users. Therefore, RPC port should be secured for private nodes
  3. Not only pruning RPC but also other RPC methods can be used to attack node if RPC port is not secured.
1 Like

RPC port has to be open for the validator page on the app to work correctly. So stake / unstake process would require private key, but port has to be open to the internet for the app (validators) to work ?

When I call "getPruneState" I get "Result": null instead of a list of shards. The prune command returned "Message": "Success" so I think it is running. Do I just have to wait longer before I can get any status or am I missing something else?

Yes, the PruneState will update after it completes. You can run docker stats or htop and see the CPU crunching away to know it is working.

So what does “Status”: 1 mean? :smirk:

I was informed there’s an issue calling getPruneState

1 Like

Ok, cool, then it makes sense.

1 Like
  1. RPC port has to be open for the validator page on the app to work correctly app use full node not the private node
  2. but port has to be open to the internet for the app (validators) to work ? RPC port has not to be opened to the internet. Only p2p listener port is required for validator to work well

Adding a vnode in the app:


Isn’t that the RPC port ?

Yes that is RPC port. To setup vNode with mobile app, RPC port of validator node is necessary to get validator key of node and check which the key is matched with user key in the mobile wallet.
So when you setup vNode with mobile app, you should ensure connect in the private IP. Another way is setting whitelist IPs which are able to interact with validator RPC port, if you want to connect by public IP

3 Likes