[Outdated & Changed] Dynamic committee size and dynamic sharding: implementation phase

Objective:

  1. Improve the robustness of the chain: avoid the chain blocked by the unavailability of nodes, network disconnected, inadequate node power.
  2. Adapts well with the variety of the number of validators
  3. Strengthen the security: trustless to shard committee

Estimate End Date: June 2021

July 2020

  • Beacon sync, insert and store all shard blocks
    • Beacon node must sync and insert shard from beginning
  • Refactor Committee State in Blockchain package for later implementation
  • Technical Specification for New Staking Flow

August 2020

  • Implement new staking flow

    • Staking to become candidate
    • Candidate Life cycle (not included monitoring and report)
    • Unstake flow
    • Refine Stop auto re-staking flow
  • Implement new swap rule

    • Not included increasing/decrease (case 3,4) shard
  • Technical Specification of Slashing Misbehaved Validator

September 2020

  • Integrate new Staking flow and swapping rule with with Smart Contract
  • Implement Slashing Misbehaved Validator

October 2020

  • New Staking flow and swapping rule are ready for local test (TestNet in happy case)
  • Implement, Integrate and testing Slashing Misbehaved Validator with Smart Contract and other Feature

November 2020

  • Ship New Staking flow and swapping rule at the end of Nov or early Dec
  • Ship Slashing Misbehaved Validator at the end of Nov or early Dec

March 2021

  • Reduce committee size dynamically

June 2021

  • Dynamic grow/shrink number of Shards

Key results:

  • Committee size is dynamically changed with the number of validators
  • Dynamic sharding
  • Beacon is able to verify the correctness of every single shard. As a result, a shard could verify cross-shard transactions through the beacon chain.

Note:

27 Likes

This task will begin in 1st Jun:

  1. Design working flow @hy @hungngo @0xkumi @dungtran
  2. Implementation @hy @hungngo
  3. Unittest, local testing @hy @hungngo
  4. Integration testing by QC
7 Likes

Update 2020 Jun 8:

Github link: https://github.com/incognitochain/incognito-chain/tree/dev/dynamic-committee-size
Result:

  • Finish implement Beacon sync, insert, store all shard blocks

Next:

  • Testing
7 Likes

Update 2020 Jun 12:

Result:

  • In testing phase and fixing bug

Next week:

  • Testing
  • Move to next task while waiting for QC
5 Likes

On June 24th, 2020

  • Update the random shard assignment scheme to keep the balance of the shards’ validators.
  • Update the solution for problem 2,3

Detail design: https://docs.google.com/document/d/19KChrg0B2LmfT_t-K8vovCG-mXUkAYG88j5FQsKPXT4/edit#

7 Likes

Update 26 June 2020

Finish:

  • Agreement on refactoring code approach
  • Refactor Instruction related to committee (stake, assign, swap, stop-auto stake)

Next:

  • Refactor Committee State, remain the current assign and swap rule
  • Test after finish refactoring code

This week, we do not spend much time on the main task as the code based of incognito-chain is very huge now. After thoughtfully consideration, we decide it need to be refactor into smaller component, make it easier to maintain in the future. It might take some time now, but new staking flow can be well-implemented and well-tested with high quality code.

9 Likes

Update 3 July 2020

Finish:

  • QC testing on Beacon Sync as Fullnode
  • Implement Beacon Committee State

Next:

  • Begin implementing Shard Committee State
  • Begin implementing New Staking Flow
  • Finish implementing + unit-test Beacon Committee State
6 Likes

Yesterday, we talked about dynamic committee size - new staking flow, which supposes to be released on mainnet this September. This introduces new earing lifecycle of a validator, reduce time to sync shard’s data significantly, and also help user to unstake immediately.

image

Again, checkout the technical specs here

9 Likes

Update 10 July 2020

Finish:

  • Refactor:
    • Local testing for Beacon Committee State (transaction, stake, stop auto stake)
    • Instruction unit test
    • 80% implementation Shard Committee State
  • New staking flow, swap rule
    • Agreement for new swap rule
    • 60% design for new staking flow and swap rule

Next

  • Refactor
    • Finish Implementing Shard Committee State
    • Begin local test for both Shard and Beacon Committee State
  • New Staking flow and Swap rule
  • Finish first design for implementation
  • Begin implementation
4 Likes

Update 17 July 2020

Finish:

  • Refactor:
    • Implementation Shard Refactoring
    • Merge code from dev/master-consensus-optimized to dev/dynamic-committee-size
  • New Staking flow:
    • Implementation Spec

Next

  • Refactor
    • Shard Committee local testing and fix bug
  • Review New Staking Flow implementation spec
  • Implement new staking flow
6 Likes

Update 27 July 2020

Finish:

  • Refactor:
    • Implementation Both Beacon and Shard Refactoring. Ready for QC testing
  • New Staking Flow:
    • Low Level Technical Spec (sent out later in this week)
    • Finish 20% implementation

Next

  • Refactor:
    • Testing with QC team
  • New Staking Flow:
    • Continue implementing
2 Likes

Update 3 Aug 2020

In progress:

  • Refactor:
    • Testing with QC team
  • New Staking flow: 40%
  • Unstake flow technical specification: 20% (estimated end date in this week)
3 Likes

Update 10 Aug 2020

Finish:

  • Unstake flow technical specification

In progress:

  • Refactor:
    • Testing with QC team
  • New Staking flow: 60%
  • Unstake flow: begin implementing
5 Likes

Update 17 Aug 2020

This week our team had a nice-to-have debate about design pattern and code structure. The new-staking-flow code is refactored based on our solution. It took us three days but will be rewarded.

In progress:

  • Implementing unstake
  • Refactoring code of New-staking-flow and continue to implement
  • Testing finished part with QC team
4 Likes

Update 28 Aug 2020

WONDERFUL NEWS

Finish:

  • Implementation of Unstake Feature
  • Implementation of New Swap/Assignment Rule

Let’s explain a little bit about these features. Both these features effect directly our dear end-user, especially those who run node.

  • Unstake feature is a desirable feature since our first day. It allows user to withdraw his/her stake amount whenever user’s node in candidate list, which mean it plays no role in any shards (neither committees in block creation phase nor committees in preparation phase). In short, Unstake feature tells one how long he/she must wait to get back one’s staking amount (in PRV).

  • New Swap/Assignment Rule makes Incognito’s consensus more secure and changes reward amount in short term duration.

    • The new assignment rule helps one’s node got assigned into one shard more quickly, but it must wait longer before earning something. In general, reward amount in one year might not change much. However, if you are lucky you will earn a lot more reward in the same duration compare to the current version.
    • The new swap rule allows us to development slashing feature easier.

In progres:

  • Review & Testing Unstake Feature
  • Review & Testing New Swap/Assignment Rule
9 Likes

Update 21 Sep 2020

The new swap rule (swap rule v2) which is implemented last month seem not meet our needs. So far, this month we are discussing how to implement it properly. The swap rule v2 allows beacon chain to force shard chain changing shard’s committees.

In progress:

  • Slashing node which continuously missing votes for blocks in one epoch
  • Re-design and implement swap rule v2
6 Likes

Update 25 Sep 2020

Finish:

  • Review & Unit test swap rule v2.

In Progress:

  • Implement shard block creation v2 and integrate with swap rule v2.
  • Merge Code from Swap Rule V2 and continue to implement slashing module.

With this development speed, I am confident that we will finish implementing two task above by the end of next week (2 Otc 2020). After that, team will move to testing & debugging phase. I am so excited that team found a proper design for slashing flow after a couple times of trying.

6 Likes

Update 18 Oct 2020

Finish:

  • Implementation of new Shard Block Creation V2. From now on, a new shard block is only allowed to created if it find a higher beacon block. Otherwise, shard chains will stop producing new block

  • New Swap (change committee) rule. This is a really big change to our consensus working mechanism. At first, we think that shard independently changing its committee after one epoch will make Incognito more secure. But seem that, it create more problem than helping the chain. The Shard chains and Beacon chain must synchronize too many kinds of information, which is very hard to get the same correct answer. The New Swap Rule, swap-rule-v2, forbid shard to change its committee by them self. Instead, the Beacon chain will force shard to change its committee. Less data to synchronize, less problem to tackle.

  • Slashing module by counting missing signatures from committees in 1 epoch creation. For example, if one shard creates 100 blocks in 1 epoch, each nodes must sign at least 50 blocks or it be force un-stake. This module is carried by the Beacon chain.

  • Documenting Implementation detail, integration testcase.

  • Refactoring a huge amount of code which covered by thoughtful unit-test.

In progress:

  • Review code
  • Local testing
  • Continuously defining new testcases
  • Integration

I am so glad that this feature is getting much better, clearer. It took us almost five months from researching to implementation, testing. A lot of re-design, re-work was conducted. If luck is on our side, this feature will be launched in testnet by the end of this month (October).

This is our specification link: https://docs.google.com/document/d/19KChrg0B2LmfT_t-K8vovCG-mXUkAYG88j5FQsKPXT4/edit#heading=h.k9idjn4fvxia

11 Likes

It would be useful, to me at least, that there is a process when this is rolled out that if a vNode is getting slashed there is some notification to user. Especially when this roles out, as currently I believe there may be vNodes incorrectly configured on the network, and due to the current setup the nodes are still earning even if they are missing blocks (or completely offline and unreachable). It would be nicer if when this is rolled out the nodes in committee that are not providing incorrect solutions/missing blocks were not initially penalized, but rather notified so validators could have an opportunity to fix any configuration issues they may have. Otherwise there may be quite a few validators that see drops without an understanding of why… Just something to consider.

Also,

Am I reading this correctly, that if your validator gets “slashed” it will be forced to unstake?

2 Likes

Thanks for sharing your thoughts. We will figure out a way to notify slashed vNode via out mobile application.

The current slashing rule only force Node out of committee. The staking-amount and reward which node received in that epoch will remain intact. We assume that all our users are trying to contribute to the system. When something go wrong, node get slashed, is a kind of notification to our users their nodes are not working properly

3 Likes