Dynamic committee size and dynamic sharding: implementation phase

Objective:

  1. Improve the robustness of the chain: avoid the chain blocked by the unavailability of nodes, network disconnected, inadequate node power.
  2. Adapts well with the variety of the number of validators
  3. Strengthen the security: trustless to shard committee

Estimate End Date: June 2021

July 2020

  • Beacon sync, insert and store all shard blocks
    • Beacon node must sync and insert shard from beginning
  • Refactor Committee State in Blockchain package for later implementation
  • Technical Specification for New Staking Flow

August 2020

  • Implement new staking flow

    • Staking to become candidate
    • Candidate Life cycle (not included monitoring and report)
    • Unstake flow
    • Refine Stop auto re-staking flow
  • Implement new swap rule

    • Not included increasing/decrease (case 3,4) shard
  • Technical Specification of Slashing Misbehaved Validator

September 2020

  • Integrate new Staking flow and swapping rule with with Smart Contract
  • Implement Slashing Misbehaved Validator

October 2020

  • New Staking flow and swapping rule are ready for local test (TestNet in happy case)
  • Implement, Integrate and testing Slashing Misbehaved Validator with Smart Contract and other Feature

November 2020

  • Ship New Staking flow and swapping rule at the end of Nov or early Dec
  • Ship Slashing Misbehaved Validator at the end of Nov or early Dec

March 2021

  • Reduce committee size dynamically

June 2021

  • Dynamic grow/shrink number of Shards

Key results:

  • Committee size is dynamically changed with the number of validators
  • Dynamic sharding
  • Beacon is able to verify the correctness of every single shard. As a result, a shard could verify cross-shard transactions through the beacon chain.

Note:

21 Likes

This task will begin in 1st Jun:

  1. Design working flow @hy @hungngo @0xkumi @dungtran
  2. Implementation @hy @hungngo
  3. Unittest, local testing @hy @hungngo
  4. Integration testing by QC
6 Likes

Update 2020 Jun 8:

Github link: https://github.com/incognitochain/incognito-chain/tree/dev/dynamic-committee-size
Result:

  • Finish implement Beacon sync, insert, store all shard blocks

Next:

  • Testing
6 Likes

Update 2020 Jun 12:

Result:

  • In testing phase and fixing bug

Next week:

  • Testing
  • Move to next task while waiting for QC
4 Likes

On June 24th, 2020

  • Update the random shard assignment scheme to keep the balance of the shards’ validators.
  • Update the solution for problem 2,3

Detail design: https://docs.google.com/document/d/19KChrg0B2LmfT_t-K8vovCG-mXUkAYG88j5FQsKPXT4/edit#

6 Likes

Update 26 June 2020

Finish:

  • Agreement on refactoring code approach
  • Refactor Instruction related to committee (stake, assign, swap, stop-auto stake)

Next:

  • Refactor Committee State, remain the current assign and swap rule
  • Test after finish refactoring code

This week, we do not spend much time on the main task as the code based of incognito-chain is very huge now. After thoughtfully consideration, we decide it need to be refactor into smaller component, make it easier to maintain in the future. It might take some time now, but new staking flow can be well-implemented and well-tested with high quality code.

8 Likes

Update 3 July 2020

Finish:

  • QC testing on Beacon Sync as Fullnode
  • Implement Beacon Committee State

Next:

  • Begin implementing Shard Committee State
  • Begin implementing New Staking Flow
  • Finish implementing + unit-test Beacon Committee State
6 Likes

Yesterday, we talked about dynamic committee size - new staking flow, which supposes to be released on mainnet this September. This introduces new earing lifecycle of a validator, reduce time to sync shard’s data significantly, and also help user to unstake immediately.

image

Again, checkout the technical specs here

8 Likes

Update 10 July 2020

Finish:

  • Refactor:
    • Local testing for Beacon Committee State (transaction, stake, stop auto stake)
    • Instruction unit test
    • 80% implementation Shard Committee State
  • New staking flow, swap rule
    • Agreement for new swap rule
    • 60% design for new staking flow and swap rule

Next

  • Refactor
    • Finish Implementing Shard Committee State
    • Begin local test for both Shard and Beacon Committee State
  • New Staking flow and Swap rule
  • Finish first design for implementation
  • Begin implementation
4 Likes

Update 17 July 2020

Finish:

  • Refactor:
    • Implementation Shard Refactoring
    • Merge code from dev/master-consensus-optimized to dev/dynamic-committee-size
  • New Staking flow:
    • Implementation Spec

Next

  • Refactor
    • Shard Committee local testing and fix bug
  • Review New Staking Flow implementation spec
  • Implement new staking flow
6 Likes

Update 27 July 2020

Finish:

  • Refactor:
    • Implementation Both Beacon and Shard Refactoring. Ready for QC testing
  • New Staking Flow:
    • Low Level Technical Spec (sent out later in this week)
    • Finish 20% implementation

Next

  • Refactor:
    • Testing with QC team
  • New Staking Flow:
    • Continue implementing
1 Like

Update 3 Aug 2020

In progress:

  • Refactor:
    • Testing with QC team
  • New Staking flow: 40%
  • Unstake flow technical specification: 20% (estimated end date in this week)
2 Likes

Update 10 Aug 2020

Finish:

  • Unstake flow technical specification

In progress:

  • Refactor:
    • Testing with QC team
  • New Staking flow: 60%
  • Unstake flow: begin implementing
4 Likes

Update 17 Aug 2020

This week our team had a nice-to-have debate about design pattern and code structure. The new-staking-flow code is refactored based on our solution. It took us three days but will be rewarded.

In progress:

  • Implementing unstake
  • Refactoring code of New-staking-flow and continue to implement
  • Testing finished part with QC team
3 Likes

Update 28 Aug 2020

WONDERFUL NEWS

Finish:

  • Implementation of Unstake Feature
  • Implementation of New Swap/Assignment Rule

Let’s explain a little bit about these features. Both these features effect directly our dear end-user, especially those who run node.

  • Unstake feature is a desirable feature since our first day. It allows user to withdraw his/her stake amount whenever user’s node in candidate list, which mean it plays no role in any shards (neither committees in block creation phase nor committees in preparation phase). In short, Unstake feature tells one how long he/she must wait to get back one’s staking amount (in PRV).

  • New Swap/Assignment Rule makes Incognito’s consensus more secure and changes reward amount in short term duration.

    • The new assignment rule helps one’s node got assigned into one shard more quickly, but it must wait longer before earning something. In general, reward amount in one year might not change much. However, if you are lucky you will earn a lot more reward in the same duration compare to the current version.
    • The new swap rule allows us to development slashing feature easier.

In progres:

  • Review & Testing Unstake Feature
  • Review & Testing New Swap/Assignment Rule
7 Likes

Update 21 Sep 2020

The new swap rule (swap rule v2) which is implemented last month seem not meet our needs. So far, this month we are discussing how to implement it properly. The swap rule v2 allows beacon chain to force shard chain changing shard’s committees.

In progress:

  • Slashing node which continuously missing votes for blocks in one epoch
  • Re-design and implement swap rule v2
5 Likes

Update 25 Sep 2020

Finish:

  • Review & Unit test swap rule v2.

In Progress:

  • Implement shard block creation v2 and integrate with swap rule v2.
  • Merge Code from Swap Rule V2 and continue to implement slashing module.

With this development speed, I am confident that we will finish implementing two task above by the end of next week (2 Otc 2020). After that, team will move to testing & debugging phase. I am so excited that team found a proper design for slashing flow after a couple times of trying.

4 Likes