Incognito App service

Objectives

  • Deploy all parts of the App service onto the production environment by the end of Jan, 2021.
  • Make it ready for the Incognito app team to start working on the transition from the current full-node to the new one.
  • Ability to extend easily to the future requirements if needed.

Timeline

Milestone 1: November 13rd, 2020

Finalize architecture, data models, and technologies for the new App service. The output of the milestone is to have a document for three things listed above.

Milestone 2: December 15th, 2020

Implement a full data flow (chain → full-node → workers → data storage) in a test environment.

Milestone 3: December 30th, 2020

Measure metrics for:

  • Latency - how delay (or how fast) data synchronization is for a newly created block’s data (as compared to the current full-node)

  • Key or heavy computations to see how efficient the new data models are.

Milestone 4: January 31st, 2021

Implement and expose sufficient APIs for Incognito wallet’s needs along with good documentation for reference.

Key results

  • A reliable, scalable, and handy app service in aspects of data synchronization and data organization.
  • A friendly and well-documented Restful APIs layer to serve both internal party (Incognito app team, first) and external parties (community builders, later)

Notes

  • Link to Github (TBD)
  • Proposing architecture

12 Likes

UPDATE 30 November

Milestone 1

Update 5 Nov - 16 Nov

  • Finish first draft of physical data model
  • In progress: parsing finalized blockchain data
4 Likes

UPDATE 30 November

Milestone 1

Finish:

  • MVP architecture: instead of using Apache Kafka, the team decided. to implement Incognito App Services as Go-routines in Incognito-Chain codes. This branch will be maintained separated. The only thing we need to test before put into into production is Database and our Physical Model.

Next

  • Load test and choose database. The team isn’t sure which database is the best, we are considering some options, such as: mongodb, cassandra, gg big table.

Milestone 2
Finish:

  • Database Logical Model
  • Data parser package: parse Incognito Chain data into Database Logical Model
  • Mongodb physical model & Mongodb physical layer implementation. The team will give the first try at Mongodb.

Next:

  • Load test our physical model on mongodb.
  • Implement API services layer.
4 Likes

UPDATE 13 Dec 2020

Milestone 1:

We archived this milestone with an MVP architecture and a large scale architecture. Both architecture will fully support business rules, the differences lie on scale of App Services users, Incognito Chain block/second, Durability

Milestone 2:

The team finished choosing technology and MVP code skeleton in all layers. We also write some api to test how the MVP works. But the hardest things are how to make App Service provides better APIs than current RPC.
Turn out, we can’t finalized the physical data model on Mongodb or our Code Base just based on the imagination about how system should work. We have real data on Mainnet. So we decide to sync Mainnet/Testnet data for Data Model and Code Base improvement.

For that reason, I would like to close this milestone here. Take the leftover to the next milestone.

Milestone 3:
This milestone now will have Data Model and Code Base improvement task. Mr @duc please add this task to milestone 3, thanks.

In progress:

  • Sync Testnet and continuous improvement, finished APIs:
    • getBlockCount
    • getBlockHash
    • getBlocks
    • getBlockHeader
    • retrieveBlock
    • retrieveBlockByHeight
    • listOutputCoins
    • getTransactionByHash

Community could keep track our progress via this link: https://docs.google.com/spreadsheets/d/1sKFGfaNIXEwl2EbGfcf3Z8PZVeMjxjdsWEsguViu8no/edit?usp=sharing

6 Likes

UPDATE 28 DEC 2020
Milestone 3:

Finish:

  • WONDERFUL NEWS, the benchmark show result of the slowest API listoutputcoin + decrypt output coin:
    • Number of request per second: 144.73.
    • 95% of request return api value in 860ms. Far more better than current system
  • We finish about 80% of APIs list, only short of PDE and Portal APIs.

Next:

  • Improve performance of current APIs
  • Finish Portal and PDE APIs
8 Likes

Awesome news @duc…thank you for the update… :sunglasses:

3 Likes