[Shipped] NAT traversal solution for Incognito

I think this is an important feature, and that might worth it. So far, In our infrastructure, HW is the weakest component. Anything makes HW harmful, the message can not communicate from node to node and the chain should be stopped. I see that someone user also consider about it like here. And NAT traversal solution is the way to find out better p2p solution for our network.

I think this is quite difficult to evaluate. Although the Highway may be weak, it can be overcome by expanding the Highway network.

Can you sketch the network architecture when applying Highway and P2P in parallel so that we can easily visualize what you want to do?

Expanding Highway as Centralized system is a economic issue for core team or when core-team is not survive, how to maintain this highway is hard to answer. And how is enough for expanding Highway network.

2 Likes

This should be how the network looks like in an ideal scenario. These Incognito nodes are assumed to be behind NAT.

Untitled Diagram

NAT-T: NAT traversal methods

4 Likes

@lam great post and very comprehensive explanation. HW is now centralize. A plan for decentralized network and a good topology is desirable. This proposal is very much worth doing.
My question is the same as @hyng as well

  1. How HW and P2P work together? It’s would be complicated, and need carefully integration.
  2. BFT network cost is O(N^2), it’s a matter if Incognito consider increase committee size from 32 to 300. So plz take this into consideration in research and implementation phase, it could help take on network complexity problem if needed
  3. Could you provide some information about how Ethereum/Bitcoin (big old blockchain) and Harmony/Near (new sharding blockchain) nodes communicate with each others. Do they encounter this problem as well? Especially Harmony or Near?
3 Likes

@ruler great questions.

  1. “How HW and P2P work together?” Yeah, as you said it would be complicated. As of the time when writing this reply we still exploring possible approaches to this matter, we will post an update as soon as we have something concrete.

  2. “BFT network cost is O(N^2),…” thank you for raising this concern, the increase in committee size is in the future plan of Incognito so we will take this into consideration.

  3. Ethereum & Bitcoin are depending on big mining pools which have setup large infrastructure of public IP nodes so they didn’t really have to worry much about this kind of problem. Harmony documentation is only about hosting validator node on the cloud, no user-own device is mentioned. As the case for Near, they use something like a TURN/tunneling method, the performance of method depends on how good their routing algorithm is if all devices are behind NAT.

So yes, public blockchain networks do encounter this problem as well but depending on which kind of community they want to build this issue will have big or small impact.

Thanks @lam for your information. As you mentioned, big blockchain doesn’t encounter this problem, they use public IP. Private IP or IP behind NAT it a hard problem to tackle, so maybe the bypass it. If we continue to take on this problem it would be very hard. What if we accomplish NAT traversal solution then we find out in reality it doesn’t work for private IP node (slow message transmission, small bandwidth)?
Does your solution work for public IP node? If it does, then we can use p2p for public ip then highway for private ip.

1 Like

This is a proposal for NAT Traversal and public IP nodes don’t use NAT so they can just connect directly to each other easily. We have a lot of users using Incognito node in the home, this solution mainly for these devices, if we depend on Highway for these devices which will be many, we have to build and maintain a lot of Highway servers as they will need to grow when more and more devices join the network.

I believe that reserve Highway for nodes that are in committee and use p2p for other kinds of nodes is a more cost-effective and simpler solution. Plus it will be more decentralized.

1 Like

Here is a summary of all what we have discussed above and what we have researched this week:

  1. We are not switching to fully using P2P connection, both Highway and NAT traversal methods will be used. Highway will be reserved for committee nodes, other nodes will use NAT traversal for communication.

  2. As researched, other blockchain networks are encountering the same problem when it comes to p2p, but the impact on their network is different.

  3. There are several methods used by other software to overcome this problem, some require the user to manually set up and some don’t.

  4. We are still exploring solutions that not require the user to manually change their router setting to allow inbound connection to a device behind NAT.

  5. We currently try to simulate a network of devices behind different types of NATs.

2 Likes

Integrating Tor/I2P through something like Kovri would be great for P2P communications without revealing IPs. There’s a few libp2p plugins for i2p that could be used.

4 Likes

That’s a nice idea but Tor/I2P is slow as it has to traverse many nodes and because of the current way of how node in Incognito network must communicate, using Tor/I2P will slow the network way down.
In the meantime, we have our focus on reducing dependency on Highway. We will explore solutions that hide user device’s IP address in a future proposal.

2 Likes

Weekly update May 4 -> May 8:

  • Coding for simulation is nearly done. Most features are complete such as NATPCP/Upnp protocol, detecting node NAT type.
  • We have setupe testing environment.
  • Next week, we will start testing some scenarios and fix bugs / adding improvements along the way.
1 Like

Weekly update May 11 -> May 15:

  • Due to the need for a more complex test, we have switched to using iptables alongside pfsense.

  • Continues coding for the NAT traversal test library.

1 Like

Weekly update May 18 -> May 22:
Achievement:

  • We have successfully simulated and traversal different cases of NAT types. Our current implementation of traversal methods is working as intended.
  • Here a draft of how a node can get p2p connectivity:
    nat-2

Next week:

  • We will test relay mode for cases that node can’t be connected via direct connection or hole-punching method. The objective of this test is to see whether relay mode is a viable option when Highway isn’t available.
2 Likes

Over the past month, we have researched and developed a solution to NAT traversal for nodes and decided the role of p2p and Highway in Incognito network.

Here an updated network communication logic after we have tested relay mode:
nat-3

P2P and Highway strategy:

  • Highway will be use mainly for blocks/data syncing.
  • P2P will be use for committee members to communicate consensus.
    Copy of Untitled Diagram-2

This is the final update of this proposal. For future updates and development related to this proposal please follow the linked proposal in the main post here.

2 Likes

After reading this discussion on network topology, I learned a lot. But as a non-expert, I was left with this one question after reading all of the above:

Hypothetically should the Highway fail (for any reason), would the Pnodes and Vnodes already synced and in committee be able to continue P2P functioning (achieve consensus/produce blocks) for the network?

I want to make sure I understand the sentence “We are not switching to fully using P2P connection, both Highway and NAT traversal methods will be used.” Using P2P and Highway in parallel could mean one still functions partially without the other (albeit less efficiently), but using P2P AND Highway interdependently would mean if one fails so does the other. Could you clarify this for a not so technically savvy interested reader?

2 Likes

Hi Northhill,
Our goal with this proposal is to make the committee to achieve consensus with and without Highway but currently Pnodes and Vnodes are not using P2P connection so the committee will not able to produce blocks without Highway.

There is a follow-up proposal about network topology for efficiently using P2P and Highway in parallel but we are temporally postponed it to focus on other more important topics first.

2 Likes

thanks @lam for the answer, just wanted to add one more point to this. Currently, we’re using Highway (as a proxy) for communication between nodes in the network, but when looking into our Roadmap we have also plan to research a new topology to be able to make the network more stable and resilient.

3 Likes

Thank you both @lam @duc for the responses. Just out of curiosity, I happen to be reading about Caradano Staking pools and I noticed the picture of their network topology here. This is not the complete topology I assume, but
cardanostakepoolsetup
The function of the Cardano relay nodes in relation to Core nodes seems very similar to the function of the Highway to pnodes in Incognito. Are incognito and cardano using similar network topologies?

2 Likes

Yes, we are similar in some ways.

See here for more detail about how the current Incognito network works Highway topology

1 Like