The scenario of a government setting up a Node and being able to track people’s IP-addresses was mentioned. I guess most people want to stay incognito.
oh, about that, hate to say this but you can’t stay incognito with your ISP hence your government. The IP addresses of Highway is public so the government technically can track whether you connect to the Incognito network or not.
We currently thinking of some solutions to stay incognito on the network but that is for another proposal in the future.
It is fine, it wasn’t my worry. It is probably good that it is documented now, for people who feel weary about it.
If you switch to fully P2P connection, the number of messages that each validator must receive during the BFT consensus will be really large (about O(N^2) message with N is the size of the committee), this makes it difficult for the network to achieve scalability. Do you have a solution to this problem?
We not actually switching to full p2p, Highway still have an important role. Chain committee will have priority access to Highway to communicate with each others. This solution is aiming at node that isn’t in chain committee and as a backup plan for scenarios where Highway isn’t working or got blocked.
BFT-Vote message size is relatively small so that shouldn’t be a problem. BFT-Propose message will likely to be slower when using this solution than Highway.
I think this is an important feature, and that might worth it. So far, In our infrastructure, HW is the weakest component. Anything makes HW harmful, the message can not communicate from node to node and the chain should be stopped. I see that someone user also consider about it like here. And NAT traversal solution is the way to find out better p2p solution for our network.
I think this is quite difficult to evaluate. Although the Highway may be weak, it can be overcome by expanding the Highway network.
Can you sketch the network architecture when applying Highway and P2P in parallel so that we can easily visualize what you want to do?
Expanding Highway as Centralized system is a economic issue for core team or when core-team is not survive, how to maintain this highway is hard to answer. And how is enough for expanding Highway network.
This should be how the network looks like in an ideal scenario. These Incognito nodes are assumed to be behind NAT.
NAT-T: NAT traversal methods
@lam great post and very comprehensive explanation. HW is now centralize. A plan for decentralized network and a good topology is desirable. This proposal is very much worth doing.
My question is the same as @hyng as well
- How HW and P2P work together? It’s would be complicated, and need carefully integration.
- BFT network cost is O(N^2), it’s a matter if Incognito consider increase committee size from 32 to 300. So plz take this into consideration in research and implementation phase, it could help take on network complexity problem if needed
- Could you provide some information about how Ethereum/Bitcoin (big old blockchain) and Harmony/Near (new sharding blockchain) nodes communicate with each others. Do they encounter this problem as well? Especially Harmony or Near?
@ruler great questions.
“How HW and P2P work together?” Yeah, as you said it would be complicated. As of the time when writing this reply we still exploring possible approaches to this matter, we will post an update as soon as we have something concrete.
“BFT network cost is O(N^2),…” thank you for raising this concern, the increase in committee size is in the future plan of Incognito so we will take this into consideration.
Ethereum & Bitcoin are depending on big mining pools which have setup large infrastructure of public IP nodes so they didn’t really have to worry much about this kind of problem. Harmony documentation is only about hosting validator node on the cloud, no user-own device is mentioned. As the case for Near, they use something like a TURN/tunneling method, the performance of method depends on how good their routing algorithm is if all devices are behind NAT.
So yes, public blockchain networks do encounter this problem as well but depending on which kind of community they want to build this issue will have big or small impact.
Thanks @lam for your information. As you mentioned, big blockchain doesn’t encounter this problem, they use public IP. Private IP or IP behind NAT it a hard problem to tackle, so maybe the bypass it. If we continue to take on this problem it would be very hard. What if we accomplish NAT traversal solution then we find out in reality it doesn’t work for private IP node (slow message transmission, small bandwidth)?
Does your solution work for public IP node? If it does, then we can use p2p for public ip then highway for private ip.
This is a proposal for NAT Traversal and public IP nodes don’t use NAT so they can just connect directly to each other easily. We have a lot of users using Incognito node in the home, this solution mainly for these devices, if we depend on Highway for these devices which will be many, we have to build and maintain a lot of Highway servers as they will need to grow when more and more devices join the network.
I believe that reserve Highway for nodes that are in committee and use p2p for other kinds of nodes is a more cost-effective and simpler solution. Plus it will be more decentralized.
Here is a summary of all what we have discussed above and what we have researched this week:
We are not switching to fully using P2P connection, both Highway and NAT traversal methods will be used. Highway will be reserved for committee nodes, other nodes will use NAT traversal for communication.
As researched, other blockchain networks are encountering the same problem when it comes to p2p, but the impact on their network is different.
There are several methods used by other software to overcome this problem, some require the user to manually set up and some don’t.
We are still exploring solutions that not require the user to manually change their router setting to allow inbound connection to a device behind NAT.
We currently try to simulate a network of devices behind different types of NATs.
Integrating Tor/I2P through something like Kovri would be great for P2P communications without revealing IPs. There’s a few libp2p plugins for i2p that could be used.
That’s a nice idea but Tor/I2P is slow as it has to traverse many nodes and because of the current way of how node in Incognito network must communicate, using Tor/I2P will slow the network way down.
In the meantime, we have our focus on reducing dependency on Highway. We will explore solutions that hide user device’s IP address in a future proposal.
Weekly update May 4 -> May 8:
- Coding for simulation is nearly done. Most features are complete such as NATPCP/Upnp protocol, detecting node NAT type.
- We have setupe testing environment.
- Next week, we will start testing some scenarios and fix bugs / adding improvements along the way.
Weekly update May 11 -> May 15:
Due to the need for a more complex test, we have switched to using iptables alongside pfsense.
Continues coding for the NAT traversal test library.
Weekly update May 18 -> May 22:
- We have successfully simulated and traversal different cases of NAT types. Our current implementation of traversal methods is working as intended.
- Here a draft of how a node can get p2p connectivity:
- We will test relay mode for cases that node can’t be connected via direct connection or hole-punching method. The objective of this test is to see whether relay mode is a viable option when Highway isn’t available.
Over the past month, we have researched and developed a solution to NAT traversal for nodes and decided the role of p2p and Highway in Incognito network.
Here an updated network communication logic after we have tested relay mode:
P2P and Highway strategy:
- Highway will be use mainly for blocks/data syncing.
- P2P will be use for committee members to communicate consensus.
This is the final update of this proposal. For future updates and development related to this proposal please follow the linked proposal in the main post here.