So, some thoughts on the Binance hack. Binance is the largest user of the Cosmos software. They run a $multibillion platform without meaningful contributions or engagement with the core software. There’s a lot to learn from what happened here.
You’ve probably seen @samczsun’s excellent tweet thread demonstrating the issue. Let’s try to fill in some more details about the situation.
An official advisory and defensive patch have been published here: forum.cosmos.network
Friendly reminder, if you find a potential vulnerability in Cosmos software, please follow our responsible disclosure process: github.com
Friendly reminder, if you find a potential vulnerability in Cosmos software, please follow our responsible disclosure process: github.com
The crux of the problem is that a hacker was able to forge a merkle proof. This is not supposed to be possible - merkle proofs are supposed to provide high integrity. Blockchain light clients (and IBC) are built on merkle proofs, so it’s important to get them right.
A merkle proof is a cryptographic proof that some key-value pair exists in a data store. We might call it a “proof of inclusion”. Many blockchains store their data in a merkle tree so that proofs can be produced proving some piece of data is included in the tree.
Merkle proofs are used heavily in IBC so that e.g one blockchain can prove it has a packet destined for another. Of course, it would be a big problem if you could prove that some piece of data is in the tree but it actually isn't. This is basically what happened with Binance.
Cosmos chains use a kind of Merkle tree called IAVL. It lives in the IAVL repo. It comes with a poem about how great Merkle trees are. IAVL is a custom merklized balanced binary search tree. It’s the Cosmos equivalent of Ethereum’s patricia trie
github.com
github.com
Every blockchain developer is at some point obliged to descend into Merkle tree madness as they come to terms with the architecture and algorithms of these structures. It’s ok, form support groups.
The IAVL repo exposes an API for constructing and verifying proofs using a “RangeProof” object. A range proof is a proof that some range of keys exist side-by-side in a merkle tree.
A range proof can also be used to prove a single key-value pair (range of size 1), or to prove that a key is not in the tree (range of size 2 with keys on either side of the absent one).
The IAVL repo uses the RangeProof object for all three kinds of proofs (inclusion, absence, keys-in-range). But it turns out there’s a serious bug in the RangeProof’s inner workings.
A proof should consist of a leaf node and a series of inner nodes which outline a path through the tree, from leaf to root, with enough information to compute the tree’s merkle root hash and verify the leaf is actually part of the tree.
Since this is a binary tree, each inner node can have a Left & Right branch. But in a proof, you’re tracing a path through the tree, so inner nodes should only contain *either* their Left or Right branch hash. The other is constructed from hashes of other nodes in the proof.
This is where the IAVL RangeProof’s code got into trouble. The IAVL RangeProof allows both the Left and Right field in the InnerNode to be populated. This should never happen.
An attacker basically took advantage of being able to stick information into the Right field that never got validated and never impacted the hash computation in order to make the verifier believe some leaf was part of the tree. Hence they succeeded in forging a merkle proof.
Notably, the issue depends on the attacker being able to add leaves to a single proof, as RangeProof allows you to prove multiple leaves at a time. So even if your protocol only expects to prove one key at a time, using RangeProof opens up surface area for an attacker.
So it’s not a good idea to use RangeProof. But we can also issue a simple defense - reject proofs up front if any inner nodes have both Left and Right fields populated. That should stop this issue.
While RangeProof is part of a core Cosmos repo (IAVL), it’s not actually used in blockchain protocols within the Cosmos stack. The IAVL tree is itself used by all Cosmos-SDK chains, but the RangeProofs are not. This is key to understand!
Instead, for merkle proofs within IBC, a new specification was developed following more rigorous processes set by the IBC standards. This spec is called ICS23. It lives in the IBC spec repo github.com and its implemented in github.com
What’s ICS23? It’s a common standard for merkle proofs that supports multiple kinds of merkle trees, including IAVL trees. ICS23 defines a general purpose format for serializing and verifying merkle proofs.
Rather than use the IAVL tree’s built-in RangeProof system, IBC uses the ICS23 standard to produce and verify merkle proofs from IAVL trees. And the ICS23 code does not have this bug.
This isn’t just a matter of using a different piece of code and getting lucky. This represents a more fundamentally different approach to software engineering.
ICS23 followed a more rigorous design process intended to minimize surface area while still being general purpose - a difficult task! As part of this, it explicitly *rejected* range proofs. There are no range proofs in ICS23.
So the very object of the exploit is itself not even admissible in the ICS23 spec. Good. IBC’s goal is to make cross-chain communication safer.
Of course, the IBC specifications and protocol may not be perfect and will continue to be improved ; as a complex protocol and software implementation, it may even have undiscovered vulnerabilities our communities will have to respond to. Security takes a community.
We must all take security seriously. If you find a potential vulnerability, please disclose it responsibly: github.com. And if you can contribute to improving the software and protocols, we invite you to do so!
Overall, this incident is an opportunity to remind everyone of the importance of strong security practices in the software development lifecycle, to spread some awareness of what IBC is and how it works, and to invite the entire ecosystem to help improve IBC.
Bridge hacks are a real problem for our industry and they won’t get better without serious commitment to higher security and standards processes. Let IBC be a shining example of what that can look like.
There is also an important lesson here about using open-source software: follow best practices, stay up-to-date, and contribute resources upstream! Would be great to see @binance become a more responsible and collaborative ecosystem participant!
Loading suggestions...