Many people in the cryptocurrency space have heard the term “XPub” but if you look into the tech, it can be pretty daunting. At TokenSoft, we not only help clients set up and manage wallets to hold digital assets, but we also help them understand how it works. One of the things that can be confusing is the term XPub, so here are some notes to help people understand what it is and how it is used.
At the heart of many blockchain-based protocols is Public Key Cryptography. This is how accounts are secured by proving you have a Private Key that can digitally sign a message corresponding to a Public Key. An analogy that can be used is that the Public Key is like your bank account number where people can send funds and your Private Key is like a physical signature that can authorize a paper check to pay money out of your account.
When Bitcoin was first created, the wallet software would create a new public/private key pair every time you wanted to generate an address for someone to send you BTC. This meant that your wallet database was a giant list of keys for every transaction ever performed in your wallet. Backing this up meant keeping track of every key pair and introduced a lot of room for error.
I am not sure who originally proposed it, but some of the Bitcoin developers worked together to created BIP32, which defined how a single secret seed could be used to generate child key pairs to be used for this same use case. Instead of having to keep a list of all the key pairs, you could just create a single seed value that is capable of generating a tree of key pairs. This seed value is just a really, really large random number that no one else could ever guess. Then, when an entire wallet needed to be backed up, you could just save off this one value and recreate all the generated key pairs when the wallet was restored from backup.
Every key pair generated can define an XPub and an XPriv key that can be used to generate the public and private keys and then also derive any child keys. The XPub can only generate child public keys but the XPriv can generate any child public and private keys. In the diagram above, you can see that a single key called the “Master Node” can derive everything else.
In addition to making the backup process much easier for a wallet, it also created some other interesting features. One interesting use is that you could define a path down the tree of key pairs to have a different meaning. A path to the key that starts with 1 might be reserved for Bitcoin and a path that starts with 2 might be reserved for Ethereum. This would allow a single seed to be used across many different blockchains (have you ever wondered how a single Ledger hardware wallet can sign across all the different chains with a single seed phrase?). The actual spec is defined in BIP32 and BIP44. You can see the IDs that all the blockchains have tried to claim here… my favorite is QuarkChain which uses 99999999 as the ID (numbers 123 or 150 would just be too boring).
Additionally, another interesting feature is that since every node in the tree of key pairs generates an XPub that can generate all child public keys, you can create a way to “view” a read-only wallet. The XPub generates all the children public addresses and checks to see if funds had been paid to that address. This is very important for monitoring for incoming payments in a wallet, for letting auditors see into your activities, and letting others generate addresses on your behalf without the ability to steal any funds. Cool stuff.
So now that we know what an XPub is and what it is used for, let's get a little deeper. When you see an XPub like:
…you can understand that there is a lot of data encoded in there. It includes a network id, like “xpub” or “tpub”, which helps the wallet determine what blockchain it should be used on, the depth of where it is in the chain, a fingerprint to verify the parent key, the child index, the chain code which is extra data for key derivation, and the actual public or private key. All that data is encoded into your XPub and allows an entire wallet to be created from this small piece of data.
In the diagram above, we can see that the XPub string is decoded to raw bytes, and the BIP32 definition determines which pieces of data are used for different reasons.
Since 4 bytes at the beginning of the XPub determine the version, it can be used to determine what type of key is encoded. Each different version encoded into base58 will turn into a different string that humans can differentiate.
<pre><code>> bs58check.encode(Buffer.from('0488b21e...', 'hex'))</code></pre>
<pre><code>> bs58check.encode(Buffer.from('043587CF...', 'hex'))</code></pre>
<pre><code>> bs58check.encode(Buffer.from('04b24746...', 'hex'))</code></pre>
<pre><code>> bs58check.encode(Buffer.from('049d7cb2...', 'hex'))</code></pre>
The best reference for the use of each of the versions of HD wallets can be found at Satoshi’s Labs Improvement Proposal. In this doc, you can see all of the different HD wallet versions and their use case.
The most widely used scenario for different version codes, other than the standard Bitcoin version, is to designate a SegWit wallet. When creating a new wallet, importing an XPub could cause confusion whether the addresses derived should be SegWit or not. To solve this, some BIPs were created to define different network IDs for native SegWit wallets (ZPub) and backward compatible SegWit wallets (YPub). Also, if you ever see a TPub, this means Bitcoin Testnet and RPub means Bitcoin Regtest. Again, these different key prefixes are only used to inform your wallet the purpose of this key and the network it should be used on. However, one caveat is that not all wallets implement these features, and many wallets will assume native SegWit or standard bitcoin addresses since they do not support any other types.
As you can see, the HD wallets and how they are tracked and shared using XPubs can be pretty complicated. At the root of these features though, is simplifying wallet management and making it easier to be portable across all the different wallet software in the Bitcoin ecosystem.