Recovery of a Peer Node in Hyperledger Fabric

Overview

This article is to show the recovery process of a peer node. For simulation, we will first bring down a running peer node, and resume it to a fully functioning node. Through this process we can understand more how various parts in Hyperledger Fabric are working, including channel, ledger, chaincode, etc.

Test Setup

We only use a fabric node, with proper fabric container images, fabric binary tools and fabric samples. You can refer to this article how to prepare such a fabric node (link).

We are using the Fabcar example provided in fabric-samples. It is done through running the setup script.

Upon completion we have a fabric network (First Network) with Fabcar chaincode deployed. Here is a summary of this setup.

  • One orderer
  • Two organizations (Org1 and Org2)
  • Each organization comes with a Certificate Authority and two Peer Nodes (designated as peer0 and peer1). Therefore total four peer nodes are running.
  • Each peer node comes with a couchdb as the world state portion of the ledger.
  • A channel mychannel is created and joint by all peers.
  • Fabcar chaincode is installed in all four peer nodes.
  • Fabcar chaincode is instantiated in mychannel. The endorsing poilcy requires endorsement from one member from each organization.
  • The chaincode function initLedger() is invoked, and there are 10 sets of car record in the ledger

As usual, it is more convenient to open up various terminals for each peer. Use these commands to open various terminals for different peer nodes.

Here is the setup we will use during the demonstration (only the four peer nodes are shown).

To verify things are working well, we will do the checking on ALL peer nodes. We will get the same result.

And we can take a look on the world state in each node. The couchdb can be observed through a browser (the port is 5984, 6984, 7984 and 8984, corresponding to peer0.org1, peer1.org1, peer0.org2 and peer1.org2, respectively)

Now the network is ready, and we can start our observation when a peer node is down.

Shutdown a Peer Node

To shutdown peer0.org2.example.com completely, we will use the following docker commands.

We can check docker ps and see those containers are gone. The network looks like this.

The network is still functioning, in a sense that chaincode invoke and query can still be done across other peer nodes. We can perform query on the running peer nodes and we still get back the data stored in the ledger.

For invoking chaincode function, remember that the endorsing policy requires a member from each organization. We now cannot send request to peer0.org2.example.com as it is down already.

But we can still reach peer1.org2.example.com as it is also a member of Org2.

A quick summary: as far as we still have a peer node in the organization, the endorsing policy is still met and the fabric network is still working.

In the next part we will take a look what happens when we bring up this peer node back, and how we can resume the process.

Recover this Peer Node into Full Service

To recover this node back to a functioning peer, here are three steps.

  1. Bring up the peer node (container) and the world state database (couchdb)
  2. Join it back to channel
  3. Install chaincode

Let’s observe what happens in each step.

1. Bring Up the Peer Node and CouchDB

In this setup, the bringing up the node and the couchdb through docker compose files.

After we see them back as running containers, we can reach peer0.org2.example.com through the CLI terminal. Let’s first see if the channel is already there.

We see peer0.org2.example.com is not part of mychannel yet. The next move is to join it back to mychannel. We can also see that the couchdb is empty on this peer.

This is what happens after bringing up the peer node. It is not useable yet.

2. Join the Peer Node to mychannel

To make the peer node holding the same ledger as other peer nodes in the same channel, we do not need to copy the whole blockchain and world state. Instead we can simply join the peer node to the channel and the ledger will be “sync-ed” to the node.

All we need is just the channel genesis block for that channel. In our environment this file mychannel.block is already in the CLI directory. In other case, you may get it directly from existing nodes.

This is the command to join the channel with the channel genesis block.

We see now peer0.org2.example.com now joined mychannel.

If we take a look on its blockchain information, we see the full blockchain is already there. That means the remaining blocks are obtained from other peers.

We can compare the result with other peer node: for example, it is taken from peer1.org1.example.com.

If we quickly check the world state, we see the state is also updated.

This means that, after joining the channel, this peer node can get back all in the ledger, both the blockchain and world state.

As we have the ledger now, can we query or invoke chaincode function? Unfortunately we cannot yet.

It is because the chaincode is not installed yet back to the peer node. We will do this in next step.

3. Install Chaincode

If we check the chaincode information on this node, we can see the chaincode is not installed yet, though the chaincode is already instantiated in mychannel, in which this peer is part of it.

Now we install the chaincode back to peer0.org2.example.com.

We can check again and now the chaincode is properly installed.

This is how it looks like after chaincode is installed back to the node.

And we can query the chaincode from this node now.

Back to Normal

At this moment, we have completely brought up the peer0.org2.example.com node back to normal, serving as a full-function peer node in our network. We can try it again by invoking a chaincode function and pointing to peer0.org2.example.com as an endorsing peer.

We do it again on the terminal CLI peer0.org1.example.com, as what we did before in previous session.

We see the node peer0.org2.example.com successfully perform endorsement for this proposal.

Observation

When the peer node is “physically” brought up and not joining the channel yet, it can do nothing in the fabric network. In our case, after we brought up the peer node using docker-compose, the peer node is not yet part of the network.

After the peer node joins the channel, it gets back the whole ledger from the network. Now the peer node is in the same page as other peer nodes in the fabric network. We see both the blockchain and the world state are sync-ed after joining the channel.

Having the same ledger content does not mean that the node can process chaincode invoke or query. What we need is to install the proper chaincode back to this peer. After chaincode installation, the peer node can handle chaincode query and perform the endorsing process. At this stage we can say this peer node is fully functioning.

The steps mentioned above are only applicable when this peer node is already configured in the fabric network, i.e., it is already defined in the fabric network. In our case this peer node is in the network at the very beginning. Therefore it is not a brand-new peer installation.

In case it is a brand-new peer node, for example, adding peer2.org2.example.com into the fabric network, the process is different. We need to have crypto material for this new peer and add its information to the ledger first before we can join the peer to the fabric network. The process is comparable to “adding an organization on a fabric network”. You can take a look on an explanation about this process (article).

Clean Up

As good practice, always clean up everything after demonstration.

Hope this article helps you understand more about Hyperledger Fabric.

Written by

Happy to share what I learn on blockchain. Visit http://www.ledgertech.biz/kcarticles.html for my works. or reach me on https://www.linkedin.com/in/ktam1/.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store