Recovery of a Peer Node in Hyperledger Fabric

Overview

This article is to show the recovery process of a peer node. For simulation, we will first bring down a running peer node, and resume it to a fully functioning node. Through this process we can understand more how various parts in Hyperledger Fabric are working, including channel, ledger, chaincode, etc.

Test Setup

We only use a fabric node, with proper fabric container images, fabric binary tools and fabric samples. You can refer to this article how to prepare such a fabric node (link).

We are using the Fabcar example provided in fabric-samples. It is done through running the setup script.

cd fabric-samples/fabcar
./startFabric.sh

Upon completion we have a fabric network (First Network) with Fabcar chaincode deployed. Here is a summary of this setup.

  • One orderer
  • Two organizations (Org1 and Org2)
  • Each organization comes with a Certificate Authority and two Peer Nodes (designated as peer0 and peer1). Therefore total four peer nodes are running.
  • Each peer node comes with a couchdb as the world state portion of the ledger.
  • A channel mychannel is created and joint by all peers.
  • Fabcar chaincode is installed in all four peer nodes.
  • Fabcar chaincode is instantiated in mychannel. The endorsing poilcy requires endorsement from one member from each organization.
  • The chaincode function initLedger() is invoked, and there are 10 sets of car record in the ledger

As usual, it is more convenient to open up various terminals for each peer. Use these commands to open various terminals for different peer nodes.

# for peer0.org1.example.com
docker exec -it cli bash
# for peer1.org1.example.com
docker exec -it -e CORE_PEER_ADDRESS=peer1.org1.example.com:8051 -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/tls/ca.crt cli bash
# for peer0.org2.example.com
docker exec -it -e CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp -e CORE_PEER_ADDRESS=peer0.org2.example.com:9051 -e CORE_PEER_LOCALMSPID="Org2MSP" -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt cli bash
# for peer1.org2.example.com
docker exec -it -e CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp -e CORE_PEER_ADDRESS=peer1.org2.example.com:10051 -e CORE_PEER_LOCALMSPID="Org2MSP" -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/tls/ca.crt cli bash

Here is the setup we will use during the demonstration (only the four peer nodes are shown).

Only the peer nodes are shown in the test setup.

To verify things are working well, we will do the checking on ALL peer nodes. We will get the same result.

peer channel getinfo -c mychannelpeer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'
The same result is seen in all peer nodes.

And we can take a look on the world state in each node. The couchdb can be observed through a browser (the port is 5984, 6984, 7984 and 8984, corresponding to peer0.org1, peer1.org1, peer0.org2 and peer1.org2, respectively)

This is from couchdb0, port 5984, for peer0.org1.example.com. Modify the port to see other peer nodes.

Now the network is ready, and we can start our observation when a peer node is down.

Shutdown a Peer Node

To shutdown peer0.org2.example.com completely, we will use the following docker commands.

docker kill peer0.org2.example.com
docker rm peer0.org2.example.com
docker volume rm net_peer0.org2.example.com
docker kill couchdb2
docker rm couchdb2

We can check docker ps and see those containers are gone. The network looks like this.

peer0.org2.example.com and its couchdb are killed and removed.

The network is still functioning, in a sense that chaincode invoke and query can still be done across other peer nodes. We can perform query on the running peer nodes and we still get back the data stored in the ledger.

For invoking chaincode function, remember that the endorsing policy requires a member from each organization. We now cannot send request to peer0.org2.example.com as it is down already.

peer chaincode invoke -o orderer.example.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n fabcar --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:9051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["changeCarOwner", "CAR0", "KC"]}'
peer0.org2.example.com is not reachable.

But we can still reach peer1.org2.example.com as it is also a member of Org2.

peer chaincode invoke -o orderer.example.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n fabcar --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer1.org2.example.com:10051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/tls/ca.crt -c '{"Args":["changeCarOwner", "CAR0", "KC"]}'peer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'
peer1.org2.example.com can fulfil the endorsing policy.

A quick summary: as far as we still have a peer node in the organization, the endorsing policy is still met and the fabric network is still working.

In the next part we will take a look what happens when we bring up this peer node back, and how we can resume the process.

Recover this Peer Node into Full Service

To recover this node back to a functioning peer, here are three steps.

  1. Bring up the peer node (container) and the world state database (couchdb)
  2. Join it back to channel
  3. Install chaincode

Let’s observe what happens in each step.

1. Bring Up the Peer Node and CouchDB

In this setup, the bringing up the node and the couchdb through docker compose files.

cd first-networkdocker-compose -f docker-compose-cli.yaml -f docker-compose-couch.yaml up -d couchdb2 peer0.org2.example.comdocker ps
Both couchdb and peer0.org2.example.com containers are up.

After we see them back as running containers, we can reach peer0.org2.example.com through the CLI terminal. Let’s first see if the channel is already there.

peer channel list
on CLI Terminal for peer0.org2.example.com

We see peer0.org2.example.com is not part of mychannel yet. The next move is to join it back to mychannel. We can also see that the couchdb is empty on this peer.

Note that it is on port 7984, which is couchdb2 for peer0.org2.example.com.

This is what happens after bringing up the peer node. It is not useable yet.

The peer node is not yet part of the channel.

2. Join the Peer Node to mychannel

To make the peer node holding the same ledger as other peer nodes in the same channel, we do not need to copy the whole blockchain and world state. Instead we can simply join the peer node to the channel and the ledger will be “sync-ed” to the node.

All we need is just the channel genesis block for that channel. In our environment this file mychannel.block is already in the CLI directory. In other case, you may get it directly from existing nodes.

This is the command to join the channel with the channel genesis block.

peer channel join -b mychannel.blockpeer channel list
on CLI Terminal for peer0.org2.example.com

We see now peer0.org2.example.com now joined mychannel.

If we take a look on its blockchain information, we see the full blockchain is already there. That means the remaining blocks are obtained from other peers.

on CLI Terminal for peer0.org2.example.com

We can compare the result with other peer node: for example, it is taken from peer1.org1.example.com.

on CLI Terminal for peer1.org1.example.com

If we quickly check the world state, we see the state is also updated.

Note that it is on port 7984, which is couchdb2 for peer0.org2.example.com.

This means that, after joining the channel, this peer node can get back all in the ledger, both the blockchain and world state.

join channel helps the peer node get back the whole ledger

As we have the ledger now, can we query or invoke chaincode function? Unfortunately we cannot yet.

unable to query chaincode as the chaincode is not installed yet on this peer node.

It is because the chaincode is not installed yet back to the peer node. We will do this in next step.

3. Install Chaincode

If we check the chaincode information on this node, we can see the chaincode is not installed yet, though the chaincode is already instantiated in mychannel, in which this peer is part of it.

peer chaincode list --installedpeer chaincode list --instantiated -C mychannel
No chaincode is installed yet in peer0.org2.example.com

Now we install the chaincode back to peer0.org2.example.com.

peer chaincode install -n fabcar -v 1.0 -p github.com/chaincode/fabcar/go

We can check again and now the chaincode is properly installed.

Now we see the chaincode installed on peer0.org2.example.com

This is how it looks like after chaincode is installed back to the node.

And we can query the chaincode from this node now.

Back to Normal

At this moment, we have completely brought up the peer0.org2.example.com node back to normal, serving as a full-function peer node in our network. We can try it again by invoking a chaincode function and pointing to peer0.org2.example.com as an endorsing peer.

We do it again on the terminal CLI peer0.org1.example.com, as what we did before in previous session.

peer chaincode invoke -o orderer.example.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n fabcar --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:9051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["changeCarOwner", "CAR0", "John"]}'peer chaincode query -C mychannel -n fabcar -c '{"Args":["queryCar","CAR0"]}'

We see the node peer0.org2.example.com successfully perform endorsement for this proposal.

Observation

When the peer node is “physically” brought up and not joining the channel yet, it can do nothing in the fabric network. In our case, after we brought up the peer node using docker-compose, the peer node is not yet part of the network.

After the peer node joins the channel, it gets back the whole ledger from the network. Now the peer node is in the same page as other peer nodes in the fabric network. We see both the blockchain and the world state are sync-ed after joining the channel.

Having the same ledger content does not mean that the node can process chaincode invoke or query. What we need is to install the proper chaincode back to this peer. After chaincode installation, the peer node can handle chaincode query and perform the endorsing process. At this stage we can say this peer node is fully functioning.

The steps mentioned above are only applicable when this peer node is already configured in the fabric network, i.e., it is already defined in the fabric network. In our case this peer node is in the network at the very beginning. Therefore it is not a brand-new peer installation.

In case it is a brand-new peer node, for example, adding peer2.org2.example.com into the fabric network, the process is different. We need to have crypto material for this new peer and add its information to the ledger first before we can join the peer to the fabric network. The process is comparable to “adding an organization on a fabric network”. You can take a look on an explanation about this process (article).

Clean Up

As good practice, always clean up everything after demonstration.

cd first-network
./byfn.sh down
docker rm $(docker ps -aq)
docker rmi $(docker images dev-* -q)

Hope this article helps you understand more about Hyperledger Fabric.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store