Add an Orderer to a Running Raft-based Orderer Cluster

1. Overview

Raft-based ordering service provides crash fault tolerant cluster setup for ordering service. Since its introduction in release 1.4.1, it becomes a better cluster solution than Kafka, and recently becomes the recommended ordering service setup in v2.0. One of the most common ask is about the process of adding an orderer in a running orderer cluster. While fabric documentation provides some guideline, there are some good posts addressing this. Here I come up this article to give a more detail step-by-step illustration on the overall process. Despite its complex and tedious flow, I hope this provides a clearer instruction, and we can also know more about Hyperledger Fabric software.

I am using v1.4.2 for this demo.

2. Concept Review: Channel

For most of us working on Hyperledger Fabric, we always care about the channel we build and chaincode we deploy on the channel. A more accurate term for this channel is called Application Channel, where it is used for application purpose, that is, more for the business related.

There is another channel, a more foundation one used within Fabric software. This is the System Channel or Orderer System Channel. From the name we know the purpose of this channel is for orderers. In fact it is the first channel bootstrapped or created.

As we can see later, both System Channel and Application Channel contain information about orderers. When we add a new orderer to a running fabric network, we need to first add it to System Channel and then Application Channel before the newly added orderer can serve the application.

3. First Network: Setup

As our demonstration is to build on top of the First Network, let’s take a look on the original setup of First Network.

3.1 Raft-based Ordering Service

First Network provides an option to bring up raft-based ordering service. When selected, total five orderers are configured. The configuration files about the raft-based ordering service are:

• crypto-config.yaml: defining the host name for the orderers, all under one organization.
• configtx.yaml: defining the OrdererOrg. A profile SampleMultiNodeEtcdRaft is defined for this raft-based cluster of five orderers, with its client and server certificates specified for each of them. We are using this profile when creating the genesis block, which is preloaded in orderer.
• docker-compose-cli.yaml and docker-compose-etcdraft2.yaml: These two files defines the containers for the five orderers. They are separated into two files for historical reason. Before raft comes the default setup is Solo, and one orderer is defined (the former). With raft added, one more file (latter) is created to include the other four orderers. The updated byfn.sh script will bring up these two docker-compose files properly. Besides, base/peer-base.yaml and base/docker-compose-base.yaml are extended in these two docker composer files.

To bring up First Network with raft, we are using ./byfn.sh -o etcdraft.

3.2 Peer Organizations, Peer Nodes and Channel

First Network comes with two peer organizations (Org1 and Org2), each of which two peers are provisioned. All these four peers join the mychannel.

3.3 Demo Chaincode

While the First Network is running after all peers joining mychannel and anchor peers are configured, the script byfn.sh comes with a default chaincode chaincode_example02, which will be deployed once the fabric network is ready for use. We will also use this chaincode for testing after we add orderer node.

3.4 First Network Script: byfn.sh

First Network comes with a very comprehensive script byfn.sh, providing many options when we test fabric network and/or chaincode. Here is a typical flow when the script is running.

• Generate the crypto material based on crypto-config.yaml. The tool cryptogen is used.
• Based on configtx.yaml, generate genesis block for the whole fabric network, and channel configuration transaction later being used when creating mychannel. Configuration updates are also generated for anchor peers of each organization. The tool configtxgen is used.
• Bring up the containers such that the whole fabric network is up and running. In our case both docker-compose-cli.yaml and docker-compose-etcdraft2.yaml are being brought up.
• Create block #0 for mychannel, and have the four peers join mychannel.
• Install and instantiate chaincode_example02 chaincode, with initial value a=100, b=200.
• After deployment, invoke the “invoke” function invoke(a, b, 10) to perform a simple operation. The result is to move 10 from a to b. Therefore the result becomes a=90, b=210.

The script ends with this, and we see the result as the script finally invokes (queries) on value of a.

3.5 Observe the Configuration of First Network

Let’s first execute the script. After that we will take a look into a configuration block.

./byfn.sh up -o etcdraft

And we are using the CLI to fetch the config block with this. The result is a PB (protocol buffer) format, and we use configtxlator to convert (decode in this case) PB to JSON, a human readable format.

docker exec -it cli bash
peer channel fetch config -c mychannel
configtxlator proto_decode --input mychannel_config.block --type common.Block --output mychannel_config.json

When we take a look on the mychannel_config.json file, there are two parts containing the relevant configuration about the orderer cluster.

First it is the TLS certificates for each orderer. Note that the same certificate is being used as client certificate and server certificate. Also the certificates are encoded into base64. It is located in channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters.

When we further scroll down we see the end point addresses for these five orderers. It is located in channel_group.values.OrdererAddresses.value.addresses.

Before moving to next session, tear down everything by,

./byfn.sh down

4. Process of Joining a New Orderer to Existing Cluster

4.1 Process Overview

On top of First Network, we are going to bring up a new orderer: orderer6.

Fabric Documentation (link) has outlined the flow. After comparing notes with some good implementation on this topic (link, link), here I break it down into this process.

1. Prepare crypto material for orderer6
2. Bring up First Network, with raft-based orderer cluster of five orderers up and running
3. Add TLS certificate to System Channel
4. Fetch the latest configuration block as the genesis block for orderer6
5. Bring up orderer6 container
6. Add orderer6 end point to System Channel
7. Add TLS certificate to Application Channel
8. Add orderer6 end point to Application Channel

Then we can test whether orderer6 is functioning well

Steps 3, 6, 7 and 8 involving a new configuration block with update.

Before we start, here are some points for discussion.

4.2 Crypto material for orderer6

In this demo, the requirement is that orderer6 has a separate set of crypto material (signing key and certificate) issued by same MSP as other orderers. Here I am using the same way as the script generating other orderers, that is, use cryptogen with crypto-config.yaml configuration file. I modify the configuration file to include orderer6. This is to ensure material of orderer6 is generated from the same MSP as other orderers.

If we have a CA running for the OrdererOrg, we can use CA to generate the crypto material any time. Since First Network does not come with CA in OrdererOrg, we do not use this method. For those interested in using CA for crypto material generation, please refer to Test Network in Fabric v2.0 (check test-network/organizations/fabric-ca/registerEnroll.sh).

4.3 Configuration Update

Step 3, 6, 7 and 8 involve a process of update the configuration through creating an update transaction, signing and submitting to orderer. The whole process is documented here (link), and there is a good tutorial of adding an new organization (link) in Fabric Documentation. I also wrote an article to break the process down by elaborating the tutorial (link).

In summary, this involves several rounds of editing and format conversion:

a. Fetch latest configuration block from a channel (as a PB format).
b. Convert (decode) it into JSON, and extract the data portion of interests.
c. Use editor or jq tool to add/modify the configuration.
d. Convert (encode) the modified JSON file back to PB format.
e. Convert (encode) the output of step b to PB format.
f. Compute the difference between output of step d and step e. The result is also in PB format.
g. Convert (decode) the difference (output of step f) into JSON format.
h. Insert an envelope.
i. Convert (encode) the JSON file back to a PB format.

The flow of creating a configuration update.

The output of final step is something ready for signing. We need to do the same process for Step 3, 6, 7 and 8 described above. Instead of doing this interactively in a terminal (you can try!), we come up with scripts. You can see this flow of process in steps 3, 6, 7 and 8 above. We will examine one script for this flow later.

4.4 Signature

We need to sign the transaction output. Here in this demo I am using OrdererOrg crypto material, as it involves both the System Channel and Application Channel.

To specify the OrdererOrg, I bring up an orderercli container. The setting is largely similar to CLI container in First Network, with proper environment variables setting. As you can see in the script, I am using this orderer-cli to perform all the tasks.

4.5 Configuration Files

There are some configuration files added or updated.

crypto-config.yaml

As mentioned, I am modifying the crypto-config.yaml to generate the crypto material for orderer6.

docker-compose-orderer6.yaml

This file is mainly adopted from other orderers. The main difference is the genesis block. As we will see later, orderer6 is not using the original genesis block, but a block after TLS material of orderer6 is added. The file will be called latest_config.block, and is mapped to the genesis block for orderer6 using volume (line 19).

docker-compose-orderercli.yaml

This file is adopted from the CLI container of docker-compose-cli.yaml. The environment variables are modified to use the crypto material of OrdererOrg.

5. Demonstration

The steps follow the flow mentioned in Session 4.1.

(1) Preparation of Configuration Files

We first create a new directory for our demonstration setup. For example,

cd fabric-samples
cp -r first-network kc-first-network
cd kc-first-network

Based on the previous steps, modify the crypto-config.yaml, and create both docker-compose-orderer6.yaml and docker-compose-orderercli.yaml.

(2) Bring up First Network

Bring up First Network with raft-based orderer cluster.

./byfn.sh up -o etcdraft

This should be similar to a normal First Network, with five orderers running, and the chaincode_example02 chaincode is deployed and invoked. Note that the result is of a is 90 at this moment.

The crypto material for orderer6 is generated, with our modified crypto-config.yaml file.

And only five orderers are running at this moment.

(3) Bring up Orderer CLI Container

We first bring up the orderer-cli as all commands in the scripts are issued on this container.

docker-compose -f docker-compose-orderercli.yaml up -d

(4) Add TLS to System Channel

This corresponds to our Step 3 in the process of Session 4.1. Note that the system channel in First Network is byfn-sys-channel. We execute this script 3_addTlsSysCh.sh.

It follows the Configuration Update in Session 4.3. Note we need to encode the certificate into base64, and a temporary file org6consenter.json keeping the content to be added to the configuration file.

chmod +x 3_addTlsSysCh.sh
./3_addTlsSysCh.sh

The result that now the TLS certificates for orderer6 is recorded in the system channel.

(5) Fetch the Latest Configuration Block

After the update is done, we need the latest configuration block. As mentioned above, the latest configuration block of system channel is used as the genesis block for orderer6.

chmod +x 4_fetchConfBlkSysCh.sh
./4_fetchConfBlkSysCh.sh

And we see the latest_config.block in channel-artifacts directory.

(6) Bring up Orderer6 Container

docker-compose -f docker-compose-orderer6.yaml up -d

(7) Add End Point to System Channel

We will execute this script 6_addEndPointSysCh.sh, corresponding to Step 6 mentioned in Session 4.1. This script is similar to (4), with difference on the content being modified (line 3).

chmod +x 6_addEndPointSysCh.sh
./6_addEndPointSysCh.sh

(8) Observe Orderer6

After the previous step, the system channel is ready for use. When we take a look on the log on orderer6, we can find some interesting information.

The orderer6 now sees all existing nodes in system channel byfn-sys-channel.

Besides,

Orderer6 has learnt the existence of an application channel, mychannel, from other orderers in the cluster. However, it claims “not belong to channel mychannel”. The reason is that orderer6 is not added to mychannel yet. It will be done in coming steps.

We can also compare the content of orderer.example.com and orderer6.example.com.

First about byfn-sys-channel

They are of the same length.

Now about mychannel

They are of different length. It is because orderer6 does not belong to mychannel yet.

(9) Add TLS to Application Channel

The application channel in First Network is mychannel. We will execute this script 7_addTlsAppCh.sh, corresponding to Step 7 mentioned in Session 4.1. You can see the whole process is almost identical to (4). The only difference is that this time it is applied to mychannel.

chmod +x 7_addTlsAppCh.sh
./7_addTlsAppCh.sh

(10) Add End point to Application Channel

Finally, we execute this script 8_addEndPointAppCh.sh, corresponding to Step 8 mentioned in Session 4.1. Similarly it is the same as (7), except that everything is applied to mychannel.

chmod +x 8_addEndPointAppCh.sh
./8_addEndPointAppCh.sh

(11) Observe Orderer6

It takes some time to get things settled (five minutes as mentioned in document). After a while, we see this log on orderer6.

It tells us orderer6 is now available for mychannel to use.

Again we can also compare the content of orderer.example.com and orderer6.example.com on mychannel:

They have the same content (blockchain) now.

(12) Test Orderer6

We now invoke chaincode function with target the newly added orderer6.

docker exec cli peer chaincode invoke -o orderer6.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer6.example.com/msp/tlscacerts/tlsca.example.com-cert.pem --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:9051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -C mychannel -n mycc -c '{"Args":["invoke","a","b", "10"]}'
docker exec cli peer chaincode query -C mychannel -n mycc -c '{"Args":["query","a"]}'

The result is 80, which means that orderer6 has successfully received and processed the transaction. (It was 90 before. After moving 10 from a to b, a is now 80).

6. Summary

Adding orderer to a running fabric network and a running orderer cluster involves a lot of steps. It is a lengthy, somehow repetitive and tedious process. Nevertheless it shows that an orderer can be added “on-the-fly” to raft-based orderer cluster. Though we will not interact too much about system channel in real life, through this process we do learn something about it in this process.