Introduction
This is another “behind the scene” demonstration about Hyperledger Fabric. Gossip plays an important role in Hyperledger Fabric. In this article, we will inspect gossip when a fabric network is started, by breaking down the starting process and observing the logs of individual components. We will come across some concepts in fabric, such as leader and anchor peer, and see how gossip helps building a scalable consortium platform in Hyperledger Fabric.
Overview
Many of us begin practicing Hyperledger Fabric with sample networks like First Network. First Network provides a script, byfn.sh, showing us a typical flow of bringing up a network:
- generating crypto material and channel artifacts
- bringing up network components (orderers, peers, etc)
- joining of peers to channel
- updating anchor peers
The network is then ready for use. We can deploy chaincode implementing our smart contract logic.
In fact there is some interesting process behind the scene. Here we take a look on the role of gossip, and how it helps building a scalable solution. Hyperledger Fabric adopts gossip as the peer to peer communication. To optimize whole flow, the gossip data dissemination protocol is implemented to achieve a scalable network deployment and operation.
The architecture reference provides a comprehensive description about this whole picture. Here I just extract some information relevant to us, and we will see it in action during the demonstration.
Gossip Bootstrap Configuration
Gossip bootstrap is configured in each peer. It contains one or a list of peers within an organization. When a peer is up and running, it will reach that list of peers for gossiping.
In the First Network setup, as we have two peers in each organization, a reasonable setup is to have the other peer as gossip bootstrap. And this is what we observe in configuration (base/docker-compose-base.yaml). Here only peers of Org1 are shown, and we see similar pattern in peers of Org2.
We will take a look at what happens when a peer is up and running: it will look for the bootstrap for gossiping. Besides we will also take a look later at what happens if no bootstrap is defined.
Leader
Leader in an organization is responsible for receiving blocks from ordering service, and distributing blocks to other peers in the same organization. Remember that a fabric network requires all peers in all channel members (organizations) receiving the new block from ordering service. The introduction of “leader” optimizes the overall process as ordering service only sends to the leader of each organization, and leader to other peers in this organization.
Leader is per organization. We will see in the demonstration that each organization will have its own leader. Leader is also per channel. Before joining a channel, there is no “leader” concept. It becomes relevant after the peer joins a channel.
Inside an organization, the leadership can be statically configured or dynamically elected. The default setup in First Network is through election. The configuration is in base/peer-base.yaml.
Anchor Peer
Anchor peer is needed in a network for peers to learn channel members outside its own organization. Without an anchor peer, knowledge of channel members is only limited to just one’s organization. For example, without an anchor peer, peers in Org1 only know peers in Org1 as channel members, but no knowledge of peers in Org2. At least one anchor peer is needed on a channel such that any peers in every organization know all other members in this channel.
Other than a preset configuration, anchor peer requires configuration update to a live channel. An update transaction is first created using configtxgen and a proper configuration file. This transaction is signed and sent to ordering service. A new configuration block carrying this anchor peer update transaction, is generated by ordering service. After this block reaches all peers joining this channel, each peer commits the block and now knows the anchor peer. Through gossiping each peer now knows all peers, both inside and outside its own organization, as channel members.
We will see later in illustration before and after an anchor peer is configured in the network.
First Network and Script byfn.sh
As we are using First Network, a network of two peer organizations, Org1 and Org2, each is configured with two peers (peer0 and peer1). A channel mychannel is created and joined by all four peers. Finally peer0 of both orgs are configured as anchor peers.
The script byfn.sh, when executed, performs the following tasks (steps shown in bracket is the step in demonstration)
- Generate crypto material and channel artifacts for the whole network (Step 1)
- Bring up all network components (containers) with docker-compose (Step 2–5)
- Create genesis block (block #0) for mychannel (Step 6)
- Join all four peers (from both organizations) to mychannel (Step 7–10)
- Update anchor peer configuration for both organizations (Step 11–12)
The network is ready for deployment. Each peer knows other peers in this channel, in its own organization and others. Here is the result after executing the script (using -n option: no chaincode deployed).
In our illustration we largely follow this flow. To make better observations, we break down docker-compose for peers one by one. We later also modify the configuration file or rearrange the sequence to make observations.
Demonstration
Step 0: Preparation
A new directory is directly copied from first-network/.
cd fabric-samples
cp -r first-network kc-first-networkWe will work on this kc-first-network/.
Meanwhile, the original CLI container depends on all the network components up and running. Since we will bring components one-by-one, we comment out the dependency.
Step 1: Generate crypto material and channel artifacts
We leverage byfn.sh to generate the required crypto material and channel artifacts for the network.
cd kc-first-network
./byfn.sh generateStep 2: Bring up Orderer and CLI
docker-compose -f docker-compose-cli.yaml up -d orderer.example.com cliStep 3: Bring Up peer0.org1
docker-compose -f docker-compose-cli.yaml up -d peer0.org1.example.comOpen a terminal to inspect the log for peer0.org1.example.com.
docker logs -f peer0.org1.example.comLet’s only focus on gossip related activities. From the log we leant that gossip initializes with configuration (external point and bootstrap) and gossip instance starts.
However, since gossip bootstrap is configured, peer0.org1.example.com is reaching peer1.org1.example.com, which is not running yet.
Step 4: Bring Up peer1.org1
docker-compose -f docker-compose-cli.yaml up -d peer1.org1.example.comWe don’t see the reaching problem as peer0.org1.example.com. In fact peer1.org1.example.com is also configured with peer0.org1.example.com as its gossip bootstrap, and starts reaching peer0.org1.example.com. Since peer0.org1.example.com is up and running, we see from the log that it reaches peer0.org1.example.com already (grpc.method=Ping, grpc.code=OK).
And almost at the same time when peer1.org1.example.com is started, peer0.org1.example.com also reaches peer1.org1.example.com. The timestamps match well.
- gossip instance of peer1.org1.example.com started in 06:39:05.878 (above)
- peer0.org1.example.com reached peer1.org1.example.com in 06:39:05.892 (below)
With that the two peers are connected at gossip level. Note that there is no channel related role like leader or anchor peer yet. This happens when we join the peers to channel.
Step 5: Bring up Peers of Org2
To keep the demo flow of byfn.sh script, we bring up peer0.org2.example.com and peer1.org2.example.com as well. We observe similar behaviour.
docker-compose -f docker-compose-cli.yaml up -d peer0.org2.example.com peer1.org2.example.comAnd since both peers are up and running almost the same time, we do not see the connecting problem message as we see in Step 3.
Step 6: Prepare Genesis Block for mychannel
docker exec cli peer channel create -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -c mychannel -f ./channel-artifacts/channel.txThe result is a genesis block file mychannel.block stored in CLI container. This is used for joining peers to mychannel.
Step 7: Join peer0.org1 to mychannel
Note the default setting of CLI targets to peer0.org1.example.com using Admin of Org1.
docker exec cli peer channel join -b mychannel.blockObservation from log
- Create ledger for mychannel with genesis block
- Commit block #0
- Join gossip network for mychannel (with 2 organizations learnt from block #0)
- Find no configuration of anchor peers from the two organizations yet (will be done in final step)
- Become a leader, and elected as a leader
Step 8: Join peer1.org1 to mychannel
docker exec -e CORE_PEER_ADDRESS=peer1.org1.example.com:8051 -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/tls/ca.crt cli peer channel join -b mychannel.blockSimilar observation of logs of peer0.org1.example.com, except for the leadership.
Therefore, from channel view, peer0.org1.example.com is the leader for Org1, responsible for receiving new blocks from ordering service and sending to other peers in Org1.
Step 9: Peers of Org1 Learn Channel Membership
After leadership of an organization is done, we immediately see logs from both peers showing what they learn about channel membership.
peer0.org1.example.com: now know peer1.org1.example.com as channel member
peer1.org1.example.com: now know peer0.org1.example.com as channel member
Step 10: Joining Peers of Org2 to mychannel
docker exec -e CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp -e CORE_PEER_ADDRESS=peer0.org2.example.com:9051 -e CORE_PEER_LOCALMSPID="Org2MSP" -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt cli peer channel join -b mychannel.blockdocker exec -e CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp -e CORE_PEER_ADDRESS=peer1.org2.example.com:10051 -e CORE_PEER_LOCALMSPID="Org2MSP" -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/tls/ca.crt cli peer channel join -b mychannel.block
Similar result as expected. About the channel membership on these two peers. Now we see
peer0.org2.example.com: as a leader, and also know peer1.org2.example.com as channel member
peer1.org2.example.com: see someone as leader, and also know peer0.org2.example.com as channel member
As of now, we know that gossip works well within an organization, as peers know each other inside an organization. But it is not the final picture yet, as peers also need to know peers in other organizations. This is not yet available until an anchor peer is configured.
Step 11: Update Anchor Peer for Org1
docker exec cli peer channel update -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -c mychannel -f ./channel-artifacts/Org1MSPanchors.txWe first see a new block #1 is created, received and committed by all peers of both Org1 and Org2. Let’s make some observations.
First, we observe how the new block reaches each peer. We know that new blocks are created by orderer. From the tcpdump trace, we only see orderer.example.com was communicating with peer0.org1.example.com and peer0.org2.example.com. Not to peer1 on both organizations. It is because both peer0 of both organizations are leaders.
Note that this is just showing two packets. It is a series of packets between orderer to both peer0. But we do not see either peer1.
Also, by capturing the log of receiving block #1 on each peer, we can tell that
peer0.org1.example.com (timestamp: 07:01:34.583)
peer1.org1.example.com (timestamp: 07:01:34.588)
peer0.org2.example.com (timestamp: 07:01:34.631)
peer1.org2.example.com (timestamp: 07:01:34.637)
We see peer1 receiving the block a little bit later than peer0, in both organizations. While it cannot be a solid proof, at least it aligns to what we are told that a leader is sending to other peers within an organization. This is how scalability is achieved, by introducing leaders and breaking the “broadcast” to channel into two tiers.
Secondly, we will observe what happens after every peer commits the block into ledger. They all perform gossiping once they know an anchor peer is known to them. After several rounds of gossiping, we see the result of channel membership.
peer0.org1.example.com
peer1.org1.example.com
peer0.org2.example.com
peer1.org2.example.com
The whole membership map for each peer is completely learnt. As a bonus, we also see how peers of Org2 learn peer1.org1.example.com in subsequent gossiping.
Now each peer inside a channel knows all other peers, inside and outside its organization. With this, the network should be ready for use.
Step 12: Update Anchor Peer for Org2
We see just one anchor peer in a channel makes the network ready for use. It is highly recommended to have an anchor peer per organization.
docker exec -e CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp -e CORE_PEER_ADDRESS=peer0.org2.example.com:9051 -e CORE_PEER_LOCALMSPID="Org2MSP" -e CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt cli peer channel update -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -c mychannel -f ./channel-artifacts/Org2MSPanchors.txWe see now the channel membership view change, as the one in the previous step is already the latest view.
Variation
For learning purposes, we will do some observations by varying some steps. This helps us to understand more how the change affects the fabric network.
Removing Gossip Bootstrap from Peer Configuration
The gossip bootstrap is set in base/docker-compose-base.yaml. Comment the variables CORE_PEER_GOSSIP_BOOTSTRAP on each peer.
We repeat the overall flow above. Here I just highlight the findings from the logs.
After Step 3 and 4, we see two peers in Org1 not talking to each other at gossip level. Similar in Org2 after Step 5.
After we join all the peers to mychannel with the block file, we observe all peers are now elected as leader.
peer0.org1.example.com
peer1.org1.example.com
peer0.org2.example.com
peer1.org2.example.com
It is due to the lack of gossip within an organization.
Is this a problem? Let’s see what happens if we update the anchor peer (Step 11).
peer0.org1.example.com
peer1.org1.example.com
peer0.org2.example.com
peer1.org2.example.com
After gossiping, all peers in the channel have got the right channel membership view. Also we observe some peers have stopped or renounced the leadership.
peer0.org1.example.com
peer0.org2.example.com
Our observation is, without gossip bootstrap configured on each peer, the leader election within an organization is out of control, as every peer claims to be leader. By all means this is not the desired situation as orderer needs to talk to every leader. After an anchor peer configuration is updated, all peers finally learn the whole channel membership view. At the same time, some peers also stop being a leader. This achieves what we wish to have. Therefore, the final picture should work well, even if we do not have gossip bootstrap configured on each peer.
Having Anchor Peer Update before Joining another Organization
In byfn.sh the anchor update (Step 11) is done after peers in Org2 join the channel (Step 10). Here we observe what happens if we swap these two steps. That is to say, mychannel already has an anchor peer before peers of Org2 join mychannel.
Here we only join peer0.org2.example.com to mychannel. After joining, we first see the same process happening as what we saw in Step 7.
- Create ledger for mychannel with genesis block
- Commit block #0
- Join gossip network for mychannel (with 2 organizations learnt from block #0)
- Find no configuration of anchor peers from the two organizations yet (will be done in final step)
- Become a leader, and elected as a leader
Right after that, peer0.org2.example.com receives block #1 (from orderer). And after committing block #1, it learns an anchor peer in peer0.org1.example.com, and begins gossiping.
The result is positive: peer0.org2.example.com learns both peers in Org1 as channel members.
And at almost the same time, peers in Org1 also learn this new peer as a channel member.
peer0.org1.example.com
peer1.org1.example.com
The result seems the same as what the byfn.sh script is performing, with a change in when anchor peer is updated. I see this a more realistic scenario, as in many cases a new organization is joining a live network (where anchor peers should have already been there). It also eliminates the short window of splitting when anchor peer configuration is not updated yet.
Summary
In this article we examined the role of gossip in bringing up a fabric network. We followed the script byfn.sh from First Network by breaking down into steps, and observed from the logs the behaviours when bringing up a peer, joining a peer to channel, the role of anchor peer in cross-organization membership learning. We also made some variations to explore more on gossip and fabric network. Hope you can gain some more ideas on Hyperledger Fabric.
