Overview
It begins with a question I have been asked several times in the past few months. Some have completed the tutorial Add an Org to a Channel, successfully adding a new organization to a channel. When they try to add this organization to a new channel, they encounter problems. The cause drives me to some system design of Hyperledger Fabric, in particular, on system channel.
The term channel we always refer to in fact is Application Channel, in which channel members are sharing the same ledger and chaincode for a specific business purpose. We can have as many Application Channels as possible to meet our business needs. Besides, there is one System Channel in which consortium configuration is stored. Whenever an update on the consortium, such as a new organization joining the consortium, we need to update this System Channel. And only those organizations defined in the consortium can be added as members of Application Channels. In the tutorial mentioned above we only act on Application Channel, not the System Channel. As a result we cannot add this new organization to a new channel.
In this article, we first replicate this situation, that is, after completing the tutorial, we cannot add this new organization into a new channel. By inspecting the configuration and blocks, we will locate the root cause. In the final part we will add this new organization to System Channel as a consortium member. With this, we are able to create a new channel that this new organization can be a member of.
We use Test Network as our base, with proper modification to meet our demonstration. A repository is created such that you can try in your environment.
Note: there is another article in which system channel is involved: adding an orderer in an ordering cluster. You can do a quick reference to that article, and see similar process when update configuration in the system channel.
Test Network: Modified for Demonstration
In this demonstration, we are using the Test Network (v2.0+). It is a two-peer-org setup, one peer on each org, plus a Solo-based ordering service. It comes with good scripts (network.sh) to bring up containers, create channels and deploy sample chaincode (fabcar). Besides it also provides scripts to implement the tutorial Adding an Org to a Channel.
For easy demonstration some files are modified such that we do not need to switch between files. Here is the repository for the demonstration.
Here is the highlight of modification.
cryptogen
We use the cryptogen (not Fabric CA Server) for simplicity. In the original directory structure organizations/cryptogen/, we have
crypto-config-orderer.yamlcrypto-config-org1.yamlcrypto-config-org2.yaml
We add one more crypto-config-org3.yaml, with a similar setup as other peer organizations (one peer, one admin and one user). The crypto material for Org3 will be generated upfront as well.
channel artifacts
The configuration file is configtx/configtx.yaml.
To prepare Org3 material, we add back the Org3 definition in the organizations section. Again, they are similar to Org1 and Org2.
We keep the two profiles: TwoOrgsOrdererGenesis and TwoOrgsChannel, as we need to remain the first part of demonstration. Meanwhile, we add one more profile: Channel1–3, and make Org1 and Org3 as the members of this new channel.
docker compose file
The only docker compose file we are using is docker/docker-compose-test-net.yaml. Inside we add peer0.org3.example.com. The container setup is largely similar to other peers, except for the correct crypto material and port number to be used.
repository
With these configuration files modified, we have generated certain material for our demonstration. Detail is skipped here. The material we have generated in the repository are
- crypto material for all organizations, including Org3, using
cryptogen - genesis block, configuration update transactions for mychannel, and configuration transaction for channel1–3.
- the content to be inserted into configuration in JSON, named
org3.json, is generated and kept inpeersOrganization/org3.example.com/
Finally, we have set up several files for terminal setup for different organizations. They are terminalorg1, terminalorg2, terminalorg3 and terminalorderer, with proper environment variables. We will source the file to set our terminal for the specific organization.
We first replicate the finding.
Tutorial Replay
Here we follow the tutorial Adding an Org to a Channel. Instead of repeating every command, here we just showed the screenshots of the results on main steps.
Bring up containers
We simply bring up all the containers, including peer0.org3.example.com.
Join mychannel and check status
After joining peer0.org1.example.com and peer0.org2.example.com to mychannel, here is the status of the three organizations.
Prepare transaction update and sign by both organizations
Following the tutorial, here is the signing from both Org1 and Org2.
After submitting, we see new blocks arriving to both peers (now the blockchain height is 4).
Join peer0.org3 to mychannel
With this, we can fetch block zero of mychannel and join peer0.org3.example.com.
Peer will get back the whole blockchain (see the blockchain height, and block hash, same as peer0.org1.example.com and peer0.org2.example.com).
Here is what we have completed so far.
We have repeated the whole process of tutorial. Now we can simulate the problem.
Simulating the Problem
Create the genesis block file for channel1–3. Remember that in channel1–3 we have defined Org3 as a channel member.
peer channel create -o localhost:7050 -c channel1-3 --ordererTLSHostnameOverride orderer.example.com -f ./channel-artifacts/channel1-3.tx --outputBlock ./channel-artifacts/channel1-3.block --tls --cafile $ORDERER_CAHere is what we expect: even if we have added an organization to an application channel, this organization is NOT YET in the consortium, that is, this organization is not yet in configuration of the system channel. We will take a look at both system channel and application channel, and their relation with our channel configuration file configtx.yaml.
Observation: System Channel and Application Channel
As we mentioned before, there is one system channel when the consortium network is up and running. Inside the system channel we find two main configuration settings: orderers and consortium members. This system channel needs to be updated when any change of ordering service (e.g. one more orderer added to a cluster) or new consortium member (organization) is added.
We will take a look at the configuration file and the blocks being fetched from both channels.
Configuration file: configtx.yaml
We first take a look at the configuration file configtx.yaml. The part we are interested in is the profiles.
The first profile TwoOrgsOrdererGenesis is used when we create the genesis block. In this profile, Org1 and Org2 are included. The command we are using (not shown in this document).
configtxgen -profile TwoOrgsOrdererGenesis -channelID system-channel -outputBlock ./system-genesis-block/genesis.blockAs we can see the command is acted on system-channel.
Meanwhile, the second profile TwoOrgsChannel is used when we generate transaction updates for the application channel. And for this setup, again, Org1 and Org2 are included in this application channel. This is valid as Org1 and Org2 is already in the consortium (system channel).
The command is acting on mychannel.
configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/mychannel.tx -channelID mychannelBesides the configuration file, we can also fetch blocks from both system channel and application channel and make some observations.
Configuration Block captured in System Channel
We use Terminal for Orderer in order to fetch the system channel.
source terminalordererpeer channel fetch config config_block_sys.pb -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com -c system-channel --tls --cafile $ORDERER_CA
We can see that there is a blockchain of length 2, and the configuration block is block #0.
We can see that inside the SampleConsortium, only Org1MSP and Org2MSP defined.
Configuration Block captured in Application Channel
We use Terminal for Org1 to capture the latest configuration update (should be block #3).
After decoding, we can see that inside the Application, Org1MSP, Org2MSP and Org3MSP are defined. This is the result after we have completed the tutorial, adding a new organization to a channel (mychannel).
Observation
Even though Org3 is being added in an application channel (mychannel), it is not yet in consortium and therefore we cannot create a new channel in which Org3 is a member. What we need is to add it back to the system channel.
Add Org3 to System Channel and then Create Channel with Org3
Now we understand that, in order to allow new channels to be created for a new organization, this organization must be added in consortium first. Here we demonstrate how to add organization to the consortium (system channel) such that we can add a new channel.
Fetch configuration block from system channel
Note: all commands here is done in terminal for Orderer.
source terminalordererpeer channel fetch config config_block_sys.pb -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com -c system-channel --tls --cafile $ORDERER_CA
Construct configuration update
With this, we go through the similar process of adding Org3 to Application Channel as before. But now this process is acting upon the configuration block from the system channel.
Here I just listed all the commands.
configtxlator proto_decode --input config_block_sys.pb --type common.Block | jq .data.data[0].payload.data.config > config.jsonjq -s '.[0] * {"channel_group":{"groups":{"Consortiums":{"groups":{"SampleConsortium":{"groups": {"Org3MSP":.[1]}}}}}}}' config.json ./organizations/peerOrganizations/org3.example.com/org3.json > modified_config.jsonconfigtxlator proto_encode --input config.json --type common.Config --output config.pbconfigtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pbconfigtxlator compute_update --channel_id system-channel --original config.pb --updated modified_config.pb --output org3_update.pbconfigtxlator proto_decode --input org3_update.pb --type common.ConfigUpdate | jq . > org3_update.jsonecho '{"payload":{"header":{"channel_header":{"channel_id":"'system-channel'", "type":2}},"data":{"config_update":'$(cat org3_update.json)'}}}' | jq . > org3_update_in_envelope.jsonconfigtxlator proto_encode --input org3_update_in_envelope.json --type common.Envelope --output org3_update_in_envelope.pb
Now we have the configuration update transaction.
Sign the update with orderer admin
Keep working in the Terminal for Orderer as orderer admin is needed to sign and submit the transaction.
peer channel update -f org3_update_in_envelope.pb -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com -c system-channel --tls --cafile $ORDERER_CAAnd we fetch the latest configuration block and see again if org3 is already there.
peer channel fetch config config_block_sys_update.pb -o localhost:7050 --ordererTLSHostnameOverride orderer.example.com -c system-channel --tls --cafile $ORDERER_CAWe see now a new block (block #2) is received. After decoding we can see three organizations inside.
Create Channel with Org3
Now create new channel channel1–3, which includes Org3 as a channel member.
Go to Terminal for Org1
peer channel create -o localhost:7050 -c channel1-3 --ordererTLSHostnameOverride orderer.example.com -f ./channel-artifacts/channel1-3.tx --outputBlock ./channel-artifacts/channel1-3.block --tls --cafile $ORDERER_CANow we receive block 0 for channel1–3 (channel1–3.block). We can join peer0.org1.example.com and peer0.org3.example.com to this channel.
This screenshot shows the final summary of channel members for both mychannel and channel1–3.
- mychannel: Org1, Org2 and Org3
- channel1–3: Org1 and Org3
And this is what we have now.
Summary
We begin with a problem we meet: according to the tutorial, we can add an organization to a channel, but we cannot add new channels with this newly added organization. The reason behind is that an organization needs to be in consortium before we can create new channels with this newly added organization. We find that we are in fact dealing with two different types of channels: one system channel, keeping the consortium configuration, and one or more application channels, which is the “channel” we always refer to, where shared ledger and business logic is enforced among selected consortium members. In this article we have deep-dived into configuration files, blocks captured on the system channel and application channel. We also demonstrated how to add a new organization into a consortium, such that this organization can later be added into another application channel. Hope this provides a good reference when you study Hyperledger Fabric.
