Two Ways to Generate Crypto Materials in Hyperledger Fabric: Cryptogen and CA Server
Overview
One of the most asked questions on Hyperledger Fabric is about identity. A permissioned blockchain requires that an entity, be it a client, an administrator or a network component, must be first identified and permissioned before accessing a consortium network. Hyperledger Fabric uses several concepts like organization, membership service providers, digital certificate, which makes the understanding of the whole picture a little challenging for new learners. Here in this article again I try to elaborate these concepts through analyzing Test Network, a network example in fabric material, and the two different ways to generate the crypto material for these entities. I hope to give you another perspective on this topic.
Organization and Membership Service Provider
Hyperledger Fabric is always chosen to build a consortium network to address a business problem. The participants are usually business entities in this case. An example is a trade finance platform, where participants are mainly the banks, plus some governance or regulatory bodies. Hyperledger Fabric uses organizations to represent these participating entities. In the trade finance platform, each bank is an organization. A governance body is also another organization. This is how we understand organization from a business perspective.
In Hyperledger Fabric, each organization is characterized by its Membership Service Provider (MSP). On one side, MSP creates credential material to all entities within an organization. On the other side, MSP represents the organization to participate in a consortium network. Technically the consortium network is formed by the MSP of all organizations. They trust one another when the MSPs are functioning well.
MSP is implemented in Public Key Infrastructure (PKI). In a typical PKI environment each entity is holding a private key and a digital certificate (certificate) containing the public key and relevant information about this entity. This certificate is signed and issued by a Certificate Authority (CA) and becomes the identity of this entity.
When an entity signs a message with its private key, it produces a signature on that message. Everyone who obtain this message, signature and the signer’s certificate, can
- tell that only the private key owner can create this signature (through public key from certificate)
- know the identity of the private key owner (through information recorded in the certificate)
- trust that the signer is the one claimed in the certificate (through verifying CA’s signature on the certificate, with the assumption that CA’s certificate is trusted).
This signing and verification process happens everywhere in a consortium network. Each organization knows the MSP of other organizations in a consortium network, and trust of this consortium network is built up from a “who’s who” perspective. As an example, here is a part of a transaction showing the endorsement, and we can see those information mentioned above.
In this transaction two endorsements are included. Each endorsement contains
- endorser: the certificate (identity) of endorser in base64
- signature: produced by that endorser against the message in proposal response payload (collapsed).
Everyone in the consortium can verify that it is the endorser in the certificate signing this proposal response, and therefore this endorsement is accepted.
Test Network
It is always too abstract if just reading the material without examples. Here we use Test Network from fabric samples as an example. Similar to First Network in previous releases, Test Network comes with a well-designed script in fabric v2.0. It is a three-organization design, with one orderer organization and two peer organizations.
In general the structure of an organization includes
- MSP, or we use the term Identity CA (or simply CA) representing it
- Network components, either orderers or peers
- Network users, who represent the actual human or client application acting on the network
We use network components and network users separately if the discussion is specific to each of them. If something applies to both network components and network users, we will use the term entities of an organization.
CA is responsible for handling the identity of all entities within an organization. Each entity obtains a credential (crypto material) from CA. The credential includes a private key and a digital certificate representing this entity. The security is maintained as long as the entity is keeping the private key safe.
While the main purpose of CA is on identity area (prove an entity is what it claims to be), we will encounter another set of crypto material for another purpose. In Hyperledger Fabric communication between components are also secured by TLS, for example, client application to peers, cluster communication between orderers, etc. Although this article is more focused on identity, we will also take a look on TLS, but you should always see them as two separate systems for different purposes.
A Note about Directory Structure of Crypto Material
For those who have gained some experience in First Network in the previous releases or Test Network in v2.0, you may have noticed that the crypto material is always stored in a fixed directory structure.
We will examine the directory in detail, but I wish to point out that there is not a must to follow such a data structure. If we trace back a bit history, at the beginning we are using cryptogen (more detail in next session) to create the crypto material, and docker-compose to bring up network components. Inside the docker-compose configuration files directories are mapped to each network component. For example, crypto material for peer0.org1.example.com is correctly installed in this peer through docker-compose files.
It is a decent and systematic way as it can be coded into configuration. Nevertheless, you also can mount the crypto material one-by-one and manually, if it is desired.
As we still rely on docker-compose files for bringing up network components, we will follow this directory structure. It is automatically generated with cryptogen, but some more copy-n-paste is needed when we are using CA Server.
With these, we can start looking into the first way to generate crypto material: cryptogen.
Generating Crypto Material using Cryptogen
Hyperledger Fabric provides a tool that crypto material can be generated with minimum configuration. The tool is bin/cryptogen. Working with a configuration file, the crypto material of Test Network is generated and the result is kept as the directory structure shown above. With that, we can bring up the consortium network with docker compose files.
Directory Structure
We first use ./network.sh script to bring up Test Network and make some observations on the directory structure.
cd test-network
./network.sh upPer script design the material is kept in organizations/. Here we take Org1 as an example for exploration.
Here are a quick summary for all these components
ca/: a directory holding crypto material for CA of Org1peers/: a directory holding a list of peers under Org1, and each peer has its own crypto material, of both identity and TLSusers/: a directory holding a list of network users under Org1, and each peer has its own crypto material, of both identity and TLStlsca/: a directory holding the TLS CAmsp/: it is the material joining a consortium network.connection-org1.yamlandconnection-org1.json: these are the connection profile files in different formats. They will be used in client applications.
We break it further down and observe some relevant items.
CA Certificate
We first take a look at the CA of Org1. It is this CA issuing identity (certificate) to all entities within Org1.
From the directory we see a private secret key and a certificate. We will inspect the subject and issuer of the certificate.
It is a self-signed certificate (subject == issuer). It is a typical design for non-production environments, as we allow a new self-signed certificate as the CA to issue identity for the whole organization. This is what cryptogen can provide, and this is also its limitation (see discussion later).
Network Components (peer)
As Org1 is a peer organization, the network component is “peer”. In the peers/ we see a list of peers defined per the crypto configuration file (will be examined later). In the Test Network we have only one peer.
Double click to this directory, there are two directories. msp/ is for identity, and tls/ is for TLS material. This directory structure will be seen in every entity.
We focus on the identity part. We will revisit the TLS material when we cover TLS later.
Inside msp/ the directory structure looks like this. We will be very familiarized with this directory structure later, as it is the structure for all entities, both network components and network users.
For identity, the private secret key of this peer is stored in msp/keystore/, while the certificate is kept in msp/signcerts/.
We take a look at the certificate of this peer. The issuer is the CA of Org1 mentioned above, and the subject is CN=peer0.org1.example.com and OU=peer.
Network Users
Network users are kept in users/. From the directory we see a list of users defined in the crypto configuration file (will be examined later). In Test Network we have one Admin, and one user named User1.
Let’s take Admin as an example. Again there are two directories. msp/ is for identity, and tls/ is for TLS material.
Here is the private key and certificate for Admin.
Finally we take a look at the certificate of Admin, both the subject and issuer. We see that the subject contains OU=admin, and the certificate is issued by the CA of Org1.
Same directory structure for User1. There is a difference in the certificate subject. We see OU=client for User1. Obviously it is the difference of roles in these two network users. Admin is an admin, while User1 is a client user. If some actions only allow admin role to perform, User1 is denied.
TLS
Though TLS is not relevant to the identity system, the crypto material for TLS is also generated during the cryptogen process. Here we just make a highlight and locate those materials inside the directory structure.
A TLS CA is also created for Org1. The TLS CA Certificate is stored in tlsca/. TLS crypto material for all entities in Org1 is generated from this TLS CA.
We see this is a self-signed certificate. Therefore in cryptogen the Identity CA and TLS CA are different systems, and do not have any relationship to each other.
In a TLS communication, network components play a TLS server role. Therefore each network component is installed with a TLS server key and server certificate. As in Org1 we only have one peer, we can inspect the material inside peers/peer0.org1.example.com/tls/.
And again we see the TLS server certificate server.crt is issued by TLS CA.
Network user plays a TLS client role. Therefore each network user is installed with a TLS client key and client certificate. We take Admin as an example here.
And the client certificate client.crt is also issued by TLS CA.
This is a diagram showing the relationship of crypto material generated with cryptogen. Note that the Identity CA for each organization is just a CA’s private key and a CA Certificate. There is no software currently running as a CA Server. You will not see a container running in any CA Servers generating this. This certificate issuance process is done by cryptogen software itself. Nevertheless, one can easily bring up a CA Server with CA’s private key and CA Certificate when identity for more entities is needed.
cryptogen tool in Test NetworkConfiguration File
We have seen the result of cryptogen. Now we take a look at the configuration files, which tell cryptogen how to generate the crypto material shown above.
The configuration for Test Network is kept inside organizations/cryptogen/. Inside these files we can specify the organization structure of the network (peer organization and orderer organization), and detail of entities within an organization. These configuration files are self-explanatory. For sake of completeness we put them here.
For orderer organization. The result is one orderer with name orderer.example.com. One Admin user is generated by default.
For peer organization (here we see Org1). The result is one peer (Count 1 in Template) and the name is by default peer0.org1.example.com. One Admin user is generated by default. The Count 1 in Users means one additional user is generated as well, with a name User1 by default.
This matches all the entities we see above in Org1.
Use of cryptogen
In the script network.sh, the crypto material is generated in this command.
cryptogen generate --config=<configuration_file> --output=<output_directory>This command is executed for all the three organizations: Org1, Org2 and Orderer Organization.
We can see how cryptogen works: with this command and configuration files, we can build the directory structure of crypto material shown above.
Discussion
Cryptogen by far is the simplest way for us to generate crypto material for typical network designs. No matter how many orderers we are to deploy (one for Solo, five for Raft, etc) in orderer organization, how many peers needed for each peer organization, or how many client users needed in each organization, we can specify the requirement in the configuration files and generate the material with a simple command.
This is also where the limitation lies. From Hyperledger Fabric documentation, we learn that cryptogen is good for generating crypto material for testing purposes, and would normally not be used in production operation. There are several considerations.
One is that the CA certificates are all generated as self-signed certificates. In real life we may ask for different settings. For example, we cannot use an existing CA as the MSP for our organization, or we cannot chained certificates.
Also it is predefined with organization type: orderer organizations and peer organizations are defined separately. If for example we need orderers owned by peer organizations, we have to use some workaround (refer to this article on this requirement).
Last but not the least, material is generated once based on configuration file, but in real life we may need more peers or more clients after deployment. Although it is still doable, as we can generate the new entities with CA’s private key, it is not as simple as running cryptogen on a configuration file.
If you are just creating for testing, the above should not be your concern, and cryptogen is good enough. Otherwise, you can take another approach: bring up a CA server and generate the crypto material for entities with this CA according to your desired network design. This is the second way to generate crypto material for a consortium network.
Generating Crypto Material using CA Server
We see some limitations in using cryptogen. To gain more flexibility and make it more practical, a better way is to bring up a CA Server, and we can generate crypto material for entities according to our network design. This involves more steps, but provides a standard way to either generate crypto material that cryptogen cannot generate, or for future use if more network components and users are needed.
Overview
A CA Server is a software to generate crypto material on request. Inside a CA Server there is a CA, represented by a private secret key and a CA Certificate. Any certificates signed and issued by this CA is verifiable with the CA Certificate (or CA’s public key). CA’s private secret key and CA Certificate is the essence of a CA, while a CA Server is just a software tool to perform this signing.
Technically speaking, once one gets hold the CA’s private secret key and CA Certificate, one can issue certificates. It can be done on a CA Server, or it can be done by common tools such as openssl. Therefore, always make a difference between a CA (representing the ownership of that private key) and a CA Server (where certificate is signed and issued).
There are many CA Server products in the market. Hyperledger Fabric comes with one, Fabric-CA Server. This whole process begins with bringing up a Fabric-CA Server, running as a docker container. We either use an existing CA for this CA Server, or generate a self-signed certificate as root CA. In the Test Network example, it is a self-signed certificate, and we then generate all crypto material from it.
To interact with Fabric-CA Server, we need another tool on our side. It can be either a Fabric-CA Client or Fabric SDKs. Fabric-CA Client is used in Test Network script. Note that Fabric-CA Client is running on our localhost, and it interacts with the Fabric-CA Server which is a running container. You can refer to a comprehensive explanation in fabric documentation (link).
Directory Structure and Content
As mentioned before, we need the same directory structure in order to match the docker-compose configuration. Therefore the result of using CA needs to be the same as the result after using cryptogen. This involves some name changing and file moving. Instead of repeating everything about the directory structure, here we highlight some difference on the content and process.
We first use ./network.sh script to bring up the minimum setup and observe the resulting directory structure and content. Note the option -a is required to bring up crypto material using CA instead of cryptogen.
cd test-network
./network.sh up -caThe majority of steps are in fact executed in the script fabric-ca/registerEnroll.sh.
We see three CAs, one for each organization, up and running. As a comparison, no CA containers are running when we bring up the network using cryptogen.
Only the CA Certificate is here, which is a public information, but there is no private secret key (scroll up to the session on cryptogen and see a comparison). It is because the private secret key is in the CA Server. This is a more realistic way as it is the CA Server holding that secret, and it should not be available and visible outside the CA Server. We will see more detail about this CA Certificate when we dive into the Fabric-CA Server
The structure for entities (peer and users) are similar. Here we just bring out the certificate of the peer and both users. They are all issued by CA of Org1, with proper identifiers in OU.
We now take a look at the TLS. First the TLS CA. Interesting enough, we see the same certificate as CA.
In the previous session about cryptogen, we see a separate TLS CA for all TLS material. When using a CA, the script of Test Network generates all crypto material for TLS with the same CA. There is no strict requirement whether using one CA for each purpose or for both purposes.
Finally we see that only network component is equipped with TLS server certificates, and network users (e.g. Admin, User1) is not. It works fine as most setup only server authentication (one-way authentication) is required in TLS. To know more about TLS settings in Hyperledger Fabric, refer to the fabric documentation about this (link).
This is how it looks like in Test Network after launching it using CA Server.
Fabric CA Server in Test NetworkFabric-CA Server
We have seen the result. Now we can take a look on the CA Server side, and locate the material inside CA Server.
Our Fabric-CA Server is being brought up as a docker container. The docker compose file for it is ./docker/docker-compose-ca.yaml. It brings up all the three CA servers. We take a look at the one for Org1.
Some observations in this configuration.
There is a predefined configuration file fabric-ca-server-config.yaml for Org1, stored in organizations/fabric-ca/org1. From the volumes (line 23–24) it is mapped to the container. Therefore this file exists in the container when fabric-ca-server is running. This is a practical way for managing configuration, as we can edit in localhost while the edited configuration file is directly in the container.
Several environment variables are set in the container. What is defined here is to override the configuration file.
Finally the command to bring up the Fabric CA Server is fabric-ca-server start with a bootstrap identity -b admin:adminpw. It will be used in the next step when we enroll a CA Admin with Fabric-CA Client.
Certain material is kept in /etc/hyperledger/fabric-ca-server when the Fabric-CA Server is running. We can examine the content in localhost, according to the mapping in the container (line 23–24).
Among them we will see
msp/: where the crypto material for CA of Org1 storedca-cert.pem: the CA Certificate for Org1tls-cert.pem: the TLS server certificate for CA Server, used when accessing the CA Server with TLSfabric-ca-server.db: the database of the Fabric CA Server, recording the user registration and certificate issuanceIssuerPublicKeyandIssuerRevocationPublicKey: the names are self-explanatory.
We first take a look at the CA Certificate ca-cert.pem for Org1.
This is a self-signed certificate. Again, this is just one option. With Fabric-CA Server we can incorporate an existing certificate (with its private key).
Fabric-CA Server, as a server, supports TLS. Therefore a TLS server certificate is generated and kept as tls-cert.pem.
And this is just a TLS server certificate issued by CA of Org1.
The corresponding private keys are kept in msp/keystore/.
It does not take too much effort to correlate the two secret keys matching the CA Certificate for Org1 and TLS server certificate for the Fabric-CA Server.
The Process in Detail
We follow the process executed in network.sh script. Here is the overall flow of generating crypto material using CA Server. For an organization,
- Bring up the Fabric-CA Server which is used as the CA of that organization.
- Use Fabric-CA Client to enroll a CA Admin.
- With the CA Admin, use Fabric-CA Client to register and enroll every entity (peer, orderer, user, etc) one by one to the Fabric-CA Server.
- Move the result material to the directory structure.
Proper configuration is provided whenever needed. And repeat this process for all organizations, as each of them has its own organization CA.
We will take Org1 as an example and take a look on the steps. And we will follow the instruction in network.sh and perform this process once. This helps us understand what happens behind.
Before, clean up what has been deployed with,
cd test-network
./network.sh down# if you see permission error in removing some directories, perform the following
sudo rm -r organizations/fabric-ca/org1/msp/
sudo rm -r organizations/fabric-ca/org2/msp/
sudo rm -r organizations/fabric-ca/ordererOrg/msp/
Step 1: Fabric-CA Server
IMAGE_TAG=latest docker-compose -f docker/docker-compose-ca.yaml up -dCheck if the CAs are up and running.
Step 2: Enroll CA Admin
As we are using Fabric-CA Client to interact with the Fabric-CA Server, we first need to specify the Fabric-CA Client home directory.
mkdir -p organizations/peerOrganizations/org1.example.com/export PATH=$PATH:/home/ubuntu/fabric-samples/bin
export FABRIC_CA_CLIENT_HOME=${PWD}/organizations/peerOrganizations/org1.example.com/
Now we can enroll a CA Admin.
fabric-ca-client enroll -u https://admin:adminpw@localhost:7054 --caname ca-org1 --tls.certfiles ${PWD}/organizations/fabric-ca/org1/tls-cert.pemIn this enrollment,
- bootstrap identity
admin:adminpwis specified - the CA name is ca-org1
- the correct TLS server certificate for CA Server
Now our Fabric-CA Client has the credential of a CA Admin, and ready to perform the enrollment for each entity. For information this credential is kept in the home directory (msp/keystore and msp/signcerts).
And the certificate is issued by the CA of Org1.
Step 3: Register and Enroll Each Entity
For each entity (peer0, Admin and User), we will first register in the CA Server, and then enroll the entity to obtain the credential.
We need a config.yaml file to enable the OU identifiers, and keep the OU identifiers for each type of entity. They are peer, orderer, client and admin.
NodeOUs:
Enable: true
ClientOUIdentifier:
Certificate: cacerts/localhost-7054-ca-org1.pem
OrganizationalUnitIdentifier: client
PeerOUIdentifier:
Certificate: cacerts/localhost-7054-ca-org1.pem
OrganizationalUnitIdentifier: peer
AdminOUIdentifier:
Certificate: cacerts/localhost-7054-ca-org1.pem
OrganizationalUnitIdentifier: admin
OrdererOUIdentifier:
Certificate: cacerts/localhost-7054-ca-org1.pem
OrganizationalUnitIdentifier: ordererStore this file under client home directory: organizations/peerOrganizations/org1.example.com/msp/
Here we just take peer0.org1.example.com as an example.
We first register this entity.
fabric-ca-client register --caname ca-org1 --id.name peer0 --id.secret peer0pw --id.type peer --id.attrs '"hf.Registrar.Roles=peer"' --tls.certfiles ${PWD}/organizations/fabric-ca/org1/tls-cert.pemIn this registration, besides the CA Name and TLS certificate of the CA Server, there are several items relevant to the identity of peer0.org1.example.com,
id.name: peer0, this is the name of entityid.secret: peer0pw, this is the secret to be used when enrol this entityid.type: peer, here is where we specify the type of this entity. The correct OU identifier will be set according to the config.yaml above.id.attrs: “hf.Registrar.Roles=peer”
This effectively puts the information into the Fabric-CA Server database for later enrollment. Nothing is received after registration.
Then we can use the name (id.name) and secret (id.secret) to enrol the peer0.org1.example.com, with this command.
fabric-ca-client enroll -u https://peer0:peer0pw@localhost:7054 --caname ca-org1 -M ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp --csr.hosts peer0.org1.example.com --tls.certfiles ${PWD}/organizations/fabric-ca/org1/tls-cert.pemBesides the CA Name and TLS certificate of the CA Server, we need to specify
- CA’s URL with entity’s name and secret, in the format as
https://<name>:<secret>@<CA’s URL> - the MSP location (local directory) to store the crypto material received
- the host in the default CSR (in
fabric-ca-client.yaml) is changed topeer0.org1.example.com. This is to override the default setting.
Now the crypto material for peer0.org1.example.com is stored in organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp/, as specified in the enroll command.
The directory is familiar, as it is the same directory structure we observe before. Here we just show the certificate (inside signcerts/).
Besides the subject and issuer, we also see some more information about the certificate of peer0.org1.example.com. Take a note on the X509v3 Subject Alternative Name with peer0.org1.example.com, and also the attributes specified in registration and enrollment.
We also need to enrol a TLS server cert.
fabric-ca-client enroll -u https://peer0:peer0pw@localhost:7054 --caname ca-org1 -M ${PWD}/organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls --enrollment.profile tls --csr.hosts peer0.org1.example.com --csr.hosts localhost --tls.certfiles ${PWD}/organizations/fabric-ca/org1/tls-cert.pemThe command is similar to enrol the credential for identity. The only difference is that the enrollment.profile is tls, and CSR hosts.
The result is stored in organizations/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/, as specified in the enroll command.
Again we just see the TLS server certificate (in signcerts/).
Note the X509v3 Extended Key Usage. As we specify enrollment profile as TLS, we see TLS Web Server Authentication and TLS Web Client Authentication here.
This is for peer0.org1.example.com. The process is repeated for Admin and User1. Note that the network.sh script does not enrol the TLS client certificate for them.
Step 4: Move material to the right directory
The directory of identity is kept inside <entity>/msp/, and the structure is good for use in docker-compose files. However the directory of <entity>/tls/ is not the format. Therefore we need to move TLS CA (inside tlscacerts/), server cert (inside signcerts/) and server key (inside keystore/) out and fit them into the <entity>/tls/ with proper names.
Also copy the CA certificates, we can reconstruct the desired directory structure. This is good enough for bringing up containers with docker-compose.
You can read the remaining steps in network.sh and registerEnroll.sh script to complete the whole process.
Summary
In this article, we use Test Network to show the two different ways of generation crypto material for a consortium network. While cryptogen is a fast and easy way with minimum configuration to bring up a workable network, it has certain limitations and therefore is always considered for testing purposes. A more practical way for production is to bring up a CA server and generate the material according to one’s need in a consortium network. In the Test Network, we see how Fabric-CA Server is used as the CA Server, and how to use Fabric-CA Client to perform the whole process.
