Examining the Behaviour of Hyperledger Fabric when World State is Tampered
Overview
When we say “immutability” in blockchain, our general perception is that everything in the blockchain system is immutable, that is, no one can modify it. In fact, in a blockchain network, each node is maintaining both the state and the transaction logs. When things are running normally, the consensus mechanism ensures the state maintained in every node is identical, which makes blockchain a robust and trustworthy environment for business use.
Hyperledger Fabric has an implementation allowing a CouchDB as the world state database. It may become a big issue if someone can modify the data on CouchDB.
This article tries to explore what happens when data is tampered on the world state, and see how endorsing policy can have an impact on the fabric network. A proper endorsing policy can address the data tampering situation and protect the fabric network against bad behaviour.
Review: Ledger in Hyperledger Fabric
Data immutability is always one of the major features when we talk about blockchain. That is, no one can modify the data once it is written in the blockchain. To be more precise, the data written in the data structure (called blockchain) cannot be easily modified, and any attempt of modification should be easily detected.
If we take a look on Hyperledger Fabric, each peer node is keeping its own copy of ledger. The ledger of each node holds the identical content, which is the result of consensus mechanism of Hyperledger Fabric. Inside the ledger are two parts: the world state and the blockchain. World state keeps the current state (values) of data in the fabric network, while blockchain keeps transaction log in a blockchain data structure, that is, the endorsed transactions are placed into a block and the blocks are linked in sequence with hashing mechanism.
When we say immutability, it is the blockchain part, that is, the chain structure of endorsed and validated transactions, the history, unable to be modified easily. The world state, in contrast, keeps the latest result after those transactions are executed, and therefore cannot be said immutable. Nevertheless, the consensus mechanism will ensure that both the blockchain and world state are identical.
Of no surprise, both are implemented as databases. While they are implemented with LevelDB, the world state database can optionally be implemented in a CouchDB.. CouchDB provides additional features such as richer data types and querying capability. When implemented, it is a one-to-one mapping: one CouchDB for one peer node. Note that CouchDB themselves are not forming a cluster but working separately.
CouchDB, as other third party software, comes with its own tools. In fact this is also this part triggering the purpose of this article. There is a chance that the state stored in CouchDB of a node is tampered. What happens when the state is tampered? We will examine in the following sessions.
Setup of Testing Environment
As we are simulating the data tampering, we need a fabric network of multiple organizations. Here we first build a fabric network with three organizations (Org1, Org2 and Org3). Each organization has one peer (peer0.org1, peer0.org2 and peer0.org3). Each peer has one CouchDB as its world state database. Besides, we will have an orderer (in OrdererOrg) and a CLI for accessing to the peer nodes.
I here omit the detail configuration files and steps of bringing up the network and channel creation. You can refer to the repository for the environment setup.
After the nodes are up and running, we will see total eight containers and mychannel added. All the three peers join mychannel.
Baseline: A Normal Operation
We first perform a normal operation where data is not tampered. This serves as a baseline with which we can compare test scenarios later when data is tampered.
Note that CLI by default is pointing to peer0.org1. If we issue commands for peer0.org2 and peer0.org3, we need proper environment variables (specified by -e in docker exec commands).
Step 1: Install chaincode FabCar on all peers.
Step 2: Instantiate chaincode on mychannel, with endorsing policy. Here we first use AND(all peers), which means that transaction requires response from all peers before it is further processed.
Step 3: Invoke initLedger() to initialize the ledger with predefined dataset. Refer to the chaincode about the information of ten cars.
Step 4: Query CAR0 on all nodes and we see contents are identical.
Step 5: Invoke changeCarOwner() for CAR0 on peer0.org1. Then query CAR0 on all nodes and see the change happens in all nodes.
Data Tampering
In our setup, CouchDB is running in as a separate container. That means we can use the built-in utility to access and even modify the data written on it.
According to our setup, couchdb0, couchdb1 and couchdb2 expose port 5984, 6984 and 7984 to localhost, respectively. If we access the http://<couchdb>/_utils we can see database created. We see a database called mychannel_fabcar, and it is obviously our chaincode fabcar on the channel mychannel. If we go into mychannel_fabcar, we see the data set by initLedger(). It is where we can modify our database.
To modify a data, select the ID (e.g. CAR0), and update the content. After saving change, the data is now modified directly on CouchDB, without going through fabric network. During the test scenarios, the “colour” of CAR0 is changed from “blue” to “yellow”.
We will use this method to modify the data of CAR0, and then check using chaincode if the world state is updated.
NOTE: In real life, CouchDB should be well-protected and should not be accessed in this manner. This is just for testing and demo purpose.
Test Scenario 1: Endorsing Policy requires endorsement from ALL peers (i.e., AND)
Step 1: Chaincode installation in all peers (same Step 1 in baseline)
Step 2: Chaincode instantiation with endorsing policy ALL(org1, org2, org3)
Step 3: initLedger()
Step 4: We follow the same step above. After initLedger() is executed, we modify the “colour” of CAR0 from “blue” to “yellow”. After modification we query CAR0 on three nodes. Now we see peer0.org1 has been modified.
Now we have a tampered node peer0.org1.
Step 5: Invoke changeCarOwner() from the tampered node.
We see failure when invoking this method. It is due to inconsistence of received responses from all the three peer nodes, i.e., “Proposal Response Payloads do not match”. Obviously the response from peer0.org1 is not the same as peer0.org2 and peer0.org3 on the data.
To certain extent, this is what we wish to have. Once the world state of a node is tampered, the network should not proceed until the data tampering is removed. We will see how to handle this later.
Test Scenario 2: Endorsing Policy requires endorsement just from ONE peer (i.e., OR)
Step 1: Chaincode installation in all peers (same Step 1 in baseline)
Step 2: Chaincode instantiation with endorsing policy OR(org1, org2, org3)
Step 3: initLedger()
Step 4: We follow the same step above. After initLedger() is executed, we modify the “colour” of CAR0 from “blue” to “yellow”. After modification we query CAR0 on three nodes. Now we see peer0.org1 has been modified.
Step 5: Invoke changeCarOwner() from the tampered node.
We see a huge problem here: the tampered data (“colour”: “yellow”) is propagated to the whole fabric network from the tampered node. Therefore, under certain endorsing policy, the transaction invoked by the node with tampered data will cause propagation of tampered data. It is highly undesirable as it ruins the fabric network.
Observation
From these tests we see the importance of endorsing policies. Besides business reasons (for example, it makes sense that certain transactions should be endorsed by various organizations), endorsing policy also has security effect, in particular when data tampering happens.
In real life world state should not be modified easily and directly. However, when this happens, certain endorsing policy may protect the fabric network.
If security is our top priority, it is desired that fabric network should stop functioning when data tampering happens. It is to protect data stored in the fabric network against further tampering. Adding more endorsing peers helps. For example, in endorsing policy,
- use
ANDwith specified or all organizations - use >1
OutOfspecified or all organizations
In either case, the tampered node cannot successfully invoke chaincode methods. The fabric network is protected against propagation of tampered data.
The drawback is that the network cannot endorse transactions when it involves the tampered data, as the responses returned are no longer consistent (see our test scenario 1 above). However, as security is our top priority, we should accept this outcome, and find a proper way to resume the network.
How about endorsing policy using only one of specified or all organizations (i.e. OR)? There will be two possible cases.
- If the tampered node invokes transaction on the tampered data, the result is that tampered data gets propagated, as what we see in test scenario 2 above.
- If a normal node invokes transaction on the same piece of data being tampered in another node, it is interesting that the normal data will “fix” the tampered data in that tampered node. That is, the tampered data is changed back to normal data from that normal node.
We should not be happy with second case, in which there is a chance to fix the data tampering. There is still a chance that first case happens (in particular in a compromised peer node). It is highly undesirable to have the tampered data propagated. As a result, never use weak endorsing policy like OR in production fabric network.
How to Fix the Data Tampering
When we are on the Test Scenario 1, using AND(org1, org2, org3) as the endorsing policy. When data tampering on Org1 happens, the fabric network is unable to process transactions involving this tampered data.
Assuming that we have confirmed that it is peer0.org1 causing this issue. We should remove (kill) this peer (peer0.org1.example.com) and its CouchDB (couchdb0 in our example) from the network. If there is another peer (say peer1.org1) untampered, the fabric network should still function as this peer fulfils the endorsing policy requirement.
In our setup, unfortunately peer0.org1 is the only peer in Org1. We need to bring up the peer0.org1.example.com and couchdb0 in order to resume the fabric network.
Here we remove the tampered peer and database, and then bring back a new peer and a new database back to the network. As far as the node detail is unchanged, the peer can join back the fabric network and channel, and get back the transactions. Through applying the transactions upon an empty world state database, the peer should get back the same state as other peers. This means the peer is back to normal and able to handle upcoming transaction.
Step 1: Perform Test Scenario 1 with AND(org1, org2, org3) as endorsing policy and same data tampering happens. Now peer0.org1 cannot invoke transaction changeCarOwner().
Step 2: We kill both the peer0.org1.example.com and couchdb0
Step 3: We bring up peer0.org1.example.com and couchdb0 with docker-compose
Step 4: If we query CAR0 on this peer, we see that it is back to “blue”, which is the current state of the fabric network.
Step 5: Now peer0.org1 can invoke changeCarOwner() and the result is updated correctly across the whole fabric network.
Summary
The purpose of this article is to examine what happens if the world state is tampered. With proper endorsing policy we can somehow protect the fabric network from tampered data propagation. We also see how bad it will be in case a weak endorsing policy is applied to the network, in which the tampered data may get propagated. This ruins the fabric network as a trust system. Finally, by bringing down the tampered peer node and bringing it up again, the node recommitting back the blocks on the fabric network and can resume working.
