How to implement Hyperledger Fabric External Chaincodes within a Kubernetes cluster
Tags: blockchain, kubernetes, docker, hyperledger
Published: 2020-07-21 14:54:31

Recently released Hyperledger Fabric 2.0 was very welcomed as it introduced some interesting features on how to manage the chaincode with its new decentralized governance for chaincodes, private data enhancements and the ability to use the External chaincode launcher.

The last feature mentioned finally eliminates the docker dependency for launching chaincodes, which caused many difficulties in setting the blockchain networks in some environments, specially when not using the **docker-compose** setting that is used as tutorials on the [Hyperledger Fabric official documentation](https://hyperledger-fabric.readthedocs.io/en/release-2.0/test_network.html).

Since the distribution of the blockchain components are done through Docker, many use cases have been developed using Kubernetes as the container orchestration platform to launch the entire or some part of the blockchain network, and the dependency of the Docker daemon in the peer leaded to some configurations that might not be acceptable in production environments, such as the use of the Docker Socket of the worker nodes in the peer to deploy the chaincode containers, hence making these containers out of Kubernetes domain, or the use of **Docker-in-Docker (DinD)** solutions, which required these containers to be privileged.

With these constraints and limitations, the newly released version enables the preferred way on how these chaincodes are deployed, and in this tutorial, jointly made with Laura Esbri, we are going to see how to deploy them using the **External Chaincodes launcher**.

# Prerequisites

The following is a list of what we are going to need in order to make a simple Fabric network work within a dev Kubernetes cluster.

* **A Kubernetes cluster**. It can perfectly be one cluster using simple tools such as `minikube` or a single-node `kubeadm` cluster. The latter is the environment we are going to showcase here. You will need the tools to manage a Kubernetes cluster such as `kubectl`. Setting up a K8s cluster is out of scope in this article.

* **Hyperledger Fabric 2.2.0 docker images**. The images will be pulled when we are launching the deployments of the Kubernetes yaml descriptors.

* **Hyperledger Fabric 2.2.0 binaries**. We will need them to create the `crypto-config` and the `channel-artifacts`. You can download them in this [link](https://github.com/hyperledger/fabric/releases/download/v2.2.0/hyperledger-fabric-linux-amd64-2.2.0.tar.gz).

* You will find the repository with all the files used in this tutorial here: https://github.com/vanitas92/fabric-external-chaincodes

# Install the binaries

Install the binaries required using the following instructions:

```
wget https://github.com/hyperledger/fabric/releases/download/v2.1.0/hyperledger-fabric-linux-amd64-2.2.0.tar.gz

tar -xzf hyperledger-fabric-linux-amd64-2.2.0.tar.gz

# Move to the bin path
mv bin/* /bin

# Check that you have successfully installed the tools by executing

configtxgen --version

# Should print the following output:
# configtxgen:
#  Version: 2.2.0
#  Commit SHA: 5ea85bc54
#  Go version: go1.14.4
#  OS/Arch: linux/amd64
```

# Launching the network

Once we have the Kubernetes cluster up and ready, we will launch the network. But first we need to generate the basic crypto material necessary to establish the identity and the genesis block of the network. There has been some changes in the `configtx.yaml` file that are needed to be implemented in order to make the new chaincode lifecycle work, which are necessary to use the External Chaincode Launcher.

The blockchain network will consist of a RAFT orderer service with 3 instances, 2 organisations that have a peer for each one (*org1* and *org2*) with a CA for each organisation. This is already encoded in the `configtx.yaml` and `crypto-config.yaml` and there is no need to modify these files.

There are a few new configuration options on `configtx.yaml` as the new lifecycle and endorsement policies. These options can be defined in this file to set which role is allowed to sign when an endorsement action is happening in the network. In this case, we have set this to the peers that belong to the organization MSP, with the `Endorsement` policy option:

```yaml
Organizations:
    - &org1
        # DefaultOrg defines the organization which is used in the sampleconfig
        # of the fabric.git development environment
        Name: org1MSP

        # ID to load the MSP definition as
        ID: org1MSP

        MSPDir: crypto-config/peerOrganizations/org1/msp

        # Policies defines the set of policies at this level of the config tree
        # For organization policies, their canonical path is usually
        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
        Policies: &org1Policies
            Readers:
                Type: Signature
                Rule: "OR('org1MSP.admin', 'org1MSP.peer', 'org1MSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('org1MSP.admin', 'org1MSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('org1MSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('org1MSP.peer')"
                
    - &org2
        # DefaultOrg defines the organization which is used in the sampleconfig
        # of the fabric.git development environment
        Name: org2MSP

        # ID to load the MSP definition as
        ID: org2MSP

        MSPDir: crypto-config/peerOrganizations/org2/msp

        # Policies defines the set of policies at this level of the config tree
        # For organization policies, their canonical path is usually
        #   /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
        Policies: &org2Policies
            Readers:
                Type: Signature
                Rule: "OR('org2MSP.admin', 'org2MSP.peer', 'org2MSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('org2MSP.admin', 'org2MSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('org2MSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('org2MSP.peer')"

        AnchorPeers:
            # AnchorPeers defines the location of peers which can be used
            # for cross org gossip communication.  Note, this value is only
            # encoded in the genesis block in the Application section context
            - Host: peer0-org2
              Port: 7051
            # - Host: peer1-org2
            #   Port: 7051
```

After that, we can set the application capabilities to set the signing policies when there is an endorsement happening. There are two rules to be defined, the `LifecycleEndorsement` and the `Endorsement` rules. We set them to the `MAJORITY` rule, which indicates that any endorsement must be approved by more than half of the network participants (50% + 1 signatures required):

```yaml
Application: &ApplicationDefaults
    # Organizations is the list of orgs which are defined as participants on
    # the application side of the network
    Organizations:

    # Policies defines the set of policies at this level of the config tree
    # For Application policies, their canonical path is
    #   /Channel/Application/<PolicyName>
    Policies: &ApplicationDefaultPolicies
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"
        LifecycleEndorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
        Endorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
```

Once you are all set with the configuration of the network, you can now generate the crypto materials and the genesis block of the blockchain network. We will use the `fabricOps.sh` script with the following command:

> **Note**: Make sure that this script has execution permissions. Changing to 744 permision should be enough.

```sh
$ ./fabricOps.sh start
```

This will create all the crypto materials such as all the certificates of the peers orderers and CAs, as well as the genesis block of the channel.

We will first need to create a new namespace for the Hyperledger Fabric workload, create one with the following command:

```sh
$ kubectl create ns hyperledger
```

Create a folder in the VM for storing the persistent data for all the Hyperledger Fabric workload:

```sh
mkdir /home/storage
```

Once we have created the namespace, we are ready to deploy the workload:

> **Note**: Check and modify if needed the hostpath of the volumes attached on the orderers’ deployment, the yamls use absolute paths and it is assumed here that the repository is in the `/home` folder.

```sh
$ kubectl create -f orderer-service/
```

Check that everything is running fine by issuing the following command:

```sh
$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                        READY   STATUS    RESTARTS   AGE
orderer0-58666b6bd7-pflf7   1/1     Running   0          5m47s
orderer1-c4fd65c7d-c27ll    1/1     Running   0          5m47s
orderer2-557cb7865-wlcmh    1/1     Running   0          5m47s
```

> **Note 2**: Again, check and modify if needed the hostpath of the volumes attached on the peer, CA and CLI deployments, the yamls use absolute paths and it is assumed here that the repository is in the `/home` folder.

Now create the **org1** workload, which will deploy the peer and the CA of this organisation:

```
$ kubectl create -f org1/
```

Check that everything is running fine by issuing the following command:

```
$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                          READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-9px4s      1/1     Running   0          19m
cli-org1-bc9f895f6-zmmdc      1/1     Running   0        2m56s
orderer0-58666b6bd7-pflf7     1/1     Running   0          79m
orderer1-c4fd65c7d-c27ll      1/1     Running   0          79m
orderer2-557cb7865-wlcmh      1/1     Running   0          79m
peer0-org1-798b974467-vv4zz   1/1     Running   0          19m
```

Repeat the same steps with **org2** workload:

> **Note 3**: Again, check and modify if needed the hostpath of the volumes attached on the peer, CA and CLI deployments, the yamls use absolute paths and it is assumed here that the repository is in the `/home` folder.

```
$ kubectl create -f org2/
```

Check that everything is running fine by issuing the following command:

```
$ kubectl get pods -n hyperledger

### Should print a similar output
NAME                          READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-9px4s      1/1     Running   0          71m
ca-org2-7454f69c48-q8lft      1/1     Running   0        2m20s
cli-org1-bc9f895f6-zmmdc      1/1     Running   0          55m
cli-org2-7779cc8788-8q4ns     1/1     Running   0        2m20s
orderer0-58666b6bd7-pflf7     1/1     Running   0         131m
orderer1-c4fd65c7d-c27ll      1/1     Running   0         131m
orderer2-557cb7865-wlcmh      1/1     Running   0         131m
peer0-org1-798b974467-vv4zz   1/1     Running   0          71m
peer0-org2-5849c55fcd-mbn5h   1/1     Running   0        2m19s
```

All workload above should appear right now in the *hyperledger* namespace.

> **Note 4**: If there are CrashloopBackoff errors, check that the paths are correctly set and crypto files are in place and correctly set. Check the logs by issuing the `kubectl logs pod_name -n hyperledger` command, pointing to the faulty pod.

# Setting up the network channels

Once all workload is deployed, we are ready to create the channel and join the peers to the channel. Enter in the cli pod of the **org1**:

```
$ kubectl exec -it cli_org1_pod_name sh -n hyperledger
```

Once inside the cli pod, execute the following command:

```
$ peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA

### Should print a similar output
2020-03-06 11:54:57.582 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 11:54:58.903 UTC [cli.common] readBlock -> INFO 002 Received block: 0
```

The channel `mychannel` is created and ready to be used. Next, join the peer of **org1** to the channel:

```
$ peer channel join -b mychannel.block

### Should print a similar output
2020-03-06 12:01:41.608 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:01:41.688 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
```

We are going to perform the same steps on **org2** cli, but since the channel is already created by **org1**, we are going to fetch the genesis block from the orderer service. Enter the pod first:

```
$ kubectl exec -it cli_org2_pod_name sh -n hyperledger
```

Once inside the cli pod, execute the following command:

```
$ peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-06 12:18:14.880 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:18:14.895 UTC [cli.common] readBlock -> INFO 002 Received block: 0
```

Then join to the channel from the genesis block:

```
$ peer channel join -b mychannel.block

### Should print a similar output
2020-03-06 12:20:41.475 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-03-06 12:20:41.561 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
```

You can check anytime if any of the peers has joined the channel by executing the following command:

```
$ peer channel list

### Should print a similar output
2020-03-06 12:22:41.102 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Channels peers has joined: 
mychannel
```

# Installing the External Chaincode

Now we are going to do the interesting stuff 😄. We are going to deploy the marbles chaincode as an External chaincode. You can find the original chaincode in the [fabric-sample Github repository](https://github.com/hyperledger/fabric-samples/tree/master/chaincode/marbles02/go), but in order to deploy it as External Chaincode, we need to perform some changes in the imports and the init function to declare it as external chaincode server.

The new version of Hyperledger also comes with the new chaincode lifecycle process for installing a chaincode on your peers and starting it on a channel. In order to use the [External Chaincodes](https://hyperledger-fabric.readthedocs.io/en/release-2.2/cc_launcher.html) feature, we have to use this new process as well.

The first thing we have to do is to configure the peer to process external chaincode. The external builders are merely based on buildpacks and so we have create 3 scripts, `detect`, `build` and `release`. These 3 scripts have to be defined inside the peer container. These scripts can be found in the `buildpack/bin` folder.

> **Note 5**: Make sure that these 3 scripts have execution permissions in the `buildpack/bin` folder. The scripts are executed in the peer pod as root so changing to 744 permission should be enough. Failing to do so will cause the externalbuilder to fail its execution.

The definition of the builders can be found inside the org1 folder in the `builders-config.yaml`. This file has all the default options to configure the peer of the `core.yaml` except the *core_chaincode_externalbuilders* option, which has a custom builder configuration like the following.

```yaml
            # List of directories to treat as external builders and launchers for
            # chaincode. The external builder detection processing will iterate over the
            # builders in the order specified below.
            externalBuilders:
              - name: external-builder
                path: /builders/external
                environmentWhitelist:
                   - GOPROXY
```

Because the Kubernetes deployment descriptors of the peers have environment variables to configure the option, these environment variables override the default option from the `core.yaml`. This was done this way because the environment variables do not accept array formats and this way is able to configure each peer with his own configuration but establish this builder. It is used by all organizations in this network.

The new lifecycle process packages the chaincode and installs it in a different way than previous versions. Since we are using the external chaincode feature, the chaincode code does not have to be compiled and installed within the peer pod itself, rather be on another pod. The only code to be installed in the peer process is the information required to be able to connect to the external chaincode process.

> **Note 6**: We are going to perform the steps for the **org1** peer chaincode, hence execute the commands on the **org1** cli pod. These steps will be repeated for **org2** peer but changing some configuration of the chaincode later.

To achieve this, we need to [package the chaincode](https://hyperledger-fabric.readthedocs.io/en/release-2.2/cc_service.html#packaging-chaincode) with some requirements. There has to be a `connection.json` file that contains the information of the connection to the external chaincode server. This includes the address, TLS certificates and dial timeout configurations.

```json
{
    "address": "chaincode-marbles-org1.hyperledger:7052",
    "dial_timeout": "10s",
    "tls_required": false,
    "client_auth_required": false,
    "client_key": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----",
    "client_cert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
    "root_cert": "-----BEGIN CERTIFICATE---- ... -----END CERTIFICATE-----"
}
```

This file needs to be packaged into a tar file called `code.tar.gz`. Achieve this by executing the following command.

> **Note 6**: You can perform these commands inside the cli tools pod or outside the cli pod in your host machine where your chaincode hostpath storage points. In the end, the resulting tar files are needed in the cli pod to launch commands, so i recommend it doing it from the cli pod. If you are going to perform these commands in the cli pod, then go to this path: `/opt/gopath/src/github.com/marbles/packaging`

```
$ cd chaincode/packaging
$ tar cfz code.tar.gz connection.json
```

Once we have the `code.tar.gz` file, then we have to repackage it again alongside with another file called `metadata.json`, which includes information about the type of chaincode to be processed, the path where the chaincode resides and the label we want to give to that chaincode.

```json
{"path":"","type":"external","label":"marbles"}
```

We package this two files with the following command:

```
$ tar cfz marbles-org1.tgz code.tar.gz metadata.json
```

When you got the tar file, you are ready to install it in the peer using the new lifecycle process.

```
$ peer lifecycle chaincode install marbles-org1.tgz

### Should print a similar output
2020-03-07 14:33:18.120 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nGdmarbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064\022\006marbles" > 
2020-03-07 14:33:18.126 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: marbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064
```

Copy the chaincode code package identifier as we will need it later. Nevertheless, you can always retrieve it back by executing the following command:

```
$ peer lifecycle chaincode queryinstalled

### Should print a similar output
Installed chaincodes on peer:
Package ID: marbles:030eec59c7d74fbb4e9fd57bbd50bb904a715ffb9de8fea85b6a6d4b8ca9ea12, Label: marbles
```

Now we are going to repeat the same steps above for **org2** but since we want to have a different pod to serve the same chaincode for **org2** peer, we have to change the address config option in `connection.json` file. Modify the file address value like this:

```json
"address": "chaincode-marbles-org2.hyperledger:7052",
```

Then repeat the same steps as before, executing the install command in the **org2** cli pod:

```
$ rm -f code.tar.gz
$ tar cfz code.tar.gz connection.json
$ tar cfz marbles-org2.tgz code.tar.gz metadata.json
$ peer lifecycle chaincode install marbles-org2.tgz

### Should print a similar output
2020-03-07 15:10:15.093 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nGmarbles:c422c797444e4ee25a92a8eaf97765288a8d68f9c29cedf1e0cd82e4aa2c8a5b\022\006marbles" > 
2020-03-07 15:10:15.093 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: marbles:c422c797444e4ee25a92a8eaf97765288a8d68f9c29cedf1e0cd82e4aa2c8a5b
```

Copy the chaincode code package identifier as before. It should have a different hash from the **org1** as we changed the address for **org2**.

What happened in the peer while installing the chaincode? If there are some external builders or launchers defined in the peer, then they get executed before attempting the classic option of building the Docker container by the peer itself. As we have defined the `external-builder`, the following order of scripts are executed.

The `detect` script simply evaluates whether the chaincode that is getting installed has the `type` key of `metadata.json` as a value `external`. If that script fails then the peer considers that this external builder does not have to build the chaincode and iterates to other external builders until there are no more defined and fallback to the Docker building process.

```sh
#!/bin/sh

# The bin/detect script is responsible for determining whether or not a buildpack 
# should be used to build a chaincode package and launch it. 
# 
# The peer invokes detect with two arguments:
# bin/detect CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR
#
# When detect is invoked, CHAINCODE_SOURCE_DIR contains the chaincode source and 
# CHAINCODE_METADATA_DIR contains the metadata.json file from the chaincode package installed to the peer. 
# The CHAINCODE_SOURCE_DIR and CHAINCODE_METADATA_DIR should be treated as read only inputs. 
# If the buildpack should be applied to the chaincode source package, detect must return an exit code of 0; 
# any other exit code will indicate that the buildpack should not be applied.

CHAINCODE_METADATA_DIR="$2"

set -euo pipefail

# use jq to extract the chaincode type from metadata.json and exit with
# success if the chaincode type is golang
if [ "$(cat "$CHAINCODE_METADATA_DIR/metadata.json" | sed -e 's/[{}]/''/g' | awk -F"[,:}]" '{for(i=1;i<=NF;i++){if($i~/'type'\042/){print $(i+1)}}}' | tr -d '"')" = "external" ]; then
    exit 0
fi

exit 1
```

If the `detect` script succeeds, then the `build` script is called. This script should generate the artifacts or binaries if we were building the chaincode inside the peer, but as we want to have as external service, it is enough to simply copy the `connection.json` file to the build output directory.

```sh
#!/bin/sh

# The bin/build script is responsible for building, compiling, or transforming the contents 
# of a chaincode package into artifacts that can be used by release and run.
#
# The peer invokes build with three arguments:
# bin/build CHAINCODE_SOURCE_DIR CHAINCODE_METADATA_DIR BUILD_OUTPUT_DIR
#
# When build is invoked, CHAINCODE_SOURCE_DIR contains the chaincode source and 
# CHAINCODE_METADATA_DIR contains the metadata.json file from the chaincode package installed to the peer.
# BUILD_OUTPUT_DIR is the directory where build must place artifacts needed by release and run. 
# The build script should treat the input directories CHAINCODE_SOURCE_DIR and 
# CHAINCODE_METADATA_DIR as read only, but the BUILD_OUTPUT_DIR is writeable.

CHAINCODE_SOURCE_DIR="$1"
CHAINCODE_METADATA_DIR="$2"
BUILD_OUTPUT_DIR="$3"

set -euo pipefail

#external chaincodes expect connection.json file in the chaincode package
if [ ! -f "$CHAINCODE_SOURCE_DIR/connection.json" ]; then
    >&2 echo "$CHAINCODE_SOURCE_DIR/connection.json not found"
    exit 1
fi

#simply copy the endpoint information to specified output location
cp $CHAINCODE_SOURCE_DIR/connection.json $BUILD_OUTPUT_DIR/connection.json

if [ -d "$CHAINCODE_SOURCE_DIR/metadata" ]; then
    cp -a $CHAINCODE_SOURCE_DIR/metadata $BUILD_OUTPUT_DIR/metadata
fi

exit 0
```

Finally, once the `build` script completes, the `release` script gets called. This script is responsible for providing the `connection.json` to the peer by placing it in the release output directory. Hence, the peer now knows where to call the chaincode if some invocations of that chaincode are to be executed.

```sh
#!/bin/sh

# The bin/release script is responsible for providing chaincode metadata to the peer. 
# bin/release is optional. If it is not provided, this step is skipped. 
#
# The peer invokes release with two arguments:
# bin/release BUILD_OUTPUT_DIR RELEASE_OUTPUT_DIR
#
# When release is invoked, BUILD_OUTPUT_DIR contains the artifacts 
# populated by the build program and should be treated as read only input. 
# RELEASE_OUTPUT_DIR is the directory where release must place artifacts to be consumed by the peer.

set -euo pipefail

BUILD_OUTPUT_DIR="$1"
RELEASE_OUTPUT_DIR="$2"

# copy indexes from metadata/* to the output directory
# if [ -d "$BUILD_OUTPUT_DIR/metadata" ] ; then
#    cp -a "$BUILD_OUTPUT_DIR/metadata/"* "$RELEASE_OUTPUT_DIR/"
# fi

#external chaincodes expect artifacts to be placed under "$RELEASE_OUTPUT_DIR"/chaincode/server
if [ -f $BUILD_OUTPUT_DIR/connection.json ]; then
   mkdir -p "$RELEASE_OUTPUT_DIR"/chaincode/server
   cp $BUILD_OUTPUT_DIR/connection.json "$RELEASE_OUTPUT_DIR"/chaincode/server

   #if tls_required is true, copy TLS files (using above example, the fully qualified path for these fils would be "$RELEASE_OUTPUT_DIR"/chaincode/server/tls)

   exit 0
fi

exit 1
```

# Building and deploying the External Chaincode

Once we have the chaincodes installed in the peers, we can now build them and deploy them in our Kubernetes cluster. Let’s look at the changes we need to implement to the chaincodes.

As the chaincodes do not vendor and include the modules needed currently, the imports have changed, so there is the need to import them and vendoring the modules.

```go
import (
	"bytes"
	"encoding/json"
	"fmt"
	"strconv"
	"strings"
	"time"
	"os"

	"github.com/hyperledger/fabric-chaincode-go/shim"
	pb "github.com/hyperledger/fabric-protos-go/peer"
)
```

It is important to also define the `go.mod` file to establish the version of the modules before building the code. The latest builds in the shim modules are needed to enable the External Chaincode property.

```go
module github.com/marbles
  
go 1.12

require (
        github.com/hyperledger/fabric-chaincode-go v0.0.0-20200128192331-2d899240a7ed
        github.com/hyperledger/fabric-protos-go v0.0.0-20200124220212-e9cfc186ba7b
        golang.org/x/net v0.0.0-20200202094626-16171245cfb2 // indirect
        golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4 // indirect
        golang.org/x/text v0.3.2 // indirect
        google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5 // indirect
)
```

The other change is the `Chaincode Server` property which listens to a particular address and port that we want to establish. As we might want to change them by using Kubernetes yaml descriptors, we are going to use the os module to get the value of the address and the chaincode code package identifier (**CCID**). By doing this we can deploy the same image of the code but changing the parameters as needed.

```go
func main() {

	server := &shim.ChaincodeServer{
		CCID:    os.Getenv("CHAINCODE_CCID"),
		Address: os.Getenv("CHAINCODE_ADDRESS"),
		CC:      new(SimpleChaincode),
		TLSProps: shim.TLSProperties{
				Disabled: true,
		},
	}

	// Start the chaincode external server
	err := server.Start()

	if err != nil {
		fmt.Printf("Error starting Marbles02 chaincode: %s", err)
	}
}
```

We are now ready to build the chaincode container using the following `Dockerfile`:

```Dockerfile
# This image is a microservice in golang for the Marbles chaincode
FROM golang:1.14.6-alpine AS build

COPY ./ /go/src/github.com/marbles
WORKDIR /go/src/github.com/marbles

# Build application
RUN go build -o chaincode -v .

# Production ready image
# Pass the binary to the prod image
FROM alpine:3.11 as prod

COPY --from=build /go/src/github.com/marbles/chaincode /app/chaincode

USER 1000

WORKDIR /app
CMD ./chaincode
```

The chaincode is built using a golang alpine image. Once the building stage is complete, the binary is transferred to a bare alpine image for a smaller and secure image. Execute the following command:

```
$ docker build -t chaincode/marbles:1.0 .
```

If everything goes right, you should have the image ready to be deployed. Modify the files of the chaincode deployment ( `org1-chaincode-deployment.yaml` and `org2-chaincode-deployment.yaml`) the **CHAINCODE_CCID** variable to the values you got before when installing the chaincode respectively.

```yaml
#---------------- Chaincode Deployment ---------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chaincode-marbles-org1
  namespace: hyperledger
  labels:
    app: chaincode-marbles-org1
spec:
  selector:
    matchLabels:
      app: chaincode-marbles-org1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: chaincode-marbles-org1
    spec:
      containers:
        - image: chaincode/marbles:1.0
          name: chaincode-marbles-org1
          imagePullPolicy: IfNotPresent
          env:
            - name: CHAINCODE_CCID
              value: "marbles:d8140fbc1a0903bd88611a96c5b0077a2fdeef00a95c05bfe52e207f5f9ab79d"
            - name: CHAINCODE_ADDRESS
              value: "0.0.0.0:7052"
          ports:
            - containerPort: 7052
```

After that, deploy them:

```
$ kubectl create -f chaincode/k8s
```

The service and deployments will be deployed in the same k8s namespace as the other workload.

```
$ kubectl get pods -n hyperledger
NAME                                      READY   STATUS    RESTARTS   AGE
ca-org1-84945b8c7b-tx59g                  1/1     Running   0          19h
ca-org2-7454f69c48-nfzsq                  1/1     Running   0          19h
chaincode-marbles-org1-6fc8858855-wdz7z   1/1     Running   0          20m
chaincode-marbles-org2-77bf56fdfb-6cdfm   1/1     Running   0          14m
cli-org1-589944999c-cvgbx                 1/1     Running   0          19h
cli-org2-656cf8dd7c-kcxd7                 1/1     Running   0          19h
orderer0-5844bd9bcc-6td8c                 1/1     Running   0          46h
orderer1-75d8df99cd-6vbjl                 1/1     Running   0          46h
orderer2-795cf7c4c-6lsdd                  1/1     Running   0          46h
peer0-org1-5bc579d766-kq2qd               1/1     Running   0          19h
peer0-org2-77f58c87fd-sczp8               1/1     Running   0          19h
```

Now we have to approve the chaincodes for each organization. This is a new feature of the chaincode lifecycle process, each organization has to agree to approve a new definition of a chaincode. We are going to approve the marble chaincode definition for **org1**. Execute this command inside the **org1** cli pod, remember to change the **CHAINCODE_CCID**:

```
$ peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --init-required --package-id marbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-08 10:02:46.192 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [4d81ea5fd494e9717a0c860812d2b06bc62e4fc6c4b85fa6c3a916eee2c78e85] committed with status (VALID)
```

You can check the state of approvals throughout the entire network by executing the following command:

```
$ peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-required --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
Chaincode definition for chaincode 'marbles', version '1.0', sequence '1' on channel 'mychannel' approval status by org:
org1MSP: true
org2MSP: false
```

Now let’s approve for **org2**. Execute this command inside the org2 cli pod, remember to change the **CHAINCODE_CCID**:

```
$ peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --init-required --package-id marbles:25a9f6fe26161d29af928228ca1db0c41892e26e46335c84952336ee26d1fd93 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
2020-03-08 10:26:43.992 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [74a89f3c93c10f14c626bd4d6cb654b37889908c9e6f7b983d2cad79f1e82267] committed with status (VALID)
```

Check again the commit readiness of the chaincode:

```
$ peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-required --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

### Should print a similar output
Chaincode definition for chaincode 'marbles', version '1.0', sequence '1' on channel 'mychannel' approval status by org:
org1MSP: true
org2MSP: true
```

Now that we have all the approvals of all the organizations, let’s commit the definition of this chaincode in the channel. You can do this operation on any of the peers:

```
$ peer lifecycle chaincode commit -o orderer0:7050 --channelID mychannel --name marbles --version 1.0 --sequence 1 --init-required --tls true --cafile $ORDERER_CA --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt

### Should print a similar output
2020-03-08 14:13:49.516 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [568cb81f821698025bbc61f4c6cd3b4baf1aea632e1e1a8cfdf3ec3902d1c6bd] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:13:49.533 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [568cb81f821698025bbc61f4c6cd3b4baf1aea632e1e1a8cfdf3ec3902d1c6bd] committed with status (VALID) at peer0-org2:7051
```

Now we have the chaincode definition added to the channel and ready to be invoked and make queries against it! 😄

# Testing the External Chaincode

We can test the chaincodes making invoke and query commands from the cli pod. These commands are not modified by the lifecycle chaincode process and can be called as the chaincodes in version 1.x of Hyperledger Fabric. First, let’s create some marbles in the ledger. Execute the following command on one cli pod, either **org1** or **org2**:

```
$ peer chaincode invoke -o orderer0:7050 --isInit --tls true --cafile $ORDERER_CA -C mychannel -n marbles --peerAddresses 
peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer
0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["initMarble
","marble1","blue","35","tom"]}' --waitForEvent

### Should print a similar output
2020-03-08 14:23:03.569 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [83aeeaac47cf6302bc139addc4aa38116a40eaff788846d87cc815d2e1318f44] committed with status (VALID) at peer0-org2:7051
2020-03-08 14:23:03.575 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [83aeeaac47cf6302bc139addc4aa38116a40eaff788846d87cc815d2e1318f44] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:23:03.576 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 003 Chaincode invoke successful. result: status:200
```

This will initiate the ledger, the --isInit will call to the legacy `init` function to start the ledger. Now create the first marble:

```
$ peer chaincode invoke -o orderer0:7050 --isInit --tls true --cafile $ORDERER_CA -C mychannel -n marbles --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["initMarble","marble1","red","45","tom"]}' --waitForEvent

### Should print a similar output
2020-03-08 14:23:40.404 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [8391f9f8ea84887a56f99e4dc4501eaa6696cd7bd6c524e4868bd6cfd5b85e78] committed with status (VALID) at peer0-org2:7051
2020-03-08 14:23:40.434 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [8391f9f8ea84887a56f99e4dc4501eaa6696cd7bd6c524e4868bd6cfd5b85e78] committed with status (VALID) at peer0-org1:7051
2020-03-08 14:23:40.434 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 003 Chaincode invoke successful. result: status:200
```

The first c

Retrieve information from marble1:

```
$ peer chaincode query -C mychannel -n marbles -c '{"Args":["readMarble","marble1"]}'

{"docType":"marble","name":"marble1","color":"red","size":45,"owner":"tom"}
```

There are many example commands you can execute with this chaincode, just check the source code of the chaincode for more examples.

You can also check the logs of the chaincode containers by executing the following command:

```
$ kubectl logs chaincode_pod_name -n hyperledger

### Should print a similar output
invoke is running initMarble
- start init marble
- end init marble
invoke is running initMarble
- start init marble
- end init marble
invoke is running readMarble
```

# Bonus Track! ContractApi based Chaincodes with External Chaincodes feature

As a new addition that we have been playing lately, we can now deploy external chaincodes with the new ContractApi introduced in version 2.0, which makes it even more easy to develop chaincodes. We will deploy the [fabcar](https://github.com/hyperledger/fabric-samples/blob/master/chaincode/fabcar/go/fabcar.go) chaincode, you can find the original here. The modified one would be pushed in the repo under the folder named `fabcar`.

We will need to do add the new shim api which allows to add the External Chaincode server interface that allows the external communication with the peer.

```go
package main

import (
	"encoding/json"
	"fmt"
	"strconv"
	"os"

	"github.com/hyperledger/fabric-chaincode-go/shim"
	"github.com/hyperledger/fabric-contract-api-go/contractapi"
)

// SmartContract provides functions for managing a car
type SmartContract struct {
	contractapi.Contract
}

// Car describes basic details of what makes up a car
type Car struct {
	Make   string `json:"make"`
	Model  string `json:"model"`
	Colour string `json:"colour"`
	Owner  string `json:"owner"`
}

// QueryResult structure used for handling result of query
type QueryResult struct {
	Key    string `json:"Key"`
	Record *Car
}

// Main function to start the external chaincode server
func main() {
	cc, err := contractapi.NewChaincode(new(SmartContract))

	if err != nil {
		fmt.Println("Error starting a new ContractApi Chaincode:", err)
	}
	
	server := &shim.ChaincodeServer{
		CCID:    os.Getenv("CHAINCODE_CCID"),
		Address: os.Getenv("CHAINCODE_ADDRESS"),
		CC:      cc,
		TLSProps: shim.TLSProperties{
				Disabled: true,
		},
	}
	
	// Start the chaincode external server
	err = server.Start()

	if err != nil {
		fmt.Println("Error starting FabCar chaincode server:", err)
	} else {
		fmt.Println("Succesfully started new Fabcar Chaincode server with the new ContractApi")
	}
}
```

The `go.mod` file would need to import these two modules as well.

```go
module github.com/fabcar
  
go 1.14

require (
        github.com/hyperledger/fabric-chaincode-go v0.0.0-20200424173110-d7076418f212
        github.com/hyperledger/fabric-contract-api-go v1.1.0
        golang.org/x/net v0.0.0-20200202094626-16171245cfb2 // indirect
        golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4 // indirect
        golang.org/x/text v0.3.2 // indirect
        google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5 // indirect
)
```

And that would be all modifications needed in order to make it work! The instructions to install it works just the same but with the word fabcar instead of marbles. The only difference is that there is no need for `--init-required` or `--isInit` flag in the `approveformyorg` , `commit` and `invoke` commands.

```
peer lifecycle chaincode approveformyorg --channelID mychannel --name fabcar --version 1.0 --package-id fabcar:005c35f4f172c056723eca09d41e8048e0beaa2712d920c19af837640df7e2aa --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode approveformyorg --channelID mychannel --name fabcar --version 1.0 --package-id fabcar:61ab817a6ad76098d340952e5d8e928d9c5ddff1a53341dc8d0c64b4345564b0 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name fabcar --version 1.0 --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA

peer lifecycle chaincode commit -o orderer0:7050 --channelID mychannel --name fabcar --version 1.0 --sequence 1 --tls true --cafile $ORDERER_CA --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt

peer chaincode invoke -o orderer0:7050 --tls true --cafile $ORDERER_CA -C mychannel -n fabcar --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt -c '{"Args":["InitLedger"]}' --waitForEvent

peer chaincode query -C mychannel -n fabcar -c '{"Args":["QueryAllCars"]}'
```

You would also find the original implementation of this same chaincode with the External Chaincode feature by the Hyperledger team in [their github repo](https://github.com/hyperledger/fabric-samples/tree/master/chaincode/fabcar/external). Not many differences are between their implementation and ours, so we assume that this is the proper way to implement this contract based chaincodes with external feature.

# Future work and improvements

There are some improvements that can be made and we are happy to receive feedback from the community to improve these External Chaincodes.

* **External Builders definition as environment variables rather than configmap**. Although it works this way, we would like to define the External Builders in the environment variables as other options. We tried to define the external builders as environment variables but since the ExternalBuilders configurations is expected as array and the environment variables only accept strings, we were having many errors. **UPDATE 2**: *Many of you have reported many problems with externalbuilder definition not being executed succesfully or properly detected. This was caused mainly for permissions on the scripts folder (execute permission required not enabled after cloning from GitHub). For ease of use, the buildpack scripts have now been included as a configmap and attached in the peer pods with greater kubernetes integration.*
    
* **Endorsement Policy failures**: We had to disable the Endorsement and the Lifecycle Endorsment policies due to many endorsement policy failures when committing the chaincode definition on the channel. As this might need to have some better understanding and definition of the policies in version 2.0, we did not have enough time to configure it properly. **UPDATE 1**: *This has now been solved and the article has been updated with the new requirements for making the endorsement policies and lifecycle endorsement policies coded in the genesis block.*
    
* **Manual changes on the address configuration of the chaincode**. The address value of the `connection.json` file needs to be changed every time we have a new organization. And also can change if we have multiple endorser peers for each organisation. There could be some interesting pipelines to help support the deployment of these type of chaincodes.

Thanks for taking a look into it!