NSX-T
NSX Application Platform Part 2: Harbor Image Registry

NSX Application Platform Part 2: Harbor Image Registry

Harbor Image Registry Configuration

The Harbor image registry is critical to the successful deployment of the NSX Application Platform (NAPP). It holds all the images and binaries required for the application platform, which are pulled as pods are being deployed.

There are many ways to deploy Harbor on-premises including, but not limited to:

  • Using Helm to deploy a HA instance of Harbor on kubernetes
  • Tanzu Application Service with Harbor Integration
  • Deploying the Harbor image registry on a virtual machine

This guide will walkthrough deploying Harbor on a Ubuntu virtual machine.

Virtual Machine Configuration

This section details the specifications of the Ubuntu virtual machine that I have deployed for use as my Harbor image registry.

  • Ubuntu 20.0.4.3 Focal Fossa
  • 2 vCPU/8GB RAM
  • 2 Disks attached, one for the OS (40GB) and one for data (150GB)
  • Single network interface on my management network (IP address 192.168.63.100)
  • VM has internet access to generate certificates and pull files for install

Note: This article will not cover the Ubuntu deployment process.

Prerequisites – Installing Docker

Prior to installing Harbor, the Ubuntu VM must meet the prerequisites, all of which can be found here.

The official steps from Docker can be found here. The commands I ran are provided in the output below.

[email protected]:/# apt-get update

[email protected]:/# apt-get install \
>     ca-certificates \
>     curl \
>     gnupg \
>     lsb-release

[email protected]:/#  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

[email protected]:/#  echo \
>   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
>   $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

[email protected]:/# apt-get update  <--------- DO NOT SKIP THIS
[email protected]:/# apt-get install docker-ce docker-ce-cli containerd.io

Next, check that the service is running.

nsx napp harbor check docker service is running harbor image registry

Finally, I complete the prerequisite installation by installing Docker Compose.

[email protected]:/mnt/data/harbor# sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0  14434      0 --:--:-- --:--:-- --:--:-- 14434
100 11.6M  100 11.6M    0     0  14.6M      0 --:--:-- --:--:-- --:--:-- 25.8M
[email protected]:/mnt/data/harbor# sudo chmod +x /usr/local/bin/docker-compose
[email protected]:/mnt/data/harbor# docker-compose --version
docker-compose version 1.27.4, build 40524192

Harbor Repository Certificate Configuration

In this article I utilize a Lets Encrypt certificate for my Harbor registry.

Note: as of NSX-T 3.2 release (Impactor), self-signed certificates, including Active Directory issued certificates are not supported.

The certificates used for the registry must be signed by a trusted certificate authority. If you do not utilize a trusted certificate, when attempting to deploy NAPP, you will be faced with an x509 certificate issue, similar to the below output.

  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  11m                   default-scheduler  Successfully assigned cert-manager/cert-manager-69cc999bb5-khjws to impactorlab-workers-snmxl-dc89f6748-s9p4p
  Normal   Pulling    10m (x4 over 11m)     kubelet            Pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"
  Warning  Failed     10m (x4 over 11m)     kubelet            Failed to pull image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": rpc error: code = Unknown desc = failed to pull and unpack image "harbor.shank.com/impactor/clustering/third-part
y/cert-manager-controller:19067763": failed to resolve reference "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": failed to do request: Head "https://harbor.shank.com/v2/impactor/clustering/third-party/cert-manager-controller/manifests/19067763":
 x509: certificate signed by unknown authority
  Warning  Failed     10m (x4 over 11m)     kubelet            Error: ErrImagePull
  Warning  Failed     6m54s (x21 over 11m)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    108s (x44 over 11m)   kubelet            Back-off pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"

You will still be presented with this issue if you utilize the workaround here. This is because the underlying docker / containerd still doesn’t trust the certificate and chain. There is potential to work around this issue, however, I will not get into that here. If you absolutely require the use of a self-signed certificate, get in contact via Twitter, LinkedIn, or email.

Generating a Lets Encrypt Certificate

First, I install Lets Encrypt on my virtual appliance. I have listed the commands I ran in the output below. Remember, this command is being run on Ubuntu, you will need to edit the command to suit your operating system.

### Install the package and all its dependencies 
[email protected]:/home/harbor# apt install letsencrypt

### Check to ensure it is running
[email protected]:/home/harbor# systemctl status certbot.timer
nsx napp harbor check letsencrypt service

Now that Lets Encrypt is installed, I generate the certificate for the appliance. Keep in mind that this virtual appliance has internet connectivity, and I have created a DNS A record with my hosting provider for my appliance.

Issue the command below on your Harbor appliance, and follow the prompts on screen.

## Change the value after -d to match the hostname of your Harbor appliance.
[email protected]:/home/harbor# certbot certonly --standalone -d harbor.lab2prod.com.au
nsx napp harbor create certificate
harbor image registry

As per the prompt, the certificates have been generated and are available in /etc/letsencrypt/live/harbor.lab2prod.com.au/.

Setup and configure Harbor

Download the offline Harbor installer package located here, and transfer it to the Harbor appliance.

nsx napp harbor offline installer

Extract the archive using the command below.

[email protected]:/mnt/data# tar xzvf harbor-offline-installer-v2.4.1.tgz

Before continuing, I realised that the Docker data was being stored in /var/lib/docker, this wasn’t going to work for me as it was running on the smaller 40GB drive. I had to move the Docker data to the larger partition, to do so I followed this guide.

Now, copy the Harbor configuration file that is in the uncompressed Harbor installer folder or rename it.

[email protected]:/mnt/data/harbor# cp harbor.yml.tmpl harbor.yml

Edit the yml file to reflect your environment details, the output below are the lines that need to be changed.

[email protected]:/mnt/data/harbor# vi harbor.yml
#### The main options that I changed are
hostname: reg.mydomain.com
  certificate: /your/certificate/path
  private_key: /your/private/key/path
data_volume: /data

If you would like to see my harbor.yml it is located here.

Next, I install Harbor by running the command in the output below.

Note: Do not skip installing chartmuseum, it is required for NAPP.

[email protected]:/mnt/data/harbor# ./install.sh --with-chartmuseum

It will go through 5 steps to install and configure harbor and you should see something similar to the screen below once it is complete.

nsx napp harbor install

You should now be able to access the Harbor registry UI, it is also worthwhile to check that the certificate was properly installed and the site is secure.

nsx napp harbor deployed

Uploading NSX Application Platform Binaries to the Harbor Registry

The final part of this article is to push the images to the Harbor registry.

First I created a new project, to do so, login to Harbor and click on create a new project.

nsx napp harbor projects
nsx napp harbor projects create

Once the project is created, the images can be uploaded. In my environment I chose to utilize a jumpbox which has all the tools I required pre-installed, as well as the NAPP binaries transferred and uncompressed. Details and configuration of this jump box can be found in the first article of this series.

Navigate to the folder that contains the uncompressed NAPP binaries, there should be a file called upload_artifacts_to_private_harbor.sh. Open this file in a text editor, in my case I used vim.

Change the first three lines to suit your environment, instructions on the VMware website for this can be found here.

DOCKER_REPO=harbor.lab2prod.com.au/impactor
DOCKER_USERNAME=admin
DOCKER_PASSWORD=Harbor12345

Once you have made those changes, run the script to start the upload process.

upload nsx napp images to harbor registry

Ignore the errors you see regarding a connection, this is specific to my environment. At this point you can leave the images to be pushed to your Harbor registry, the time this takes can vary depending on your environment.

Once this process completes, you should see something similar on your jumpbox or whatever you chose to upload the images from.

nsx napp harbor images pushed

The repository should list 80 repositories and 18 charts.

nsx napp harbor projects repo

And that’s it for the Harbor repository deployment!

Next Steps

NSX Application Platform Part 3: NSX-T, NSX-ALB (Avi), and Tanzu

The next part in this series focusses on NSX-T, NSX-ALB, and Tanzu.

NSX Application Platform Part 4: Deploying the Application Platform

The final part of the series demonstrates the deployment process for NSX Application Platform and its security features (NSX Intelligence, Network Detection and Response, and Malware Prevention.

Previous Article

NSX Application Platform Part 1: Environment Overview

This was the first article in the series, it provides an overview to the environment.

8 thoughts on “NSX Application Platform Part 2: Harbor Image Registry

    • Author gravatar

      it will that helps a lot ,I hope have Part 3
      thank you.

    • Author gravatar

      Hi Shank,

      Are you able to please provide a bit more clarification (and perhaps a link to the relevant doco?) on:

      “Note: as of NSX-T 3.2 release (Impactor), self-signed certificates, including Active Directory issued certificates are not supported.”

      Is this is regards to NSX-T certificates or Harbour certificates?

      Cheers,
      Kane.

      • Author gravatar

        Hey Kane!
        This is in relation to the private harbor repositories certificate. Normally you could use a self signed / AD signed certificate with the repository. However, when attempting to deploy NAPP, you will likely face the issue displayed in this article. Let me know if that helps or I can try and elaborate.

        Unfortunately, there is no clear cut documentation on this yet.

        Cheers.

    • Author gravatar

      Great article! When executing the upload script I’m getting an error.

      Login Succeeded
      “helm_repo” already exists with the same configuration, skipping
      Hang tight while we grab the latest from your chart repositories…
      …Successfully got an update from the “helm_repo” chart repository
      Update Complete. ⎈Happy Helming!⎈
      The push refers to repository [harbor.homelab.com/homelab/clustering/authserver]
      e1f4fb348756: Layer already exists
      535d53d69ce8: Layer already exists
      b3df25c89779: Layer already exists
      517ed69cfd9a: Layer already exists
      824bf068fd3d: Layer already exists
      19067763: digest: sha256:fef547d24293ac57409e767871826f626e65408aa39f3e29303f9ba73170df59 size: 1367
      ApplyLayer exit status 1 stdout: stderr: lchown /usr/lib/jvm/zre-8-amd64/lib/security/policy/limited/US_export_policy.jar: invalid argument
      “docker tag” requires exactly 2 arguments.
      See ‘docker tag –help’.

      Usage: docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

      Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
      The push refers to repository [harbor.homelab.com/homelab/clustering/cluster_api]
      An image does not exist locally with the tag: harbor.homelab.com/homelab/clustering/cluster_api

Leave a Reply

Your email address will not be published.