ARM port of Project Harbor Registry on Raspberry PI4 with RockyLinux 9

Author: Ananda Kammampati

Dated : November 2022

Raspberry Pi4

Harbor Registry Portal

02-harbor-arm-about

Portainer Dashboard

03-harbor-arm-portainer

Harbor Images in Portainer

04-harbor-arm-portainer

Credits:

  • A Big Thank You! to the Contributors  (ZhuBo, Wang Yan, Steven Zou)

Goal:

  • Running ARM port of Project Harbor Registry on Raspberry Pi4 with RockyLinux 9
  • Running Portainer Commuity Edition on the same platform to administer the Containers that makes up Harbor Registry

Scope:

  • Documenting and sharing all the required steps (especially dealing with one minor hiccup that others might run into)

Out of Scope:

  • Administration and detailed usage of Harbor Registry
  • Administration and detailed usage of Portainer
  • Extensive end-to-end testing

Reference:

Step 01: Platform Verification

# uname -a
Linux management 5.15.55-v8.1.el9.altarch #1 SMP PREEMPT Tue Jul 26 04:16:30 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
# cat /etc/*release
-----
NAME="Rocky Linux"
VERSION="9.0 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.0 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.0"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"
Rocky Linux release 9.0 (Blue Onyx)
Rocky Linux release 9.0 (Blue Onyx)
Rocky Linux release 9.0 (Blue Onyx)
-----

Step 02: Have enough storage for container images

I have a 1 TB NVMe SSD connected to USB port of Raspberry Pi4, formatted with EXT4 filesystem type and mounted at /storage

# vi /etc/fstab
-----
/dev/sda1 /storage ext4 defaults 0 0
-----

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 931.5G 0 disk
└─sda1 8:1 0 931.5G 0 part /storage  ------------>> 1TB SSD mounted at /storage
mmcblk0 179:0 0 119.1G 0 disk
├─mmcblk0p1 179:1 0 286M 0 part /boot
├─mmcblk0p2 179:2 0 488M 0 part [SWAP]
└─mmcblk0p3 179:3 0 118.3G 0 part /

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 117G 12G 105G 11% /
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 120M 1.5G 8% /run
tmpfs 3.9G 4.0K 3.9G 1% /tmp
/dev/mmcblk0p1 286M 92M 195M 32% /boot
/dev/sda1 916G 1.2G 869G 1% /storage ------------>> This is where Harbor Registry components go
tmpfs 782M 0 782M 0% /run/user/0

Step 03: Follow the steps provided in Github Repository (listed under Reference)

# mkdir /var/log/harbor

# cd /storage ; mkdir harbor-arm-data

# git clone https://github.com/goharbor/harbor-arm.git

# cd harbor-arm && make download

# make compile_redis

# make prepare_arm_data

# make pre_update

# make compile COMPILETAG=compile_golangimage

# make build GOBUILDTAGS="include_oss include_gcs" BUILDBIN=true NOTARYFLAG=true TRIVYFLAG=true CHARTFLAG=true GEN_TLS=true PULL_BASE_FROM_DOCKERHUB=false

Step 04: Verify Harbor Registry docker images are built successfully for ARM

# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
goharbor/harbor-exporter dev-arm f9bc5775d454 5 minutes ago 75.1MB
goharbor/harbor-exporter-base dev-arm 0540d1f937cc 8 minutes ago 41MB
goharbor/chartmuseum-photon dev-arm 2e61c8d509f4 9 minutes ago 169MB
goharbor/harbor-chartserver-base dev-arm 46a227c6edb9 19 minutes ago 39.1MB
goharbor/redis-photon dev-arm 09e25cd5193f 20 minutes ago 210MB
goharbor/harbor-redis-base dev-arm 3f8f6f002846 20 minutes ago 210MB
goharbor/trivy-adapter-photon dev-arm e5411f93f8f0 22 minutes ago 121MB
goharbor/harbor-trivy-adapter-base dev-arm 1b75e0fe954b 23 minutes ago 39.5MB
goharbor/notary-server-photon dev-arm 3a3fab46f223 24 minutes ago 137MB
goharbor/notary-signer-photon dev-arm c0a7e69d2eed 26 minutes ago 134MB
goharbor/harbor-notary-signer-base dev-arm ce8255599265 33 minutes ago 39.1MB
goharbor/harbor-notary-server-base dev-arm 75fb85b6abea 34 minutes ago 39.1MB
goharbor/harbor-registryctl dev-arm fd9719a83e77 34 minutes ago 116MB
goharbor/harbor-registryctl-base dev-arm 0d63a8989387 35 minutes ago 39.1MB
goharbor/registry-photon dev-arm f609b511537b 35 minutes ago 75MB
goharbor/harbor-registry-base dev-arm 3f050f4b8a48 38 minutes ago 39.1MB
goharbor/harbor-log dev-arm d14a5f8d8192 40 minutes ago 153MB
goharbor/harbor-log-base dev-arm 36c2e53e56cd 40 minutes ago 153MB
goharbor/harbor-jobservice dev-arm 8df215475681 42 minutes ago 186MB
goharbor/harbor-jobservice-base dev-arm d52c76ca2eaf 42 minutes ago 41MB
goharbor/harbor-core dev-arm 1ae46f77b976 43 minutes ago 153MB
goharbor/harbor-core-base dev-arm 67049c59902f 44 minutes ago 41MB
goharbor/harbor-portal dev-arm 08114d3dff07 44 minutes ago 54MB
goharbor/harbor-db dev-arm 8046504551ce 3 hours ago 183MB
goharbor/harbor-db-base dev-arm 3c302198859f 3 hours ago 183MB
goharbor/prepare dev-arm 2fdb416a18b7 3 hours ago 259MB ----> This is the one that needs to be pointed to, in the script 'prepare'
goharbor/harbor-prepare-base dev-arm d52ad7d34036 3 hours ago 256MB
goharbor/swagger v0.25.0 ed6b2d427cab 4 hours ago 737MB
photon_build_spec 4.0 f60b3497b0c7 6 hours ago 443MB
photon 4.0 71e33fc244c2 4 days ago 34.1MB
goharbor/nginx-photon dev-arm d4b2feba0610 5 months ago 40.8MB
golang 1.15.12 92e0f4bd4b90 17 months ago 721MB

Step 05: Download the latest harbor online installer (as of this writing: harbor-online-installer-v2.6.1.tgz)

# tar zxvf harbor-online-installer-v2.6.1.tgz

# cd harbor

# cp harbor.yml.tmpl harbor.yml

# vi harbor.yml ---------------> Make changes to this file according to your setup
-----
hostname: 192.168.1.45 --------> IP Address of my Raspberry Pi

http:
port: 8080 --------------------> port number where I am exposing Harbor Registry portal/service

# https related config --------> diabling HTTPS for now
# https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path

harbor_admin_password: Harbor12345  ------> Change the admin password here
data_volume: /storage/harbor-arm-data  ---> The directory where everything will be stored
-----

Step 06: Run the installer script

# ./install.sh
....
....
[Step 3]: preparing harbor configs ...
prepare base dir is set to /storage/harbor-arm/harbor
Unable to find image 'goharbor/prepare:v2.6.1' locally ----> Note this installer is looking for the image goharbor/prepare:v2.6.1
v2.6.1: Pulling from goharbor/prepare
e82db3878067: Pull complete
4eb87f61c01b: Pull complete
38d5f9d1f8bb: Pull complete
ce463d8bd4fd: Pull complete
3ff5c5228716: Pull complete
81b6713e3f24: Pull complete
0764ab8b6c6c: Pull complete
288d8b2c0c34: Pull complete
9edcfb579c8b: Pull complete
7ff00c916846: Pull complete
Digest: sha256:7046ed9124b729a5e49c1833812cfc06dc682bae9a53c92d4898911cb74dde77
Status: Downloaded newer image for goharbor/prepare:v2.6.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/bin/python3: exec format error

Step 07: Root Cause of the problem

  1. The installer.sh script inturn calls another script named 'prepare'
  2. The 'prepare' script looks for a specific version of image named 'goharbor/prepare:v2.6.1' locally.
This is because the tar file we download is meant for v2.6.1
  3. Since that image is not available locally, the script goes ahead and downloads it from the docker hub
  4. Keep in mind that all the images provided on docker hub are meant for X64 platform and not for ARM
  5. So when the script tries to run a docker image meant for X64 platform, we will run into that 'exec format error'

Step 08: Let us get rid of the troublesome image 'goharbor:prepare:v2.6.1' that we no longer needed

# docker images | grep v2.6.1
-----
goharbor/prepare v2.6.1 6f69bcbe07f1 2 weeks ago 164MB
-----

# docker rmi -f `docker images | grep v2.6.1 | awk ' { print $3 } '`

Step 09: Fixing 'prepare' script

# vi prepare ----> Modify line number 60. Change it from goharbor/prepare:v2.6.1 to goharbor/prepare:dev-arm

-----
docker run --rm -v $input_dir:/input \
-v $data_path:/data \
-v $harbor_prepare_path:/compose_location \
-v $config_dir:/config \
-v /:/hostfs \
--privileged \
goharbor/prepare:v2.6.1 prepare $@ ---> change v2.6.1 to dev-arm
-----

// It should look like this :

-----
docker run --rm -v $input_dir:/input \
-v $data_path:/data \
-v $harbor_prepare_path:/compose_location \
-v $config_dir:/config \
-v /:/hostfs \
--privileged \
goharbor/prepare:dev-arm prepare $@
-----

Step 10: Run the installer.sh script again and Voila !

# ./install.sh
.....
.....
.....
.....
[Step 4]: starting Harbor ...
[+] Running 10/10
⠿ Network harbor_harbor Created 0.4s
⠿ Container harbor-log Started 4.0s
⠿ Container registryctl Started 7.4s
⠿ Container redis Started 7.7s
⠿ Container harbor-portal Started 7.8s
⠿ Container harbor-db Started 7.7s
⠿ Container registry Started 6.7s
⠿ Container harbor-core Started 8.9s
⠿ Container nginx Started 11.5s
⠿ Container harbor-jobservice Started 11.3s

Step 11: Verify all the Containers are instanciated and are running fine

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7897f31117c6 goharbor/harbor-jobservice:dev-arm "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice
403ff997aa23 goharbor/nginx-photon:dev-arm "nginx -g 'daemon of…" About a minute ago Up 59 seconds (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp nginx
516c3ab203cf goharbor/harbor-core:dev-arm "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core
d5f7288c798c goharbor/harbor-db:dev-arm "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db
920060b3d71d goharbor/redis-photon:dev-arm "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis
15a812707976 goharbor/harbor-portal:dev-arm "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal
245c2b2aef24 goharbor/registry-photon:dev-arm "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry
a1a45cc2d631 goharbor/harbor-registryctl:dev-arm "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl
b9e0887ba656 goharbor/harbor-log:dev-arm "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log

Step 12: Verify all the required artifacts are created 

# ls -lt /storage/harbor-arm-data
total 24
drwx------. 3 systemd-coredump ssh_keys 4096 Oct 28 22:11 database
drwxr-xr-x. 5 root root 4096 Oct 28 22:10 secret
drwxr-xr-x. 2 systemd-coredump ssh_keys 4096 Oct 28 22:10 redis
drwxr-xr-x. 2 10000 10000 4096 Oct 28 22:10 job_logs
drwxr-xr-x. 2 10000 10000 4096 Oct 28 22:10 registry
drwxr-xr-x. 2 10000 10000 4096 Oct 28 22:10 ca_download

Step 13: Now the final moment of truth 🙂

Visit http://192.168.1.45:8080

default credentails : admin / Harbor12345
01-harbor-arm-about
02-harbor-arm-about