Running an ARM port of Harbor Registry and Portainer on Raspberry PI4 with RockyLinux 8.4
Author: Ananda Kammampati
Dated : November 2021
Harbor Registry Portal
Portainer Dashboard
Harbor Images in Portainer
Credits:
- All credits on porting Project Harbor Registry to ARM Architecture goes to Bin Liang . ( A Big Thank You! )
Goal:
- Running ARM port of Project Harbor Registry on Raspberry Pi4 with RockyLinux 8.4
- Running Portainer Commuity Edition on the same platform to administer the Containers that makes up the Harbor Registry
Scope:
- Document and sharing all the required steps
Out of Scope:
- Administration and detailed usage of Harbor Registry
- Administration and detailed usage of Portainer
- Note that the version of Harbor Registry is relatively older
- Extensive end-to-end testing
References:
- https://github.com/hzliangbin/harbor-arm64
- https://github.com/goharbor/harbor/issues/11633
- https://www.how2shout.com/linux/how-to-install-portainer-docker-web-gui
- https://github.com/goharbor/harbor/releases
- https://community.portainer.io
Videos:
- Video is also for reference
Making a Wish:
- Dear Harbor Stakeholders@VMware - PLEEEEASE consider promoting ARM builds for Project Harbor Registry
- ARM SBC Enthusiasts all over the world will be thankful to you for Eternity 🙂
Step 01: Platform Verification
# uname -a Linux raspi4-rocky 5.10.52-v8.1.el8 #1 SMP PREEMPT Wed Sep 29 23:57:05 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux # cat /etc/*release ----- Rocky Linux release 8.4 (Green Obsidian) NAME="Rocky Linux" VERSION="8.4 (Green Obsidian)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="8.4" PLATFORM_ID="platform:el8" PRETTY_NAME="Rocky Linux 8.4 (Green Obsidian)" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:rocky:rocky:8.4:GA" HOME_URL="https://rockylinux.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" ROCKY_SUPPORT_PRODUCT="Rocky Linux" ROCKY_SUPPORT_PRODUCT_VERSION="8" Rocky Linux release 8.4 (Green Obsidian) Rocky Linux release 8.4 (Green Obsidian) Rocky Linux release 8.4 (Green Obsidian) -----
Step 02: Installing docker
# dnf install -y docker-ce # docker --version Docker version 20.10.10, build b485636 # systemctl start docker.service # systemctl status -l docker.service ----- ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2021-10-27 23:43:33 UTC; 10s ago Docs: https://docs.docker.com Main PID: 2879 (dockerd) Tasks: 9 CGroup: /system.slice/docker.service └─2879 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock Oct 27 23:43:29 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:29.244477824Z" level=warning msg="Your kernel does not support CPU realtime scheduler" Oct 27 23:43:29 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:29.244503509Z" level=warning msg="Your kernel does not support cgroup blkio weight" Oct 27 23:43:29 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:29.244529064Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Oct 27 23:43:29 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:29.245940749Z" level=info msg="Loading containers: start." Oct 27 23:43:30 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:30.378531331Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon o> Oct 27 23:43:31 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:31.010636585Z" level=info msg="Loading containers: done." Oct 27 23:43:32 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:32.645814347Z" level=info msg="Docker daemon" commit=e2f740d graphdriver(s)=overlay2 version=20.10.10 Oct 27 23:43:32 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:32.646210032Z" level=info msg="Daemon has completed initialization" Oct 27 23:43:33 raspi4-rocky systemd[1]: Started Docker Application Container Engine. Oct 27 23:43:33 raspi4-rocky dockerd[2879]: time="2021-10-27T23:43:33.194973416Z" level=info msg="API listen on /var/run/docker.sock" ----- # systemctl enable docker.service # docker images REPOSITORY TAG IMAGE ID CREATED SIZE # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Step 03: Installing docker-compose
# wget https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-aarch64 # chmod 755 ./docker-compose-linux-aarch64 # mv ./docker-compose-linux-aarch64 /usr/local/bin/docker-compose # docker-compose --version Docker Compose version v2.0.1
Step 04: Have enough storage for container images
I have a 1 TB NVMe SSD connected to USB port of Raspberry Pi4, formatted with EXT4 filesystem type and mounted at /data
# vi /etc/fstab ----- /dev/sda1 /data ext4 defaults 0 0 ----- # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk └─sda1 8:1 0 931.5G 0 part /data ------------>> 1TB SSD mounted at /data mmcblk0 179:0 0 14.9G 0 disk ├─mmcblk0p1 179:1 0 286M 0 part /boot ├─mmcblk0p2 179:2 0 488M 0 part [SWAP] └─mmcblk0p3 179:3 0 14.1G 0 part / # df -h Filesystem Size Used Avail Use% Mounted on /dev/root 14G 7.6G 6.3G 55% / devtmpfs 3.8G 0 3.8G 0% /dev tmpfs 3.9G 25M 3.8G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs 782M 0 782M 0% /run/user/0 /dev/mmcblk0p1 286M 66M 220M 24% /boot /dev/sda1 916G 1.4G 868G 1% /data ------------>> This is where Harbor Registry components go
Step 05: Download ARM Harbor Offline installer (Version 1.9.3) from Bin Liang's Github Repository
# mkdir /var/log/harbor
# pwd
/root
# wget https://github.com/hzliangbin/harbor-arm64/releases/download/v1.9.3/harbor-offline-installer-v1.9.3.tgz
# tar zxvf harbor-offline-installer-v1.9.3.tgz
# cd harbor
# vi harbor.yml
-----
hostname: 192.168.1.45 --------------------->> Change hostname, port, data_volume according to your setup
port: 8080
data_volume: /data
location: /var/log/harbor
-----
Step 06: Run the installer
# ./install.sh -------------------->> Click the button 'install.sh output' below to see its output
Step 07: Verify Harbor Registry docker images are downloaded successfully
# docker images REPOSITORY TAG IMAGE ID CREATED SIZE goharbor/redis-photon v1.9.3 a74eb7f8d1f5 16 months ago 93MB goharbor/clair-photon v2.1.0-v1.9.3 590f2641528f 16 months ago 215MB goharbor/harbor-registryctl v1.9.3 2e601207c097 16 months ago 95.5MB goharbor/registry-photon v2.7.1-patch-2819-2553-v1.9.3 6dc433588070 16 months ago 79.4MB goharbor/nginx-photon v1.9.3 3be597f805f7 16 months ago 39.4MB goharbor/harbor-log v1.9.3 813238123c15 16 months ago 102MB goharbor/harbor-jobservice v1.9.3 5002c9a379b9 16 months ago 131MB goharbor/harbor-core v1.9.3 243a33e77535 16 months ago 144MB goharbor/harbor-portal v1.9.3 348475a4cd83 16 months ago 46.8MB goharbor/harbor-db v1.9.3 1ee674580fdd 16 months ago 139MB goharbor/prepare v1.9.3 29f7481983a0 16 months ago 145MB
Step 08: Verify if all the containers are running fine after installation
# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1d31ed98e176 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 14 minutes ago Created nginx a1a5a3cb8db8 goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 14 minutes ago Created harbor-jobservice 5373bf506684 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 14 minutes ago Created harbor-core b3428728fad4 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 15 minutes ago Created 5000/tcp registry f75657777ca5 goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 15 minutes ago Created 5432/tcp harbor-db a3ee639d242b goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 15 minutes ago Exited (1) 14 minutes ago registryctl e20925da8095 goharbor/redis-photon:v1.9.3 "docker-entrypoint.s…" 15 minutes ago Created 6379/tcp redis d9f43d7de9a8 goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 15 minutes ago Created 8080/tcp harbor-portal 5213157448fa goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 15 minutes ago Restarting (1) 4 seconds ago harbor-log
Step 09: Fixing containers - registry, registrctl, harbor-log
- You will notice the above mentioned Containers in Red will fail to run and restart
- The explanation and a few workarounds are suggested in this Github thread
Reason for failure:
- The password time limit got expired
One of the suggested workarounds I followed (and worked):
- Create a tar file of the container named registry
- Extract its /etc/shadow file
- Change the timelimit expiry date from 90 days to a larger number (99999 days)
- Copy that /etc/shadow file to a know location (/opt/harbor-fix)
- Modify docker-compose.yml file , pass /opt/harbox-fix/shadow file to the failing containers namely regsitry, registryctl and harbor-log as voume mount
- Restart harbor registry application stack with 'docker-compose' command
# mkdir -p /tmp/harbor-fix /opt/harbor-fix # cd /tmp/harbor-fix # docker export registry -o registry.tar # tar xvfp registry.tar # sed -i 's/:90:/:99999:/g' /tmp/harbor-fix/etc/shadow # cp /tmp/harbor-fix/etc/shadow /opt/harbor-fix
# cd /root/harbor # vi docker-compose.yml ----- ... ... container_name: harbor-log -----> add lines below to this Container ... ... volumes: - /var/log/harbor/:/var/log/docker/:z - ./common/config/log/logrotate.conf:/etc/logrotate.d/logrotate.conf:z - ./common/config/log/rsyslog_docker.conf:/etc/rsyslog.d/rsyslog_docker.conf:z - type: bind ------------------------------------------------------------------>> Add this line source: /opt/harbor-fix/shadow ------------------------------------------------>> Add this line target: /etc/shadow ----------------------------------------------------------->> Add this line ... ... container_name: registry -----> add lines below to this Container ... ... volumes: - /data/registry:/storage:z - ./common/config/registry/:/etc/registry/:z - type: bind source: /data/secret/registry/root.crt target: /etc/registry/root.crt - type: bind ------------------------------------------------------------------>> Add this line source: /opt/harbor-fix/shadow ------------------------------------------------>> Add this line target: /etc/shadow ----------------------------------------------------------->> Add this line ... ... container_name: registryctl -----> add lines below to this Container ... ... volumes: - /data/registry:/storage:z - ./common/config/registry/:/etc/registry/:z - type: bind source: ./common/config/registryctl/config.yml target: /etc/registryctl/config.yml - type: bind ------------------------------------------------------------------>> Add this line source: /opt/harbor-fix/shadow ------------------------------------------------>> Add this line target: /etc/shadow ----------------------------------------------------------->> Add this line ... ... -----
Step 10: Restart Harbor Registry application stack with docker-compose command
# docker-compose down # docker-compoe up -d [root@raspi4-rocky harbor]# docker-compose up -d [+] Running 10/10 ⠿ Network harbor_harbor Created 0.3s ⠿ Container harbor-log Started 9.7s ⠿ Container harbor-db Started 11.8s ⠿ Container redis Started 12.5s ⠿ Container harbor-portal Started 12.7s ⠿ Container registry Started 12.7s ⠿ Container registryctl Started 11.1s ⠿ Container harbor-core Started 12.3s ⠿ Container nginx Started 16.1s ⠿ Container harbor-jobservice Started 16.0s
Step 11: Verify all the Containers are running fine with the workaround in place
# docker ps -a root@rockpi4c:~/harbor# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 98be1d472ed9 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 28 seconds ago Up 20 seconds (health: starting) 0.0.0.0:8080->8080/tcp nginx 39da2b2ce6bb goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 28 seconds ago Up 20 seconds (health: starting) harbor-jobservice ef4332b973a7 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 29 seconds ago Up 22 seconds (health: starting) harbor-core 88c594e06aa1 goharbor/redis-photon:v1.9.3 "docker-entrypoint.s…" 29 seconds ago Up 24 seconds (health: starting) 6379/tcp redis d1a8af7558a7 goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 29 seconds ago Up 23 seconds (health: starting) 8080/tcp harbor-portal b181227c237b goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 29 seconds ago Up 24 seconds (health: starting) 5000/tcp registry bea097cb3417 goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 29 seconds ago Up 24 seconds (health: starting) 5432/tcp harbor-db 65d3ad835462 goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 29 seconds ago Up 23 seconds (health: starting) registryctl 7b181b99a0e9 goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 29 seconds ago Up 26 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log
Video capturing output of 'docker-compose up -d'
Step 12: Install Portainer with a single command line
# docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce Unable to find image 'portainer/portainer-ce:latest' locally latest: Pulling from portainer/portainer-ce 7721cab3d696: Pull complete 0645e7e2a110: Pull complete 398e449ccacc: Pull complete Digest: sha256:af387baba14e0342e40d274c0c894fd333d3cca0d6737a8e1e0d6d9523c87a8a Status: Downloaded newer image for portainer/portainer-ce:latest 378d7204cd69279d823ab9ae9aa9f31a512fd84d9608cbd377bb332fdd721f75
Log into Harbor Registry portal with the default credentials : admin / Harbor12345
Log into Portainer portal
Output of 'htop' command with all the Containers running
