A Methodical Proof of Swami ARMananda Theorem

 

Author: Ananda Kammampati

Dated: Sept 2021

radxa_rockpi4c

Swami ARMananda's Theorem states:

 

'RockyLinux + ARM Embedded + Qemu + Virt-Manager + Virtual Machines + podman Containers + Open Source Software = Kick-a$$'
Zabbix Monitoring Dashboard on ARM SBC
final-dashboard-23
Virt-Manager + Qemu + RockyLinux VMs on ARM SBC
vms-with-wpress-final
Webmin Administration on ARM SBC
webmin-30

Goals:

  • To prove the above theorem methodically, by building the entire stack from scratch, from bottom to top on Rockpi4c SBC
  • Installing RockyLinux8 on ARM64 Rockpi4c first
  • Installing Opensource softwares namely Qemu, Virt-Manager and Zabbix, Mariadb, PHP, WordPress, webmin
  • Running two virtual machines with Qemu and Virt-Manager on Rockpi4c SBC
  • Running Zabbix Software Agents on the virtual machines
  • Running podman Containers inside each virtual machine
  • Hosting frontend of WordPress as a podman Container inside on the first virtual machine
  • Hosting backend (mariadb) of WordPress as a podman container inside the second virtual machine
  • Running Zabbix Software on the Host to monitor the virtual machines
  • Documenting all the steps needed and share
  • Giving due credits go to all sources and respective Authors

Prerequisites:

  • Patience

Disclaimer:

  • Please use this article only as a reference
  • This article is primarily written for a cut-and-paste friendly approach
  • For detailed explanations please refer the links listed under Credits
  • I only figured out how to put them all together
  • I make no claims that I understood all of it. Mostly I never will 😉

Source of Inspiration:

Required Items:

Configuration:

  • Rockpi4c with 4GB RAM, a 32GB SD Cards with 2 partition on it
  • 1 x 1GB DOS Boot partition, 1 x Linux partition taking over the rest of available space
  • 2 virtual machines 512MB RAM, 10GB diskspace (carved out of the above Linux partition)
  • Both Host and virtual machines run RockyLinux8 built for ARM64

Methodology:

  • Install virt-manager, qemu-kvm and mysql on the host (ie, Rockpi4c embedded)
  • Install Zabbix Server (monitoring software) on host
  • Assign IP address 10.1.1.160/24 to host
  • Create 2 virtual machines and install RockyLinux8 on both of them
  • Install Zabbix Agents on both of them
  • Assign IP Address 192.168.122.45/24 to rocky-vm-01
  • Assign IP address 192.168.122.191/24 to rocky-vm-02
  • Install Mariadb as the backend to WordPress, running as podman container inside rocky-vm-02
  • Install WordPress frontend, running as a podman container inside rocky-vm-01
  • Monitor both the virtual machines from Zabbix dashboard UI running on the host
  • Monitor them all with htop and Webmin UI  Dashboard

Observation:

  • Because I alloted only 512MB of RAM to each virtual machine, they were relatively slow
  • While creating the mariadb podman container, I have to give 20 minutes for the initialization alone to complete
  • (This may be a good time to go practice your Guitar, Piano, Yoga or get your groceries done)
  • These steps works consistently but does need a lot of fine tuning, if ever considered for Production

Videos:

  • I have placed videos in appropriate places for reference
  • Some are a bit longer so might have to speed up

Credits:

Special Thanks:

  • RockyLinux community member @Skip Grube for building and sharing Zabbix rpms and Rockylinux8 image for Raspberry Pi4
  • RockyLinux community members @stack and @Neil Hanlon in helping me out with Zabbix and with dnf
  • RockyLinux community member @Markvnl for building RockyLinux8 image for Rockpi4c (refer my previous article)
  • My sincere apologies if I have missed anyone else

Step 01: Installing virt-manager, qemu on Rockpi4c Host (10.1.1.160)

 

host# whoami
root

host# uname -a
Linux rockpi4c 5.10.52-200.el8.aarch64 #1 SMP Mon Aug 9 23:04:44 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux

host# cat /etc/*release
Rocky Linux release 8.4 (Green Obsidian)
NAME="Rocky Linux"
VERSION="8.4 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.4 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8.4:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
Rocky Linux release 8.4 (Green Obsidian)
Rocky Linux release 8.4 (Green Obsidian)
Rocky Linux release 8.4 (Green Obsidian)

host# dnf install -y virt-manager qemu-kvm

host# sync ; reboot

Step 02: Installing Zabbix Server On Rockpi4c Host

 

host# dnf localinstall https://skiprocky.linuxdn.org/RockyRpi/RockyRpi-extras/zabbix-server-mysql-5.0.15-1.el8.aarch64.rpm

host# vi /etc/yum.repos.d/Rocky-Extras.repo [Added @Skip Grube's above Repo]
-----
[extras]
name=Rocky Linux $releasever - Extras
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=extras-$releasever
#baseurl=http://dl.rockylinux.org/$contentdir/$releasever/extras/$basearch/os/
baseurl=https://skiprocky.linuxdn.org/RockyRpi/RockyRpi-extras/
gpgcheck=0
enabled=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
-----

host# dnf install -y zabbix-web-mysql zabbix-apache-conf

host# dnf install -y mysql mysql-server

host# systemctl start mysqld.service

host# systemctl status -l mysqld.service

host# systemctl enable mysqld.service

Step 03: Configure MySQL Server (on host)

 

host# mysql
mysql> quit

host# mysql_secure_installation

There are three levels of password validation policy: 0 (Selected LOW)
New password: rootDBpass
Re-enter new password: rootDBpass
Remove anonymous users? y
Disallow root login remotely? y
Remove test database and access to it? y
Reload privilege tables now? y

host# mysql -uroot -p'rootDBpass' -e "create database zabbix character set utf8 collate utf8_bin;"

host# mysql -uroot -p
Enter password: rootDBpass

mysql> create user 'zabbixuser'@'localhost' identified by 'zabbixDBpass1!';
mysql> GRANT ALL ON zabbix.* TO 'zabbixuser'@'localhost' with GRANT OPTION;
mysql> flush privileges;
mysql> quit;

host# mysql -uroot -p'rootDBpass' zabbix -e "set global innodb_strict_mode='OFF';"

host# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbixuser -p'zabbixDBpass1!' zabbix

host# mysql -uroot -p'rootDBpass' zabbix -e "set global innodb_strict_mode='ON';"

Step 04: Configure Zabbix Server Configuration file

 

host# vi /etc/zabbix/zabbix_server.conf
-----
SourceIP=10.1.1.160
DBUser=zabbixuser
DBPassword=zabbixDBpass1!
-----

host# systemctl restart zabbix-server

host# systemctl enable zabbix-server

host# systemctl status -l zabbix-server

Step 05: Configure php for Zabbix frontend

 

host# vi /etc/php-fpm.d/zabbix.conf
-----
php_value[date.timezone] = America/Los_Angeles
-----

host# systemctl restart httpd php-fpm

host# systemctl enable httpd php-fpm

host# systemctl status -l httpd php-fpm

Step 06: Configure Zabbix Server from UI

From Host, open up the Browser to : http://10.1.1.160/zabbix

Credentials to Login after initial configuration : Admin / rockylinux

Step 07: Invoke Virt-Manager and create 2 virtual machines

 

Now that Zabbix Server is configured on the host, time to create 2 virtual machines using Virt-Manager

 

Step 08: Downloading RockyLinux ISO for ARM and CDROM redirection

 

On the host, open up the browser and download RockyLinux8 ISO file for ARM. Then modify both Virtual Machines by pointing their CDROMs to the downloaded ISO file

 

Step 09: Booting up the virtual machines

 

With both virtual machines CDROMs redirected to the downloaded RockLinux8 ISO for ARM file, power them on. Refer the videos below for their bootup output. Please bare in mind that each video is 8-10+ minutes long. So you may want to speed them up if needed.

 

Step 10: Install Zabbix Agent on virtual machines rocky-vm-01 and rocky-vm-02

 

Repeat these steps in both the virtual machines, but with their right hostnames in their respective zabbix_agentd.conf file

rocky-vm-01# whoami
root

rocky-vm-01# hostname
rocky-vm-01

rocky-vm-01# dnf install -y logrotate

rocky-vm-01# rpm -ivh https://skiprocky.linuxdn.org/RockyRpi/RockyRpi-extras/zabbix-agent-5.0.15-1.el8.aarch64.rpm

rocky-vm-01# vi /etc/zabbix/zabbix_agentd.conf
-----
Server=10.1.1.160 ---------------------------->> IP Address of Host running Zabbix Server
ServerActive=10.1.1.160 ---------------------->> IP Address of Host running Zabbix Server
Hostname=rocky-vm-01 ------------------------->> for the second virtual machine, Hostname=rocky-vm-02
-----

rocky-vm-01# systemct restart zabbix-agent

rocky-vm-01# systemct status -l zabbix-agent

rocky-vm-01# netstat -tulpn|grep zabbix

rocky-vm-01# systemct enable zabbix-agent

 

Step 11: Adding virtual machines on Zabbix Server Dashboard

 

Once Zabbix Agents are installed on both the Virtual Machines, the next step is to add the virtual machines details onto Zabbix Server so that they can now be monitored

 

Step 12: Building Podman Containers

 

Now that Zabbix part is done, let us get on with building podman containers on each virtual machine

 

Step 13: Steps carried out in rocky-vm-02 (192.168.122.191)

 

- Configure the backend first
- So let us begin with configuring rocky-vm-02 first to create and run mariadb within a podman container
- We are going to create the following files needed to build and run podman containers in this virtual machine

  • Dockerfile needed for mariadb container
  • build.sh to build mariadb container
  • run script to run mariadb container
rocky-vm-02# hostname
rocky-vm-02

rocky-vm-02# whoami
root

rocky-vm-02# pwd
/root

rocky-vm-02# dnf install -y podman buildah

rocky-vm-02# podman --version
podman version 3.0.2-dev

rocky-vm-02# buildah --version
buildah version 1.19.8 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)

rocky-vm-02# vi /etc/containers/registries.conf
-----
[registries.insecure]
registries = ['registry.access.redhat.com', 'registry.redhat.io', 'docker.io']
insecure = true
-----

rocky-vm-02# mkdir mariadb

rocky-vm-02# cd mariadb

rocky-vm-02# pwd
/root/mariadb

rocky-vm-02# vi Dockerfile
-----
FROM arm64v8/mariadb:latest
-----

rocky-vm-02# vi build.sh
-----
#!/bin/bash -x
buildah rmi `buildah images -q mariadb` ;
buildah bud --no-cache -t mariadb . ;
buildah images -a
-----

rocky-vm-02# chmod 755 build.sh

rocky-vm-02# ./build.sh --------->> Select docker.io from the list when prompted
-----
To see the output, scroll down and click on the button 1) 'mariadb' build.sh output
-----

rocky-vm-02# mkdir /root/base

rocky-vm-02# cd /root/base

rocky-vm-02# vi Dockerfile
-----
FROM arm64v8/centos:latest
ENV container docker
RUN yum -y install epel-release ; yum -y update
RUN dnf module enable -y php:7.4
RUN dnf install -y php
RUN yum -y install wget curl net-tools iproute bridge-utils ; yum -y update
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]
-----

rocky-vm-02# vi build.sh
-----
#!/bin/bash
buildah rmi `buildah images -q base` ;
buildah bud --no-cache -t base . ;
buildah images -a
-----

rocky-vm-02# chmod 755 build.sh

rocky-vm-02# ./build.sh
-----
To see the output, scroll down and click on the button 2) 'base' build.sh output
-----
rocky-vm-02# mkdir /root/tools

rocky-vm-02# cd /root/tools

rocky-vm-02# vi db-drop.sh
-----
#!/bin/bash
mysql -h192.168.122.191 -u root -prockylinux << eof
drop database wpressdb;
flush privileges;
eof
-----

rocky-vm-02# vi db-create.sh
-----
#!/bin/bash
mysql -h192.168.122.191 -u root -prockylinux << eof
create database wpressdb;
grant all on wpressdb.* to 'wpress-user'@'192.168.122.45' identified by 'wpress-pass';
flush privileges;
eof
-----

rocky-vm-02# vi Dockerfile
-----
FROM localhost/base
RUN yum -y install mysql
WORKDIR /root
COPY db-drop.sh db-drop.sh
COPY db-create.sh db-create.sh
-----

rocky-vm-02# vi build.sh
-----
#!/bin/bash -x
buildah rmi `buildah images -q tools` ;
buildah bud --no-cache -t tools . ;
buildah images -a
-----

rocky-vm-02# chmod 755 *.sh

rocky-vm-02# ./build.sh

-----
To see the output, scroll down and click on the button 3) 'tools' build.sh output
-----

rocky-vm-02# buildah images (or # podman images)
-----
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/tools latest 7ce17f60368e 38 minutes ago 565 MB
localhost/base latest 859c17ddf14a 16 hours ago 472 MB
docker.io/arm64v8/centos latest e6a0117ec169 5 days ago 279 MB
localhost/mariadb latest 63449ffc5ac6 2 weeks ago 400 MB
docker.io/arm64v8/mariadb latest 63449ffc5ac6 2 weeks ago 400 MB
-----

rocky-vm-02# vi run-initdb.sh
-----
#!/bin/bash
clear
date > /tmp/out ;
echo " "
echo "List existing volumes first...."
echo " "
podman volume ls ;
echo " "
echo "*** Starting mariadb container ***"
echo " "
podman run --name mariadb --label mariadb -d --net host \
             -e MYSQL_ROOT_PASSWORD=rockylinux -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
             -v mariadb-data:/var/lib/mysql/data:Z mariadb ;

echo " "
echo "*** Will give 20 minutes for mariadb initialization to complete ***
sleep 1200 ;

echo " "
echo "*** Creating wpressdb Database ***"
podman run --rm --net host tools /root/db-create.sh ;
echo "Done."

podman volume ls ;

date >> /tmp/out ;
echo " "
echo "*** Start and End time ***"
echo " "
cat /tmp/out ;
rm /tmp/out
-----

rocky-vm-02# chmod 755 run-initdb.sh

rocky-vm-02# ./run-initdb.sh
-----
To see the output, scroll down and click on the button 4) 'run-initdb.sh' output
-----

Note:
-----
Let us manually verify that the Database was created successfully. Will do it by getting inside the mariadb 
container itself

rocky-vm-02# podman exec -ti mariadb /bin/bash

rocky-vm-02# mysql -h192.168.122.191 -prockylinux
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| data               |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| wpressdb           | --------------->> Database was created successfully !
+--------------------+
6 rows in set (0.596 sec)

MariaDB [(none)]> quit
Bye
rocky-vm-02# exit
exit

rocky-vm-02# podman ps -a
-----
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19e5ce1ee408 localhost/mariadb:latest mysqld 2 hours ago Up 2 hours ago mariadb
-----

Note:
-----
If for some reason we want to delete the newly created wpressdb Database alone without deleting the mariadb container,
we can run the following command as we have already added 'db-drop.sh' script along with 'db-create.sh while building
the 'tools' container

rocky-vm-02# podman run --rm --net host tools /root/db-drop.sh ;
Step 14: Steps carried out in rocky-vm-01 (192.168.122.45)


Now that we are done configuring the backend in Step 13, it is time to configure the frontend of WordPress in this 
virtual machine. We are going to create files similar to the ones we created in Step 12, but the differences will 
be evident as we move forward while building container to run this virtual machine as a frontend.

rocky-vm-01# hostname
rocky-vm-01

rocky-vm-01# whoami
root

rocky-vm-01# dnf install -y podman buildah

rocky-vm-01# vi /etc/containers/registries.conf
-----
[registries.insecure]
registries = ['registry.access.redhat.com', 'registry.redhat.io', 'docker.io']
insecure = true
-----

rocky-vm-01# mkdir /root/base

rocky-vm-01# cd /root/base

rocky-vm-01# vi Dockerfile
-----
FROM arm64v8/centos:latest
ENV container docker
RUN yum -y install epel-release ; yum -y update
RUN dnf module enable -y php:7.4
RUN dnf install -y php
RUN yum -y install wget curl net-tools iproute bridge-utils ; yum -y update
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]
-----

rocky-vm-01# vi build.sh
-----
#!/bin/bash
buildah rmi `buildah images -q base` ;
buildah bud --no-cache -t base . ;
buildah images -a
-----
rocky-vm-01# ./build.sh
-----
To see a similar output, scroll up and click on the button 2) 'base' build.sh output
-----

rocky-vm-01# mkdir /root/wpress

rocky-vm-01# cd /root/wpress

The files that we create to build and run the wpress podman container are:
  1. wp-config.php
  2. php.ini
  3. Dockerfile
  4. build.sh
  5. run.sh
rocky-vm-01# vi wp-config.php
-----
To see file contents, scroll down and click on the button 5) wp-config.php 
-----

rocky-vm-01# vi php.ini
-----
To see file contents, scroll down and click on the button 6) php.ini
-----

rocky-vm-01# vi Dockerfile
-----
FROM localhost/base
RUN yum -y install gd php-gd php-mysqlnd php-curl php-zip php-pecl-zip ; yum -y update ; yum clean all
EXPOSE 80
COPY php.ini /etc
RUN curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
RUN chmod 755 ./wp-cli.phar
RUN mv wp-cli.phar /usr/local/bin/wp
RUN wp core download --locale=en_US --skip-content --allow-root
COPY wp-config.php wp-config.php
RUN wp core install --admin_user=Admin --admin_password=rockylinux --url=http://192.168.122.45 --title=Wordpress \
                    --admin_email=none@rockylinux.io --allow-root
-----

rocky-vm-01# vi build.sh
-----
#!/bin/bash -x
buildah rmi `buildah images -q wpress` ;
buildah bud --no-cache -t wpress . ;
buildah images -a
-----
rocky-vm-01# ./buid.sh
-----
To see file contents, scroll down and click on the button 7) wpress build.sh output
-----

rocky-vm-01# podman images
-----
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/wpress latest b4f710404aec 12 hours ago 580 MB
localhost/base latest afdd445bf4a0 21 hours ago 472 MB
docker.io/arm64v8/centos latest e6a0117ec169 7 days ago 279 MB
-----

rocky-vm-01# vi run.sh
-----
#!/bin/bash
podman run --name wpress --net host --privileged -d -p 80:80 -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
       wpress wp server --host=192.168.122.45 --port=80
-----

rocky-vm-01# ./run.sh
-----
Port mappings have been discarded as one of the Host, Container, Pod, and None network modes are in use
25220026b37d2939ad62910c1ed45f31ee2af89c0a24c6417e3251a392d1aef0
-----

-----
Note: There is a way to suppress the warning messages displayed above by modifying some file. 
Will revisit this document again and update it. 
-----
Verify the wpress container is running in rocky-vm-01:

rocky-vm-01# hostname
rocky-vm-01

// Verify the wpress container is running fine on rocky-vm-01

rocky-vm-01# podman ps -a
-----
[root@rocky-vm-01 wpress]# podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
25220026b37d localhost/wpress:latest wp server --host=... 37 minutes ago Up 36 minutes ago wpress
-----

// Now verify the mariadb container is running fine in rocky-vm-02

rocky-vm-02# hostname
rocky-vm-02

rocky-vm-02# podman ps -a
-----
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0c2e08974c2 localhost/mariadb:latest mysqld 16 hours ago Up 16 hours ago mariadb
-----

Note:
One way to verify that wpress container from rocky-vm-01 indeed got connected to mariadb container in rocky-vm-02 
is by manually running mysql client on rocky-vm-02 and display that tables are indeed created in the wpressdb database.

-----
To see the output, scroll up and click on the button 8) Display mariadb tables
-----

Finally, from host, open up the browser http://192.168.122.45:80

Step 15: Performance of virtual machines while running wordpress

 

Here are a couple of videos to show a glimpse on how the virtual machines performed with wordpress running on the virtual machines. The first video shows the performance of both the virtual machines within the virt-manger's UI, along with 'htop'.

The second video shows 'htop' output after ssh'ing into the host.

 

Step 16: Installing Webmin

host# dnf install -y perl

( Might need these commands - # dnf module switch-to perl:5.26 ; # dnf install -y perl )

host# vi /etc/yum.repos.d/webmin.repo
-----
[Webmin]
name=Webmin Distribution Neutral
#baseurl=http://download.webmin.com/download/yum
mirrorlist=http://download.webmin.com/download/yum/mirrorlist
enabled=1
-----

host# yum install -y wget

host# wget http://www.webmin.com/jcameron-key.asc

host# rpm --import jcameron-key.asc

host# yum install -y webmin

Open up the browser https://10.1.1.160:10000 and log in with root credentials

webmin-29
webmin-30