|
Size: 3180
Comment:
|
Size: 7297
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Incomplete, do not use! |
|
| Line 5: | Line 3: |
| We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming HTTP-requests to two webservers. | We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming requests. |
| Line 7: | Line 5: |
| * 192.168.1.45 haproxy01 * 192.168.1.46 haproxy02 |
|
| Line 8: | Line 8: |
| * 192.168.1.48 haproxy01 * 192.168.1.49 haproxy02 |
* 192.168.1.50 mariadb (virtual IP-address) * 192.168.1.54 redis (virtual IP-address) |
| Line 19: | Line 19: |
| === HA-proxy === ==== HA-proxy resource script ==== Download the HA-proxy resource script, place it in `/usr/lib/ocf/resource.d/heartbeat/haproxy` and make it executeable. {{{ wget https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy -O /usr/lib/ocf/resource.d/heartbeat/haproxy chmod +x /usr/lib/ocf/resource.d/heartbeat/haproxy }}} As we will let corosync start haproxy, we will have to disable haproxy as a service. {{{ service haproxy stop systemctl disable haproxy }}} ==== Services ==== Add the configuration for the services to `/etc/haproxy/haproxy.cfg`. Notice that the galera cluster is checked with a mariadb user called haproxy. This user will be created when the galera cluster is configured. {{{ listen www_farm bind www:80 mode http balance roundrobin cookie LBN insert indirect nocache option httpclose option forwardfor server www01 www01:80 cookie node1 check server www02 www02:80 cookie node2 check listen galera_cluster bind mariadb:3306 mode tcp balance roundrobin option tcpka option mysql-check user haproxy server mariadb01 mariadb01:3306 check weight 1 server mariadb02 mariadb02:3306 check weight 1 server mariadb03 mariadb03:3306 check weight 1 listen redis bind redis:6379 mode tcp balance first option tcp-check tcp-check send info\ replication\r\n tcp-check expect string role:master server redis01 redis01:6379 check inter 1s server redis02 redis02:6379 check inter 1s server redis03 redis03:6379 check inter 1s }}} Copy the configuration `/etc/haproxy/haproxy.cfg` to the other host. |
|
| Line 25: | Line 77: |
| cluster_name: www | |
| Line 30: | Line 83: |
| ==== Certificates ==== Generate the certificates. Note that this requires entropy, so use your system and the entropy will be generated. |
==== Authentication Key ==== Generate the authkey. Note that this requires entropy, so use your system and the entropy will be generated. Speed this up by running `dd if=/dev/vda of=/dev/null` in another terminal or install `haveged`. |
| Line 37: | Line 90: |
| Add the local network and a multicast address to `/etc/corosync/corosync.conf` in the `interface` section. | Add the local network and a multicast address to `/etc/corosync/corosync.conf` in the `interface` section. Tne `interface section is a subsection of the `totem` section. |
| Line 44: | Line 97: |
| ==== Nodes ==== Configure each of the nodes. {{{ nodelist { node { nodeid: 1 ring0_addr: 192.168.1.45 name: haproxy01 } node { nodeid: 2 ring0_addr: 192.168.1.46 name: haproxy02 } } }}} |
|
| Line 45: | Line 114: |
| Restart corosync to load the new configuration. | Copy the configuration `/etc/corosync/corosync.conf` and the authkey `/etc/corosync/authkey` to the other host. Restart corosync on both hosts to load the new configuration. |
| Line 50: | Line 121: |
| == Commmands == Run the following on startup {{{#!highlight bash # do not kill other node crm configure property stonith-enabled=false # disable quorum due to two nodes only crm configure property no-quorum-policy=ignore # configure shared IP crm configure primitive haproxySharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.47 cidr_netmask=24 op monitor interval=5s # create heartbeat for haproxy crm configure primitive haproxyLoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s |
The two nodes should now be configured to see each other. Run `crm status` to check. Notice it will take a while before the nodes are up. {{{ Stack: corosync Current DC: haproxy01 (version 1.1.16-94ff4df) - partition with quorum Last updated: Sat Feb 1 14:41:09 2020 Last change: Sat Feb 1 14:40:51 2020 by hacluster via crmd on haproxy01 |
| Line 62: | Line 128: |
| # make sure the same server has both IP and service #crm configure group haproxyIPs haproxySharedIP meta ordered=false crm configure group haproxy haproxySharedIP haproxyLoadBalance # relation between IP and haproxy servers crm configure colocation haproxyWithIPs INFINITY: haproxyLoadBalance haproxySharedIP # IP should be up before haproxy starts crm configure order haproxyAfterIPs mandatory: haproxySharedIP haproxyLoadBalance #wget http://github.com/russki/cluster-agents/raw/master/haproxy #echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf |
2 nodes configured 0 resources configured |
| Line 72: | Line 131: |
| Online: [ haproxy01 haproxy02 ] No resources |
|
| Line 74: | Line 136: |
| ==== Only two nodes ==== Add this to the quorum block of `/etc/corosync/corosync.conf`. |
==== Pacemaker ==== DOES NOT WORK FOR BUSTER, only for stretch. we need to learn about fencing before continuing Do not kill the other node. This is needed as inactive nodes normally should be killed. It can be change later on, but only after fencing has been configured. |
| Line 77: | Line 146: |
| quorum { two_node: 1 } |
crm configure property stonith-enabled=false }}} Disable quorum due to two nodes only. This will make sure that the cluster keeps running if only one node is up. {{{ crm configure property no-quorum-policy=ignore }}} Configure shared IP-address for apache. This is the virtual IP-address shared by apache servers. {{{ crm configure primitive wwwSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.47 cidr_netmask=24 op monitor interval=5s }}} Configure shared IP-address for mariadb. This is the virtual IP-address shared by the galera cluster. {{{ crm configure primitive mariadbSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.50 cidr_netmask=24 op monitor interval=5s }}} Configure shared IP-address for redis. This is the virtual IP-address shared by the redis master/replicas. {{{ crm configure primitive redisSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s }}} Create heartbeat for haproxy. {{{ crm configure primitive LoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s }}} Make sure the same server has both IP and service at the same time. {{{ crm configure group services wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance }}} IP-address should be up before haproxy starts. {{{ crm configure order haproxyAfterIP mandatory: wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance |
| Line 82: | Line 177: |
=== HA-proxy === Add the following to `/etc/haproxy/haproxy.cfg` |
== Test it == === Apache === |
| Line 86: | Line 180: |
| listen webfarm bind www:80 mode http balance roundrobin cookie LBN insert indirect nocache option httpclose option forwardfor server haproxy01 www01:80 cookie node1 check server haproxy02 www02:80 cookie node2 check |
while true; do wget --quiet http://www/ -O /dev/stdout; sleep 1; done }}} === Mariadb === {{{ while true; do mysql -u haproxy -p -h mariadb -e "select @@hostname"; sleep 1; done |
| Line 97: | Line 187: |
| Restart HA-proxy {{{ service haproxy restart }}} |
|
| Line 106: | Line 188: |
| * https://wiki.debian.org/Debian-HA/ClustersFromScratch | * https://wiki.debian.org/Debian-HA/ClustersFromScratch * https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-cluster-options.html * https://www.digitalocean.com/community/tutorials/how-to-create-a-high-availability-setup-with-corosync-pacemaker-and-floating-ips-on-ubuntu-14-04 * https://www.howtoforge.com/tutorial/how-to-setup-haproxy-as-load-balancer-for-mariadb-on-centos-7/#b-install-mariadb-galera-server * https://www.digitalocean.com/community/tutorials/how-to-use-haproxy-to-set-up-mysql-load-balancing--3 * https://manpages.debian.org/buster/corosync/corosync.conf.5.en.html * https://www.programmersought.com/article/72231082401/ * https://xahteiwi.eu/resources/hints-and-kinks/fencing-libvirtkvm-virtualized-cluster-nodes/ * https://www.suse.com/support/kb/doc/?id=000018490 * https://www.programmersought.com/article/72231082401/ * https://mvallim.github.io/kubernetes-under-the-hood/documentation/haproxy-cluster.html * https://clusterlabs.org/pacemaker/doc/crm_fencing.html |
Corosync and HA-proxy
We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming requests.
- 192.168.1.45 haproxy01
- 192.168.1.46 haproxy02
- 192.168.1.47 www (virtual IP-address)
- 192.168.1.50 mariadb (virtual IP-address)
- 192.168.1.54 redis (virtual IP-address)
Software
apt-get install corosync haproxy crmsh
Configuration
Before binding to the DNS name www make sure the DNS server knows that name. There is no physical or virtual network interfaces, so the DNS updates has to be done manually.
HA-proxy
HA-proxy resource script
Download the HA-proxy resource script, place it in /usr/lib/ocf/resource.d/heartbeat/haproxy and make it executeable.
wget https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy -O /usr/lib/ocf/resource.d/heartbeat/haproxy chmod +x /usr/lib/ocf/resource.d/heartbeat/haproxy
As we will let corosync start haproxy, we will have to disable haproxy as a service.
service haproxy stop systemctl disable haproxy
Services
Add the configuration for the services to /etc/haproxy/haproxy.cfg. Notice that the galera cluster is checked with a mariadb user called haproxy. This user will be created when the galera cluster is configured.
listen www_farm
bind www:80
mode http
balance roundrobin
cookie LBN insert indirect nocache
option httpclose
option forwardfor
server www01 www01:80 cookie node1 check
server www02 www02:80 cookie node2 check
listen galera_cluster
bind mariadb:3306
mode tcp
balance roundrobin
option tcpka
option mysql-check user haproxy
server mariadb01 mariadb01:3306 check weight 1
server mariadb02 mariadb02:3306 check weight 1
server mariadb03 mariadb03:3306 check weight 1
listen redis
bind redis:6379
mode tcp
balance first
option tcp-check
tcp-check send info\ replication\r\n
tcp-check expect string role:master
server redis01 redis01:6379 check inter 1s
server redis02 redis02:6379 check inter 1s
server redis03 redis03:6379 check inter 1sCopy the configuration /etc/haproxy/haproxy.cfg to the other host.
Corosync
Encryption
Edit /etc/corosync/corosync.conf and make the following changes.
totem {
cluster_name: www
crypto_cipher: aes256
crypto_hash: sha512
}
Authentication Key
Generate the authkey. Note that this requires entropy, so use your system and the entropy will be generated. Speed this up by running dd if=/dev/vda of=/dev/null in another terminal or install haveged.
corosync-keygen
Network
Add the local network and a multicast address to /etc/corosync/corosync.conf in the interface section. Tne interface section is a subsection of the totem` section.
interface {
bindnetaddr: 192.168.1.0
mcastaddr: 239.192.1.1
}
Nodes
Configure each of the nodes.
nodelist {
node {
nodeid: 1
ring0_addr: 192.168.1.45
name: haproxy01
}
node {
nodeid: 2
ring0_addr: 192.168.1.46
name: haproxy02
}
}Copy the configuration /etc/corosync/corosync.conf and the authkey /etc/corosync/authkey to the other host.
Restart corosync on both hosts to load the new configuration.
service corosync restart
The two nodes should now be configured to see each other. Run crm status to check. Notice it will take a while before the nodes are up.
Stack: corosync Current DC: haproxy01 (version 1.1.16-94ff4df) - partition with quorum Last updated: Sat Feb 1 14:41:09 2020 Last change: Sat Feb 1 14:40:51 2020 by hacluster via crmd on haproxy01 2 nodes configured 0 resources configured Online: [ haproxy01 haproxy02 ] No resources
Pacemaker
DOES NOT WORK FOR BUSTER, only for stretch.
we need to learn about fencing before continuing
Do not kill the other node. This is needed as inactive nodes normally should be killed. It can be change later on, but only after fencing has been configured.
crm configure property stonith-enabled=false
Disable quorum due to two nodes only. This will make sure that the cluster keeps running if only one node is up.
crm configure property no-quorum-policy=ignore
Configure shared IP-address for apache. This is the virtual IP-address shared by apache servers.
crm configure primitive wwwSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.47 cidr_netmask=24 op monitor interval=5s
Configure shared IP-address for mariadb. This is the virtual IP-address shared by the galera cluster.
crm configure primitive mariadbSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.50 cidr_netmask=24 op monitor interval=5s
Configure shared IP-address for redis. This is the virtual IP-address shared by the redis master/replicas.
crm configure primitive redisSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s
Create heartbeat for haproxy.
crm configure primitive LoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s
Make sure the same server has both IP and service at the same time.
crm configure group services wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance
IP-address should be up before haproxy starts.
crm configure order haproxyAfterIP mandatory: wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance
Test it
Apache
while true; do wget --quiet http://www/ -O /dev/stdout; sleep 1; done
Mariadb
while true; do mysql -u haproxy -p -h mariadb -e "select @@hostname"; sleep 1; done