Differences between revisions 32 and 94 (spanning 62 versions)
⇤ ← Revision 32 as of 2018-01-20 15:51:29 →
Size: 3493
Editor: Kristian Kallenberg
Comment:
← Revision 94 as of 2021-01-26 21:19:02 → ⇥
Size: 6776
Editor: Kristian Kallenberg
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
Incomplete, do not use!
Line 5: Line 3:
We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming HTTP-requests to two webservers. We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming requests.
Line 7: Line 5:
 * 192.168.1.47 www (virtual IP-address)
 * 192.168.1.48 haproxy01
 * 192.168.1.49 haproxy02		  * 192.168.1.49 haproxy01
 * 192.168.1.50 haproxy02
 * 192.168.1.51 www (virtual IP-address)
 * 192.168.1.54 mariadb (virtual IP-address)
 * 192.168.1.58 redis (virtual IP-address)
Line 19: Line 19:
=== Corosync ===

==== Encryption ====
Edit `/etc/corosync/corosync.conf` and make the following changes.
{{{
totem {
        crypto_cipher: aes256
        crypto_hash: sha512
}
}}}

==== Certificates ====
Generate the certificates. Note that this requires entropy, so use your system and the entropy will be generated. Speed this up by running `dd if=/dev/vda of=/dev/null` in another terminal.
{{{
corosync-keygen
}}}

==== Network ====
Add the local network and a multicast address to `/etc/corosync/corosync.conf` in the `interface` section.
{{{
interface {
        bindnetaddr: 192.168.1.0
        mcastaddr: 239.192.1.1
}
}}}

Restart corosync to load the new configuration.
{{{
service corosync restart
}}}		 === HA-proxy ===
Line 57: Line 28:
==== Nodes ====
Do not kill the other node.		 As we will let corosync start haproxy, we will have to disable haproxy as a service.
Line 60: Line 30:
crm configure property stonith-enabled=false
}}}
Disable quorum due to two nodes only.
{{{
crm configure property no-quorum-policy=ignore
}}}
Configure shared IP-address.
{{{
crm configure primitive haproxySharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.47 cidr_netmask=24 op monitor interval=5s
}}}
Create heartbeat for haproxy.
{{{
crm configure primitive haproxyLoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s
}}}
Make sure the same server has both IP and service at the same time.
{{{
crm configure group haproxy haproxySharedIP haproxyLoadBalance
}}}
Create the relation between IP-address and haproxy servers.
{{{
crm configure colocation haproxyWithIPs INFINITY: haproxyLoadBalance haproxySharedIP
}}}
IP-address should be up before haproxy starts.
{{{
crm configure order haproxyAfterIPs mandatory: haproxySharedIP haproxyLoadBalance
}}}
THIS HAS TO BE TESTED
{{{
#wget http://github.com/russki/cluster-agents/raw/master/haproxy
#echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf		 service haproxy stop
systemctl disable haproxy
Line 92: Line 34:


=== HA-proxy ===
Add the following to `/etc/haproxy/haproxy.cfg`		 ==== Services ====
Add the configuration for the services to `/etc/haproxy/haproxy.cfg`. Notice that the galera cluster is checked with a mariadb user called haproxy. This user will be created when the galera cluster is configured.
Line 97: Line 37:
listen webfarm listen www_farm
Line 104: Line 44:
        server haproxy01 www01:80 cookie node1 check
        server haproxy02 www02:80 cookie node2 check		         server www01 www01:80 cookie node1 check
        server www02 www02:80 cookie node2 check

listen galera_cluster
        bind mariadb:3306
        mode tcp
        balance roundrobin
        option tcpka
        option mysql-check user haproxy
        server mariadb01 mariadb01:3306 check weight 1
        server mariadb02 mariadb02:3306 check weight 1
        server mariadb03 mariadb03:3306 check weight 1

listen redis
        bind redis:6379
        mode tcp
        balance first
        option tcp-check
        tcp-check send info\ replication\r\n
        tcp-check expect string role:master
        server redis01 redis01:6379 check inter 1s
        server redis02 redis02:6379 check inter 1s
        server redis03 redis03:6379 check inter 1s
Line 108: Line 69:
Restart HA-proxy Copy the configuration `/etc/haproxy/haproxy.cfg` to the other host.

=== Corosync ===

==== Encryption ====
Edit `/etc/corosync/corosync.conf` and make the following changes.
Line 110: Line 76:
service haproxy restart totem {
        cluster_name: www
        crypto_cipher: aes256
        crypto_hash: sha512
}
Line 113: Line 83:
==== Authentication Key ====
Generate the authkey. Note that this requires entropy, so use your system and the entropy will be generated. Speed this up by running `dd if=/dev/vda of=/dev/null` in another terminal or install `haveged`.
{{{
corosync-keygen
}}}
Line 114: Line 89:
==== Network ====
Add the local network and a multicast address to `/etc/corosync/corosync.conf` in the `interface` section. Tne `interface section is a subsection of the `totem` section.
{{{
interface {
        bindnetaddr: 192.168.1.0
        mcastaddr: 239.192.1.1
}
}}}
==== Nodes ====
Configure each of the nodes.
{{{
nodelist {
    node {
        nodeid: 1
        ring0_addr: 192.168.1.49
        name: haproxy01
    }
    node {
        nodeid: 2
        ring0_addr: 192.168.1.50
        name: haproxy02
    }
}
}}}
Line 115: Line 114:
Copy the configuration `/etc/corosync/corosync.conf` and the authkey `/etc/corosync/authkey` to the other host.

Restart corosync on both hosts to load the new configuration.
{{{
service corosync restart
}}}

The two nodes should now be configured to see each other. Run `crm status` to check. Notice it will take a while before the nodes are up.
{{{
Stack: corosync
Current DC: haproxy01 (version 1.1.16-94ff4df) - partition with quorum
Last updated: Sat Feb 1 14:41:09 2020
Last change: Sat Feb 1 14:40:51 2020 by hacluster via crmd on haproxy01

2 nodes configured
0 resources configured

Online: [ haproxy01 haproxy02 ]

No resources
}}}

==== Pacemaker ====
Do not kill the other node. This is needed as inactive nodes normally should be killed. It can be change later on, but only after fencing has been configured.
{{{
crm configure property stonith-enabled=false
}}}
Disable quorum due to two nodes only. This will make sure that the cluster keeps running if only one node is up.
{{{
crm configure property no-quorum-policy=ignore
}}}
Configure shared IP-address for apache. This is the virtual IP-address shared by apache servers.
{{{
crm configure primitive wwwSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.51 cidr_netmask=24 op monitor interval=5s
}}}
Configure shared IP-address for mariadb. This is the virtual IP-address shared by the galera cluster.
{{{
crm configure primitive mariadbSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s
}}}
Configure shared IP-address for redis. This is the virtual IP-address shared by the redis master/replicas.
{{{
crm configure primitive redisSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s
}}}
Create heartbeat for haproxy.
{{{
crm configure primitive LoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s
}}}
Make sure the same server has both IP and service at the same time.
{{{
crm configure group services wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance
}}}
IP-address should be up before haproxy starts.
{{{
crm configure order haproxyAfterIP mandatory: wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance
}}}

== Test it ==
=== Apache ===
{{{
while true; do wget --quiet http://www/ -O /dev/stdout; sleep 1; done
}}}
=== Mariadb ===
{{{
while true; do mysql -u haproxy -p -h mariadb -e "select @@hostname"; sleep 1; done
}}}
Line 117: Line 181:
  * https://wiki.debian.org/Debian-HA/ClustersFromScratch  * https://wiki.debian.org/Debian-HA/ClustersFromScratch
 * https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-cluster-options.html
 * https://www.digitalocean.com/community/tutorials/how-to-create-a-high-availability-setup-with-corosync-pacemaker-and-floating-ips-on-ubuntu-14-04
 * https://www.howtoforge.com/tutorial/how-to-setup-haproxy-as-load-balancer-for-mariadb-on-centos-7/#b-install-mariadb-galera-server
 * https://www.digitalocean.com/community/tutorials/how-to-use-haproxy-to-set-up-mysql-load-balancing--3
 * https://manpages.debian.org/buster/corosync/corosync.conf.5.en.html

Corosync and HA-proxy

We will use corosync to create a virtual IP-address which is shared by two systems. One of the systems has the virtual IP-address. If the system goes down, the other system will take over the virtual IP-address. HA-proxy will be running on each of the hosts relaying incoming requests.

  • 192.168.1.49 haproxy01
  • 192.168.1.50 haproxy02
  • 192.168.1.51 www (virtual IP-address)
  • 192.168.1.54 mariadb (virtual IP-address)
  • 192.168.1.58 redis (virtual IP-address)

Software

apt-get install corosync haproxy crmsh

Configuration

Before binding to the DNS name www make sure the DNS server knows that name. There is no physical or virtual network interfaces, so the DNS updates has to be done manually.

HA-proxy

HA-proxy resource script

Download the HA-proxy resource script, place it in /usr/lib/ocf/resource.d/heartbeat/haproxy and make it executeable. 

wget https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy -O /usr/lib/ocf/resource.d/heartbeat/haproxy
chmod +x /usr/lib/ocf/resource.d/heartbeat/haproxy

As we will let corosync start haproxy, we will have to disable haproxy as a service. 

service haproxy stop
systemctl disable haproxy

Services

Add the configuration for the services to /etc/haproxy/haproxy.cfg. Notice that the galera cluster is checked with a mariadb user called haproxy. This user will be created when the galera cluster is configured. 

listen www_farm 
        bind www:80
        mode http
        balance roundrobin
        cookie LBN insert indirect nocache
        option httpclose
        option forwardfor
        server www01 www01:80 cookie node1 check
        server www02 www02:80 cookie node2 check

listen galera_cluster
        bind mariadb:3306
        mode tcp
        balance roundrobin
        option tcpka
        option mysql-check user haproxy
        server mariadb01 mariadb01:3306 check weight 1 
        server mariadb02 mariadb02:3306 check weight 1
        server mariadb03 mariadb03:3306 check weight 1

listen redis
        bind redis:6379
        mode tcp
        balance first
        option tcp-check
        tcp-check send info\ replication\r\n
        tcp-check expect string role:master
        server redis01 redis01:6379 check inter 1s
        server redis02 redis02:6379 check inter 1s
        server redis03 redis03:6379 check inter 1s

Copy the configuration /etc/haproxy/haproxy.cfg to the other host.

Corosync

Encryption

Edit /etc/corosync/corosync.conf and make the following changes. 

totem {
        cluster_name: www
        crypto_cipher: aes256
        crypto_hash: sha512
}

Authentication Key

Generate the authkey. Note that this requires entropy, so use your system and the entropy will be generated. Speed this up by running dd if=/dev/vda of=/dev/null in another terminal or install haveged. 

corosync-keygen

Network

Add the local network and a multicast address to /etc/corosync/corosync.conf in the interface section. Tne interface section is a subsection of the totem` section. 

interface {
        bindnetaddr: 192.168.1.0 
        mcastaddr: 239.192.1.1
}

Nodes

Configure each of the nodes. 

nodelist {
    node {
        nodeid: 1
        ring0_addr: 192.168.1.49
        name: haproxy01
    }
    node {
        nodeid: 2
        ring0_addr: 192.168.1.50
        name: haproxy02
    }
}

Copy the configuration /etc/corosync/corosync.conf and the authkey /etc/corosync/authkey to the other host.

Restart corosync on both hosts to load the new configuration. 

service corosync restart

The two nodes should now be configured to see each other. Run crm status to check. Notice it will take a while before the nodes are up. 

Stack: corosync
Current DC: haproxy01 (version 1.1.16-94ff4df) - partition with quorum
Last updated: Sat Feb  1 14:41:09 2020
Last change: Sat Feb  1 14:40:51 2020 by hacluster via crmd on haproxy01

2 nodes configured
0 resources configured

Online: [ haproxy01 haproxy02 ]

No resources

Pacemaker

Do not kill the other node. This is needed as inactive nodes normally should be killed. It can be change later on, but only after fencing has been configured. 

crm configure property stonith-enabled=false

Disable quorum due to two nodes only. This will make sure that the cluster keeps running if only one node is up. 

crm configure property no-quorum-policy=ignore

Configure shared IP-address for apache. This is the virtual IP-address shared by apache servers. 

crm configure primitive wwwSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.51 cidr_netmask=24 op monitor interval=5s

Configure shared IP-address for mariadb. This is the virtual IP-address shared by the galera cluster. 

crm configure primitive mariadbSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s

Configure shared IP-address for redis. This is the virtual IP-address shared by the redis master/replicas. 

crm configure primitive redisSharedIP ocf:heartbeat:IPaddr2 params ip=192.168.1.54 cidr_netmask=24 op monitor interval=5s

Create heartbeat for haproxy. 

crm configure primitive LoadBalance ocf:heartbeat:haproxy params conffile=/etc/haproxy/haproxy.cfg op monitor interval=10s

Make sure the same server has both IP and service at the same time. 

crm configure group services wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance

IP-address should be up before haproxy starts. 

crm configure order haproxyAfterIP mandatory: wwwSharedIP mariadbSharedIP redisSharedIP LoadBalance

Test it

Apache

while true; do wget --quiet http://www/ -O /dev/stdout; sleep 1; done

Mariadb

while true; do mysql -u haproxy -p -h mariadb -e "select @@hostname"; sleep 1; done

References

None: Corosync and HA-proxy (last edited 2021-08-12 09:31:08 by Kristian Kallenberg)