corosync & pacemaker群集-命令
2024-06-28 16:01:43
供稿：網(wǎng)友
使用pcs shell配置corosync & pacemaker群集PacemakerPacemaker，即Cluster Resource Manager（CRM），管理整個(gè)HA，客戶端通過pacemaker管理監(jiān)控整個(gè)集群。CRM支持ocf和lsb兩種資源類型：ocf格式的啟動(dòng)腳本在/usr/lib/ocf/resource.d/下面。lsb的腳本一般在/etc/rc.d/init.d/下面。1、常用的集群管理工具：（1）基于命令行crm shell/pcs（2）基于圖形化pygui/hawk/lcmc/pcs2、相關(guān)的資源文件：（1）/usr/lib/ocf/resource.d，pacemaker資源庫文件位置，可安裝資源包：resource-agents 獲取更多ocf格式的資源。（2）/usr/sbin/fence_***，F(xiàn)encing設(shè)備的執(zhí)行腳本名稱，可安裝資源包：fence-agents 獲取更多Fencing設(shè)備資源。3、查看使用說明： [shell]# man ocf_heartbeat_*** ## 查看OCF資源說明，man ocf_heartbeat_apache [shell]# man fence_*** ## 查看Fencing設(shè)備說明，man fence_vmware4、參考文檔https://github.com/ClusterLabshttp://clusterlabs.org/doc/http://www.linux-ha.org/doc/man-pages/man-pages.htmlhttps://access.redhat.com/documentation/en-US/Red_Hat_EnterPRise_Linux/6/html/Configuring_the_Red_Hat_High_Availability_Add-On_with_Pacemaker/index.html在群集配置過程中參考了互聯(lián)網(wǎng)上眾多優(yōu)秀文章，在此感謝原作者！！！以下記錄整理了在vmware esxi5.5 + centos6.6環(huán)境中使用PCS命令配置corosync & pacemaker群集的一些操作，由于本人水平有限，僅供參考：--------------------------------------------------1.安裝群集軟件： [shell]# yum -y install corosync pacemaker pcs [shell]# yum -y install fence-agents resource-agents2.拷貝配置文件、啟動(dòng)腳本 [shell]# mkdir -p /etc/cluster/ [shell]# ln -s /etc/rc.d/init.d/corosync /etc/rc.d/init.d/cman [shell]# ln -s /usr/sbin/corosync-cmapctl /usr/sbin/corosync-objctl [shell]# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf注意：群集需要嚴(yán)格的時(shí)間同步機(jī)制，如果啟用了防火墻需要開放相應(yīng)的端口。--------------------------------------------------PCS（Pacemaker/Corosync configuration system）命令配置群集示例：一、建立群集：1、配置群集節(jié)點(diǎn)的認(rèn)證as the hacluster user: [shell]# pcs cluster auth node11 node122、創(chuàng)建一個(gè)二個(gè)節(jié)點(diǎn)的群集 [shell]# pcs cluster setup --name mycluster node11 node12 [shell]# pcs cluster start --all ## 啟動(dòng)群集3、設(shè)置資源默認(rèn)粘性（防止資源回切） [shell]# pcs resource defaults resource-stickiness=100 [shell]# pcs resource defaults4、設(shè)置資源超時(shí)時(shí)間 [shell]# pcs resource op defaults timeout=90s [shell]# pcs resource op defaults5、二個(gè)節(jié)點(diǎn)時(shí)，忽略節(jié)點(diǎn)quorum功能 [shell]# pcs property set no-quorum-policy=ignore6、沒有 Fencing設(shè)備時(shí)，禁用STONITH 組件功能在 stonith-enabled="false" 的情況下，分布式鎖管理器 (DLM) 等資源以及依賴DLM 的所有服務(wù)（例如 cLVM2、GFS2 和 OCFS2）都將無法啟動(dòng)。 [shell]# pcs property set stonith-enabled=false [shell]# crm_verify -L -V ## 驗(yàn)證群集配置信息二、建立群集資源1、查看可用資源 [shell]# pcs resource list ## 查看支持資源列表，pcs resource list ocf:heartbeat [shell]# pcs resource describe agent_name ## 查看資源使用參數(shù)，pcs resource describe ocf:heartbeat:ipaddr22、配置虛擬IP [shell]# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 / ip="192.168.10.15" cidr_netmask=32 nic=eth0 op monitor interval=30s 3、配置Apache(httpd) [shell]# pcs resource create WebServer ocf:heartbeat:apache / httpd="/usr/sbin/httpd" configfile="/etc/httpd/conf/httpd.conf" / statusurl="http://localhost/server-status" op monitor interval=1min4、配置Nginx [shell]# pcs resource create WebServer ocf:heartbeat:nginx / httpd="/usr/sbin/nginx" configfile="/etc/nginx/nginx.conf" / statusurl="http://localhost/ngx_status" op monitor interval=30s5.1、配置FileSystem [shell]# pcs resource create WebFS ocf:heartbeat:Filesystem / device="/dev/sdb1" directory="/var/www/html" fstype="ext4" [shell]# pcs resource create WebFS ocf:heartbeat:Filesystem / device="-U 32937d65eb" directory="/var/www/html" fstype="ext4"5.2、配置FileSystem-NFS [shell]# pcs resource create WebFS ocf:heartbeat:Filesystem / device="192.168.10.18:/MySQLdata" directory="/var/lib/mysql" fstype="nfs" / options="-o username=your_name,passWord=your_password" / op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=60s6、配置Iscsi [shell]# pcs resource create WebData ocf:heartbeat:iscsi / portal="192.168.10.18" target="iqn.2008-08.com.starwindsoftware:" / op monitor depth="0" timeout="30" interval="120" [shell]# pcs resource create WebFS ocf:heartbeat:Filesystem / device="-U 32937d65eb" directory="/var/www/html" fstype="ext4" options="_netdev"7、配置DRBD [shell]# pcs resource create WebData ocf:linbit:drbd / drbd_resource=wwwdata op monitor interval=60s [shell]# pcs resource master WebDataClone WebData / master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true [shell]# pcs resource create WebFS ocf:heartbeat:Filesystem / device="/dev/drbd1" directory="/var/www/html" fstype="ext4"8、配置MySQL [shell]# pcs resource create MySQL ocf:heartbeat:mysql / binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" datadir="/var/lib/mysql" / pid="/var/run/mysqld/mysql.pid" socket="/tmp/mysql.sock" / op start timeout=180s op stop timeout=180s op monitor interval=20s timeout=60s9、配置Pingd，檢測節(jié)點(diǎn)與目標(biāo)的連接有效性 [shell]# pcs resource create PingCheck ocf:heartbeat:pingd / dampen=5s multiplier=100 host_list="192.168.10.1 router" / op monitor interval=30s timeout=10s10、創(chuàng)建資源clone，克隆的資源會(huì)在全部節(jié)點(diǎn)啟動(dòng) [shell]# pcs resource clone PingCheck [shell]# pcs resource clone ClusterIP clone-max=2 clone-node-max=2 globally-unique=true ## clone-max=2，數(shù)據(jù)包分成2路 [shell]# pcs resource update ClusterIP clusterip_hash=sourceip ## 指定響應(yīng)請(qǐng)求的分配策略為：sourceip三、調(diào)整群集資源1、配置資源約束 [shell]# pcs resource group add WebSrvs ClusterIP ## 配置資源組，組中資源會(huì)在同一節(jié)點(diǎn)運(yùn)行 [shell]# pcs resource group remove WebSrvs ClusterIP ## 移除組中的指定資源 [shell]# pcs resource master WebDataClone WebData ## 配置具有多個(gè)狀態(tài)的資源，如 DRBD master/slave狀態(tài) [shell]# pcs constraint colocation add WebServer ClusterIP INFINITY ## 配置資源捆綁關(guān)系 [shell]# pcs constraint colocation remove WebServer ## 移除資源捆綁關(guān)系約束中資源 [shell]# pcs constraint order ClusterIP then WebServer ## 配置資源啟動(dòng)順序 [shell]# pcs constraint order remove ClusterIP ## 移除資源啟動(dòng)順序約束中資源 [shell]# pcs constraint ## 查看資源約束關(guān)系， pcs constraint --full2、配置資源位置 [shell]# pcs constraint location WebServer prefers node11 ## 指定資源默認(rèn)某個(gè)節(jié)點(diǎn)，node=50 指定增加的 score [shell]# pcs constraint location WebServer avoids node11## 指定資源避開某個(gè)節(jié)點(diǎn)，node=50 指定減少的 score [shell]# pcs constraint location remove location-WebServer## 移除資源節(jié)點(diǎn)位置約束中資源ID，可用pcs config獲取 [shell]# pcs constraint location WebServer prefers node11=INFINITY ## 手工移動(dòng)資源節(jié)點(diǎn)，指定節(jié)點(diǎn)資源的 score of INFINITY [shell]# crm_simulate -sL ## 驗(yàn)證節(jié)點(diǎn)資源 score 值3、修改資源配置 [shell]# pcs resource update WebFS ## 更新資源配置 [shell]# pcs resource delete WebFS ## 刪除指定資源4、管理群集資源 [shell]# pcs resource disable ClusterIP ## 禁用資源 [shell]# pcs resource enable ClusterIP ## 啟用資源 [shell]# pcs resource failcount show ClusterIP ## 顯示指定資源的錯(cuò)誤計(jì)數(shù) [shell]# pcs resource failcount reset ClusterIP ## 清除指定資源的錯(cuò)誤計(jì)數(shù) [shell]# pcs resource cleanup ClusterIP ## 清除指定資源的狀態(tài)與錯(cuò)誤計(jì)數(shù)四、配置Fencing設(shè)備，啟用STONITH1、查詢Fence設(shè)備資源 [shell]# pcs stonith list ## 查看支持Fence列表 [shell]# pcs stonith describe agent_name ## 查看Fence資源使用參數(shù)，pcs stonith describe fence_vmware_soap2、配置fence設(shè)備資源 [shell]# pcs stonith create ipmi-fencing fence_ipmilan / pcmk_host_list="pcmk-1 pcmk-2" ipaddr="10.0.0.1" login=testuser passwd=acd123 / op monitor interval=60smark:If the device does not support the standard port parameter or may provide additional ones, you may also need to set the special pcmk_host_argument parameter. See man stonithd for details.If the device does not know how to fence nodes based on their uname, you may also need to set the special pcmk_host_map parameter. See man stonithd for details.If the device does not support the list command, you may also need to set the special pcmk_host_list and/or pcmk_host_check parameters. See man stonithd for details.If the device does not expect the victim to be specified with the port parameter, you may also need to set the special pcmk_host_argument parameter. See man stonithd for details.example: pcmk_host_argument="uuid" pcmk_host_map="node11:4;node12:5;node13:6" pcmk_host_list="node11,node12" pcmk_host_check="static-list"3、配置VMWARE (fence_vmware_soap) 特別說明：本次實(shí)例中使用了第3項(xiàng)（pcs stonith create vmware-fencing fence_vmware_soap）這個(gè)指定pcmk配置參數(shù)才能正常執(zhí)行Fencing動(dòng)作。3.1、確認(rèn)vmware虛擬機(jī)的狀態(tài)： [shell]# fence_vmware_soap -o list -a vcenter.example.com -l cluster-admin -p <password> -z ## 獲取虛擬機(jī)UUID [shell]# fence_vmware_soap -o status -a vcenter.example.com -l cluster-admin -p <password> -z -U <UUID>## 查看狀態(tài) [shell]# fence_vmware_soap -o status -a vcenter.example.com -l cluster-admin -p <password> -z -n <vm name>3.2、配置fence_vmware_soap [shell]# pcs stonith create vmware-fencing-node11 fence_vmware_soap / action="reboot" ipaddr="192.168.10.10" login="vmuser" passwd="vmuserpd" ssl="1" / port="node11" shell_timeout=60s login_timeout=60s op monitor interval=90s [shell]# pcs stonith create vmware-fencing-node11 fence_vmware_soap / action="reboot" ipaddr="192.168.10.10" login="vmuser" passwd="vmuserpd" ssl="1" / uuid="421dec5f-c484-3d69-ddfb-65af46530581" shell_timeout=60s login_timeout=60s op monitor interval=90s [shell]# pcs stonith create vmware-fencing fence_vmware_soap / action="reboot" ipaddr="192.168.10.10" login="vmuser" passwd="vmuserpd" ssl="1" / pcmk_host_argument="uuid" pcmk_host_check="static-list" pcmk_host_list="node11,node12" / pcmk_host_map="node11:421dec5f-c484-3d69-ddfb-65af46530581;node12:421dec5f-c484-3d69-ddfb-65af46530582" / shell_timeout=60s login_timeout=60s op monitor interval=90s注：如果配置fence_vmware_soap設(shè)備時(shí)用port=vm name在測試時(shí)不能識(shí)別，則使用uuid=vm uuid代替；建議使用 pcmk_host_argument、pcmk_host_map、pcmk_host_check、pcmk_host_list 參數(shù)指明節(jié)點(diǎn)與設(shè)備端口關(guān)系，格式： pcmk_host_argument="uuid" pcmk_host_map="node11:uuid4;node12:uuid5;node13:uuid6" pcmk_host_list="node11,node12,node13" pcmk_host_check="static-list"4、配置SCSI [shell]# ls /dev/disk/by-id/wwn-* ## 獲取Fencing磁盤UUID號(hào)，磁盤須未格式化 [shell]# pcs stonith create iscsi-fencing fence_scsi / action="reboot" devices="/dev/disk/by-id/wwn-0x600e002" meta provides=unfencing5、配置DELL DRAC [shell]# pcs stonith create dell-fencing-node11 fence_drac.....6、管理 STONITH [shell]# pcs resource clone vmware-fencing ## clone stonith資源，供多節(jié)點(diǎn)啟動(dòng) [shell]# pcs property set stonith-enabled=true ## 啟用 stonith 組件功能 [shell]# pcs stonith cleanup vmware-fencing ## 清除Fence資源的狀態(tài)與錯(cuò)誤計(jì)數(shù) [shell]# pcs stonith fence node11 ## fencing指定節(jié)點(diǎn)五、群集操作命令1、驗(yàn)證群集安裝 [shell]# pacemakerd -F ## 查看pacemaker組件，ps axf | grep pacemaker [shell]# corosync-cfgtool -s ## 查看corosync序號(hào) [shell]# corosync-cmapctl | grep members ## corosync 2.3.x [shell]# corosync-objctl | grep members ## corosync 1.4.x2、查看群集資源 [shell]# pcs resource standards ## 查看支持資源類型 [shell]# pcs resource providers ## 查看資源提供商 [shell]# pcs resource agents ## 查看所有資源代理 [shell]# pcs resource list ## 查看支持資源列表 [shell]# pcs stonith list ## 查看支持Fence列表 [shell]# pcs property list --all ## 顯示群集默認(rèn)變量參數(shù) [shell]# crm_simulate -sL ## 檢驗(yàn)資源 score 值3、使用群集腳本 [shell]# pcs cluster cib ra_cfg ## 將群集資源配置信息保存在指定文件 [shell]# pcs -f ra_cfg resource create ## 創(chuàng)建群集資源并保存在指定文件中（而非保存在運(yùn)行配置） [shell]# pcs -f ra_cfg resource show ## 顯示指定文件的配置信息，檢查無誤后 [shell]# pcs cluster cib-push ra_cfg ## 將指定配置文件加載到運(yùn)行配置中4、STONITH 設(shè)備操作 [shell]# stonith_admin -I ## 查詢fence設(shè)備 [shell]# stonith_admin -M -a agent_name ## 查詢fence設(shè)備的元數(shù)據(jù)，stonith_admin -M -a fence_vmware_soap [shell]# stonith_admin --reboot nodename ## 測試 STONITH 設(shè)備5、查看群集配置 [shell]# crm_verify -L -V ## 檢查配置有無錯(cuò)誤 [shell]# pcs property ## 查看群集屬性 [shell]# pcs stonith ## 查看stonith [shell]# pcs constraint ## 查看資源約束 [shell]# pcs config ## 查看群集資源配置 [shell]# pcs cluster cib ## 以xml格式顯示群集配置6、管理群集 [shell]# pcs status ## 查看群集狀態(tài) [shell]# pcs status cluster [shell]# pcs status corosync [shell]# pcs cluster stop [node11] ## 停止群集 [shell]# pcs cluster start --all ## 啟動(dòng)群集 [shell]# pcs cluster standby node11 ## 將節(jié)點(diǎn)置為后備standby狀態(tài)，pcs cluster unstandby node11 [shell]# pcs cluster destroy [--all] ## 刪除群集，[--all]同時(shí)恢復(fù)corosync.conf文件 [shell]# pcs resource cleanup ClusterIP ## 清除指定資源的狀態(tài)與錯(cuò)誤計(jì)數(shù) [shell]# pcs stonith cleanup vmware-fencing ## 清除Fence資源的狀態(tài)與錯(cuò)誤計(jì)數(shù)
