redis cluster搭建

redis cluster搭建

從redis官網看最新穩定版本已經更新到3.2.6了，我這裡補充一篇以前3.2.0的集群配置。
說明：

redis從3.0版本以後開始支持cluster模式，原理和功能大家可以上網查，對於運維人員來說最重要的去掉了代理層、避免出現單點故障，但是redis cluster至少需要三個主節點和三個從節點。每一對主從形成一組關系。因此需要三台服務器，每台服務器上兩個實例。因為一個主對應一個從，所以一台服務器上不能同時是一個主和從，必須三台服務器分開主從，來保證服務的正常運行。我這裡為了演示方便就在一台服務器上部署。注意：生產環境必須是三台機器，不然達不到高可用。

部署環境：

主機名

IP地址

操作系統版本

用途

test01

192.168.2.9

CentOS6.3（64位）

實例 7000/7001

test02

192.168.2.10

CentOS 6.3（64位）

實例 7002/7003

test03

192.168.2.7

CentOS 6.3（64位）

實例 7004/7005

部署步驟：

1、編譯安裝redis。

把安裝包放在/data/下解壓並編譯

[root@test01 data]# tar -zxf redis-3.2.0.tar.gz
[root@test01 redis]# cd redis-3.2.0
[root@test01 redis-3.2.0]# make && make install
[root@test01 redis]# ln -s /data/redis/redis-3.2.0 /usr/local/redis

2、創建redis集群節點

[root@test01 local]# mkdir redis_cluster
[root@test01 local]# cd redis_cluster/
[root@test01 redis_cluster]# mkdir 7000 7001

3、復制默認的節點配置文件到集群節點中

[root@test01 redis_cluster]# cp /usr/local/redis/redis.conf ./7000

4、修改默認配置文件

[root@test01 7000]# vim redis.conf
daemonize    yes                  //redis後台運行
pidfile  /var/run/redis_7000.pid   //pidfile文件對應7000
port  7000                         //端口7000
cluster-enabled  yes              //開啟集群  把注釋#去掉
cluster-config-file  nodes.conf    //集群的配置  配置文件首次啟動自動生成
cluster-node-timeout  6000      //請求超時
appendonly  yes                  //日志開啟，環境測試開啟，正式環境建議關閉
bind 192.168.2.9             //修改監聽的地址為本機地址

5、修改完後把配置文件復制到7001上並修改相應的端口。

[root@test01 7000]# cp redis.conf../7001/

6、另外兩台服務器重復上面配置。

7、啟動各個節點，注意一定要到端口目錄下去啟動，不然有的啟動不起來。

[root@test01 7000]# /usr/local/redis/src/redis-server redis.conf
[root@test01 7000]# cd ../7001/
[root@test01 7001]# /usr/local/redis/src/redis-server redis.conf

查看啟動進程

[root@test01 7001]# ps -ef|grep redis
root      8858     1  0 11:38 ?        00:00:00 /usr/local/redis/src/redis-server 192.168.2.9:7000 [cluster]
root      8865     1  0 11:39 ?        00:00:00 /usr/local/redis/src/redis-server 192.168.2.9:7001 [cluster]
root      8870 27799  0 11:39 pts/1    00:00:00 grep redis

再分別啟動其余服務器節點。

8、創建集群必須的軟件，三台服務器都執行

[root@test01 7000]# yum -y install ruby ruby-devel rubygems rpm-build
[root@test01 7000]# ruby -v
ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
[root@redis 7000]# rpm -qa|grep rubyge
rubygems-1.3.7-5.el6.noarch
[root@redis 7000]# gem install redis
Successfully installed redis-3.3.0
1 gem installed
Installing ri documentation for redis-3.3.0...
Installing RDoc documentation for redis-3.3.0.

9、確認所有的節點都啟動，接下來使用參數create 創建

[root@test01 7000]# /usr/local/redis/src/redis-trib.rb create --replicas 1 192.168.2.9:7000 192.168.2.10:7002 192.168.2.7:7004 192.168.2.10:7003 192.168.2.7:7005 192.168.2.9:7001
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.2.7:7004
192.168.2.9:7000
192.168.2.10:7002
Adding replica 192.168.2.9:7001 to 192.168.2.7:7004
Adding replica 192.168.2.7:7005 to 192.168.2.9:7000
Adding replica 192.168.2.10:7003 to 192.168.2.10:7002 
M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
slots:5461-10922 (5462 slots) master
M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
slots:10923-16383 (5461 slots) master
M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
slots:0-5460 (5461 slots) master
S: 6f13ca12a9be3b0c093d02c81fed337307f295af 192.168.2.10:7003
replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.2.9:7000)
M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
slots:5461-10922 (5462 slots) master
M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
slots:10923-16383 (5461 slots) master
M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
slots:0-5460 (5461 slots) master
M: 6f13ca12a9be3b0c093d02c81fed337307f295af 192.168.2.10:7003
slots: (0 slots) master
replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
M: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
slots: (0 slots) master
replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
M: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
slots: (0 slots) master
replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

--replicas 1 創建集群指定一個從節點。

因為創建集群的時候不能指定主從，它是隨機的。加入集群的節點順序不分前後，如果一台服務器上正好有主節點和從節點，而且是一組的話，必須將同一台服務器上的主從節點踢掉，換成不同服務器的節點。
從上面可以看到很不辛7003和7002分配了一台服務器上。經過測試一般都會有一台服務器上存在同一組的主和從節點。
主從關系圖：
192.168.2.10:7002 192.168.2.10:7003
192.168.2.9:7000 192.168.2.7:7005
192.168.2.7:7004 192.168.2.9:7001

10、我們把192.168.2.10:7003和192.168.2.7:7005調換下。

刪除7003節點

[root@test01 7000]# /usr/local/redis/src/redis-trib.rb del-node 192.168.2.10:7003 6f13ca12a9be3b0c093d02c81fed337307f295af
>>> Removing node 6f13ca12a9be3b0c093d02c81fed337307f295af from cluster 192.168.2.10:7003
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

del-node 刪除節點後面跟主機和對應的端口還有實例的id。這樣才能刪除。

11、修改192.168.2.7:7005的主為192.168.2.10:7002

進入192.168.2.7:7005的redis

[root@test02 redis_cluster]# /usr/local/redis/src/redis-cli -c -p 7005 -h 192.168.2.7
192.168.2.7:7005> cluster replicate 01594f84df9e743a74a47f9aaa58fa41402dfe25 #新主的id
OK

查看現在主從狀態

[root@test02 redis_cluster]# /usr/local/redis/src/redis-trib.rb check 192.168.2.10:7002
>>> Performing Cluster Check (using node 192.168.2.10:7002)
M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
slots: (0 slots) slave
replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
slots: (0 slots) slave
replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到主從關系已經改變。

12、再添加剛刪除的7003節點。

[root@test02 redis_cluster]# ps -ef|grep redis
root      4441     1  0 11:45 ?        00:00:13 /usr/local/redis/src/redis-server 192.168.2.10:7002 [cluster]
root      4673  4275  0 15:02 pts/0    00:00:00 grep redis
[root@test02 redis_cluster]# cd 7003/
[root@test02 7003]# ls
appendonly.aof  dump.rdb  nodes.conf  redis.conf
[root@test02 7003]# rm -f appendonly.aof dump.rdb nodes.conf
[root@test02 7003]# /usr/local/redis/src/redis-server redis.conf
[root@test02 7003]# ps -ef|grep redis
root      4441     1  0 11:45 ?        00:00:14 /usr/local/redis/src/redis-server 192.168.2.10:7002 [cluster]
root      4677     1  0 15:02 ?        00:00:00 /usr/local/redis/src/redis-server 192.168.2.10:7003 [cluster]
root      4681  4275  0 15:02 pts/0    00:00:00 grep redis

13、執行下列命令添加從指定的主節點。

[root@test02 7003]# /usr/local/redis/src/redis-trib.rb add-node --slave --master-id bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.10:7003 192.168.2.9:7000
>>> Adding node 192.168.2.7:7005 to cluster 192.168.2.9:7000
>>> Performing Cluster Check (using node 192.168.2.9:7000)
M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
slots:5461-10922 (5462 slots) master
0 additional replica(s)
S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
slots: (0 slots) slave
replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
slots: (0 slots) slave
replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.2.10:7003 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.2.9:7000.
[OK] New node added correctly.

從上面顯示7003節點成為7000的從節點。

14、查看主從現在情況

[root@test02 7003]# /usr/local/redis/src/redis-trib.rb check 192.168.2.10:7002
>>> Performing Cluster Check (using node 192.168.2.10:7002)
M: 01594f84df9e743a74a47f9aaa58fa41402dfe25 192.168.2.10:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: bede5b72dfbb5274e52a1f0c5f6b43170afec8af 192.168.2.9:7000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 2e26f92f8dca2dfb13f159a59b6260f106bc8cb4 192.168.2.10:7003
slots: (0 slots) slave
replicates bede5b72dfbb5274e52a1f0c5f6b43170afec8af
M: 27cedfdc0a648b9141736f156a4d89828d7bf695 192.168.2.7:7004
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: f167b98d8f78bfdb4c1823c0d6be7f1a12aff194 192.168.2.9:7001
slots: (0 slots) slave
replicates 27cedfdc0a648b9141736f156a4d89828d7bf695
S: 00333fa0ac74863e86c3108f6040abe1183a2b9b 192.168.2.7:7005
slots: (0 slots) slave
replicates 01594f84df9e743a74a47f9aaa58fa41402dfe25
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered

這樣就是一個正常的集群了~~運維人員再也不會因為宕機或者redis掛了的問題頭疼了！