最新的 1.6 版总算提供了 Replica Sets，比起有点莫名其妙的 Replica Pairs，这才是高可用集群所需要的。
1. Replica Sets
Replica Sets 使用 n 个 Mongod 节点，构建具备自动容错转移(auto-failover)、自动恢复(auto-recovery) 的高可用方案。通常使用 3 个 mongod 实例，或者 2 mongod + 1 arbiter 方案。
(1) 首先启动所需的 Mongod 节点。注意使用 replSet 参数指定 Sets Name。
$ sudo mkdir -p /var/mongodb/0 $ sudo mkdir -p /var/mongodb/1 $ sudo mkdir -p /var/mongodb/2 $ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/0 --port 27017 --replSet myset forked process: 1166 all output going to: /dev/null $ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/1 --port 27018 --replSet myset forked process: 1173 all output going to: /dev/null $ sudo ./mongod --fork --logpath /dev/null --dbpath /var/mongodb/2 --port 27019 --replSet myset forked process: 1180 all output going to: /dev/null
(2) 使用 mongo 配置 Replica Sets。
$ ./mongo
MongoDB shell version: 1.6.1
connecting to: test
> cfg = { _id: "myset", members: [
... { _id:0, host:"localhost:27017" },
... { _id:1, host:"localhost:27018" },
... { _id:2, host:"localhost:27019" }
... ]}
> rs.initiate(cfg)
{
"info" : "Config now saved locally. Should come online in about a minute.",
"ok" : 1
}
> rs.conf()
{
"_id" : "myset",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
}
]
}
如此 Replica Sets 就算配置成功。
相关配置数据保存在 local 数据库中。
> show dbs
admin
local
> use local
switched to db local
> show collections
oplog.rs
slaves
system.indexes
system.replset
> db.system.replset.find()
{ "_id" : "myset", "version" : 1, "members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
}
] }
oplog.rs 是一个固定长度的 capped collection，用于记录 Replica Sets 操作日志。
(3) 可以用 isMaster 和 status 命令查看 Replica Sets 状态。
> rs.isMaster()
{
"ismaster" : true,
"secondary" : false,
"hosts" : [
"localhost:27017",
"localhost:27019",
"localhost:27018"
],
"ok" : 1
}
> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:21:13 GMT+0800 (CST)",
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "yuhen-server64:27017",
"health" : 1,
"state" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"uptime" : 280,
"lastHeartbeat" : "Sat Aug 21 2010 15:21:11 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 2,
"uptime" : 284,
"lastHeartbeat" : "Sat Aug 21 2010 15:21:11 GMT+0800 (CST)"
}
],
"ok" : 1
}
在同一时刻，每组 Replica Sets 只有一个 Primary，用于接受写操作。而后会异步复制到其他成员数据库中。一旦 primary 死掉，会自动投票选出接任的 primary 来，原服务器恢复后成为普通成员。如果数据尚未从先前的 primary 复制到成员服务器，有可能会丢失数据。
(4) 为了观察数据复制和容错迁移，我们可以开几个终端，在前台运行 mongod。
$ sudo ./mongod --port 27017 --dbpath /var/mongodb/0 --replSet myset
在 mongo 中向 primary (27017) 插入数据。
> use test
switched to db test
> db.users.insert({name:"user1"})
会在所有 mongod 输出记录中看到相关信息。
# 27017 #
Sat Aug 21 15:35:50 [conn2] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:50 [conn2] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }
Sat Aug 21 15:35:50 [conn2] done for 0 records 0.01secs
Sat Aug 21 15:35:50 [conn12] getmore local.oplog.rs cid:4961370576520295111 getMore: { ts: { $gte: new Date(5507762976880328705) } } bytes:118 nreturned:1 3782ms
Sat Aug 21 15:35:50 [conn2] insert test.users 1029ms
# 27018 #
Sat Aug 21 15:35:51 [rs_sync] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:51 [rs_sync] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }
# 27019 #
Sat Aug 21 15:35:51 [rs_sync] building new index on { _id: 1 } for test.users
Sat Aug 21 15:35:51 [rs_sync] Buildindex test.users idxNo:0 { name: "_id_", ns: "test.users", key: { _id: 1 } }
我们用 CTRL + C 关掉 primary mongd。
# 27018 # Sat Aug 21 15:40:16 [ReplSetHealthPollTask] replSet info localhost:27017 is now down (or slow to respond) Sat Aug 21 15:40:27 [rs_sync] replSet SECONDARY # 27019 # Sat Aug 21 15:40:16 [ReplSetHealthPollTask] replSet info localhost:27017 is now down (or slow to respond) Sat Aug 21 15:40:16 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server Sat Aug 21 15:40:16 [rs Manager] replSet info electSelf 2 Sat Aug 21 15:40:16 [rs Manager] replSet PRIMARY
Mongod 27019 被选为 Primary。
因为 27107 的 socket 关闭，所以 mongo 需要重新连接。
$ ./mongo localhost:27019
MongoDB shell version: 1.6.1
connecting to: localhost:27018/test
> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:41:40 GMT+0800 (CST)",
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 0,
"state" : 1,
"uptime" : 0,
"lastHeartbeat" : "Sat Aug 21 2010 15:40:14 GMT+0800 (CST)",
"errmsg" : "connect/transport error"
},
{
"_id" : 1,
"name" : "yuhen-server64:27018",
"health" : 1,
"state" : 2,
"self" : true
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 1,
"uptime" : 486,
"lastHeartbeat" : "Sat Aug 21 2010 15:41:38 GMT+0800 (CST)"
}
],
"ok" : 1
}
查询先前插入的记录正常。
> db.users.find()
{ "_id" : ObjectId("4c6f81d58e4719f03d5ccc65"), "name" : "user1" }
我们也可以将 mongo 连接到 27018，不过无法查询数据。
重新启动 27017。
# 27017 #
Sat Aug 21 15:44:54 [rs Manager] replSet can't see a majority, will not try to elect self
Sat Aug 21 15:44:56 [ReplSetHealthPollTask] replSet info localhost:27019 is now up
Sat Aug 21 15:44:56 [ReplSetHealthPollTask] replSet info localhost:27018 is now up
Sat Aug 21 15:44:56 [rs_sync] building new index on { _id: 1 } for local.me
Sat Aug 21 15:44:56 [rs_sync] Buildindex local.me idxNo:0 { name: "_id_", ns: "local.me", key: { _id: 1 } }
Sat Aug 21 15:44:56 [rs_sync] done for 0 records 0.002secs
Sat Aug 21 15:44:56 [rs_sync] replSet SECONDARY
# 27018 #
Sat Aug 21 15:44:55 [ReplSetHealthPollTask] replSet info localhost:27017 is now up
# 27019 #
Sat Aug 21 15:44:54 [ReplSetHealthPollTask] replSet info localhost:27017 is now up
rs.status 中 heartbeat 恢复正常，但不再是 Primary。
> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)",
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 2,
"uptime" : 110,
"lastHeartbeat" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)"
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"uptime" : 790,
"lastHeartbeat" : "Sat Aug 21 2010 15:46:44 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "yuhen-server64:27019",
"health" : 1,
"state" : 1,
"self" : true
}
],
"ok" : 1
}
(5) 我们还可以在运行时添加成员。
$ sudo ./mongod --fork --port 27020 --dbpath /var/mongodb/3 --logpath /dev/null --replSet myset forked process: 2139 all output going to: /dev/null
注意: 必须连接到 primary 才能添加成员。
$ ./mongo localhost:27019
MongoDB shell version: 1.6.1
connecting to: localhost:27020/test
> rs.add("localhost:27029")
{ "ok" : 1 }
> rs.conf()
{
"_id" : "myset",
"version" : 2,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
},
{
"_id" : 1,
"host" : "localhost:27018"
},
{
"_id" : 2,
"host" : "localhost:27019"
},
{
"_id" : 3,
"host" : "localhost:27020"
}
]
}
查看状态。
> rs.status()
{
"set" : "myset",
"date" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)",
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 2,
"uptime" : 94,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 1,
"uptime" : 94,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
},
{
"_id" : 2,
"name" : "yuhen-server64:27019",
"health" : 1,
"state" : 2,
"self" : true
},
{
"_id" : 3,
"name" : "localhost:27020",
"health" : 1,
"state" : 2,
"uptime" : 92,
"lastHeartbeat" : "Sat Aug 21 2010 15:52:54 GMT+0800 (CST)"
}
],
"ok" : 1
}
(6) 从客户端连接 Replica Sets，需要 drivers 支持。
$ ipython
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
IPython 0.10 -- An enhanced Interactive Python.
In [1]: import pymongo
In [2]: conn = pymongo.Connection(host = ["localhost:27017", "localhost:27018", "localhost:27019", "localhost:27020"])
In [3]: db = conn.test
In [4]: for u in db.users.find(): print u
...:
{u'_id': ObjectId('4c6f81d58e4719f03d5ccc65'), u'name': u'user1'}
{u'_id': ObjectId('4c6f82ad8e4719f03d5ccc66'), u'name': u'user2'}
In [5]: db.users.insert({"name":"user3"})
Out[5]: ObjectId('4c6f8872499b1408b7000000')
从其他终端关掉 primary，看看效果。
In [6]: db.users.count() --------------------------------------------------------------------------- AutoReconnect Traceback (most recent call last) AutoReconnect: connection closed
由于连接被中断，引发异常也很正常，不过 pymongo 会自动重新连接，再次操作时一切正常。
In [7]: db.users.count()
Out[7]: 3
In [8]: for u in db.users.find(): print u
...:
{u'_id': ObjectId('4c6f81d58e4719f03d5ccc65'), u'name': u'user1'}
{u'_id': ObjectId('4c6f82ad8e4719f03d5ccc66'), u'name': u'user2'}
{u'_id': ObjectId('4c6f8872499b1408b7000000'), u'name': u'user3'}
因此编码时切记需要做异常保护。