一、背景知识
二、机器分配
| IP | Roles | Hostname | 备注 |
| 222.31.76.209 | client | localhost.localdomain | |
| 222.31.76.178 | mds&monitor | ceph_mds | |
| 222.31.76.74 | osd | ceph_osd0 | |
| 222.31.76.67 | osd | ceph_osd1 | |
| 222.31.76.235 | osd | ceph_osd2 |
内核版本：linux-3.8.8.tar.xz （stable2013-04-17）
ceph版本：ceph-0.60.tar.gz （01-Apr-201317:42）
三、编译与配置
-lpthread saves the day... yes
checking
for uuid_parse
in
-luuid... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: libuuid not found
See `config.log'
for more details.
for __res_nquery
in
-lresolv... yes
checking
for add_key
in
-lkeyutils... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: libkeyutils not found
See `config.log'
for more details.
-config is at least version
0.
9.
0... yes
checking
for CRYPTOPP... no
checking
for library containing _ZTIN8CryptoPP14CBC_EncryptionE... no
checking
for NSS... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: no suitable crypto library found
See `config.log'
for more details.
-config is at least version
0.
9.
0... yes
checking
for CRYPTOPP... no
checking
for library containing _ZTIN8CryptoPP14CBC_EncryptionE...
-lcryptopp
checking
for NSS... no
configure
: using cryptopp
for cryptography
checking
for FCGX_Init
in
-lfcgi... no
checking
for fuse_main
in
-lfuse... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: no FUSE found (use --without-fuse to disable)
See `config.log'
for more details.
for fuse_main
in
-lfuse... yes
checking
for fuse_getgroups... no
checking jni.h usability... no
checking jni.h presence... no
checking
for jni.h... no
checking
for LIBEDIT... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: No usable version of libedit found.
See `config.log'
for more details.
for LIBEDIT... yes
checking atomic_ops.h usability... no
checking atomic_ops.h presence... no
checking
for atomic_ops.h... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: no libatomic-ops found (use --without-libatomic-ops to disable)
See `config.log'
for more details.
for LIBEDIT... yes
checking
for snappy_compress
in
-lsnappy... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: libsnappy not found
See `config.log'
for more details.
for snappy_compress
in
-lsnappy... yes
checking
for leveldb_open
in
-lleveldb... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: libleveldb not found
See `config.log'
for more details.
for leveldb_open
in
-lleveldb... yes
checking
for io_submit
in
-laio... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: libaio not found
See `config.log'
for more details.
for sys
/wait.h that is POSIX.
1 compatible... yes
checking boost
/spirit
/include
/classic_core.hpp usability... no
checking boost
/spirit
/include
/classic_core.hpp presence... no
checking
for boost
/spirit
/include
/classic_core.hpp... no
checking boost
/spirit.hpp usability... no
checking boost
/spirit.hpp presence... no
checking
for boost
/spirit.hpp... no
configure
: error
:
in
`
/cwn
/ceph
/ceph
-
0.
60
':
configure: error: "Can't find
boost spirit headers
"
See `config.log' for more details.
if more special flags are required
for pthreads... no
checking whether to check
for GCC pthread
/shared inconsistencies... yes
checking whether
-pthread is sufficient with
-shared... yes
configure
: creating .
/config.status
config.status
: creating Makefile
config.status
: creating scripts
/gtest
-config
config.status
: creating build
-aux
/config.h
config.status
: executing depfiles commands
config.status
: executing libtool commands
CXXLD osdmaptool
CXX ceph_dencoder
-ceph_dencoder.o
test
/encoding
/ceph_dencoder.cc
: In function
'int main(int, const char**)'
:
test
/encoding
/ceph_dencoder.cc
:
196
: note
: variable tracking size limit exceeded with
-fvar
-tracking
-assignments, retrying without
CXX ceph_dencoder
-rgw_dencoder.o
In file included from rgw
/rgw_dencoder.cc
:
6
:
rgw/rgw_acl_s3.h:9:19: error: expat.h: No such file or directory
In file included from rgw
/rgw_acl_s3.h
:
12,
from rgw
/rgw_dencoder.cc
:
6
:
rgw/rgw_xml.h:62: error: 'XML_Parser' does not name a type
make[
3]
:
*
*
* [ceph_dencoder
-rgw_dencoder.o] Error
1
make[
3]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[
1]
:
*
*
* [all] Error
2
make[
1]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
make: *** [all-recursive] Error 1
-dencoder
CXXLD cephfs
CXXLD librados
-config
CXXLD ceph
-fuse
CCLD rbd
-fuse
CCLD
mount.ceph
CXXLD rbd
CXXLD rados
CXXLD ceph
-syn
make[
3]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[
1]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
Making all in man
make[1]: Entering directory `/cwn/ceph/ceph-0.60/man'
make[
1]
: Nothing to be
done
for
`all
'.
make[1]: Leaving directory `/cwn/ceph/ceph-0.60/man'
: install
: ranlib
/usr
/local
/lib
/rados
-classes
/libcls_kvs.a
libtool
: finish
: PATH
=
"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/sbin" ldconfig
-n
/usr
/local
/lib
/rados
-classes
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
--
Libraries have been installed
in
:
/usr
/local
/lib
/rados
-classes
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool,
and
specify the full pathname of the library,
or use the
`
-LLIBDIR
'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the
`LD_RUN_PATH
' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to
`
/etc
/ld.so.conf
'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/usr/local/lib/ceph" || /bin/mkdir -p "/usr/local/lib/ceph"
/usr/bin/install -c ceph_common.sh '
/usr
/local
/lib
/ceph
'
make[4]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[
3]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
make[2]: Leaving directory `/cwn/ceph/ceph-0.60/src'
make[
1]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/src
'
Making install in man
make[1]: Entering directory `/cwn/ceph/ceph-0.60/man'
make[
2]
: Entering directory
`
/cwn
/ceph
/ceph
-
0.
60
/man
'
make[2]: Nothing to be done for `install-exec-am'.
test
-z
"/usr/local/share/man/man8"
||
/bin
/mkdir
-p
"/usr/local/share/man/man8"
/usr
/bin
/install
-c
-m
644 ceph
-osd.
8 ceph
-mds.
8 ceph
-mon.
8 mkcephfs.
8 ceph
-fuse.
8 ceph
-syn.
8 crushtool.
8 osdmaptool.
8 monmaptool.
8 ceph
-conf.
8 ceph
-run.
8 ceph.
8 mount.ceph.
8 radosgw.
8 radosgw
-admin.
8 ceph
-authtool.
8 rados.
8 librados
-config.
8 rbd.
8 ceph
-clsinfo.
8 ceph
-debugpack.
8 cephfs.
8 ceph
-dencoder.
8 ceph
-rbdnamer.
8 rbd
-fuse.
8
'/usr/local/share/man/man8'
make[
2]
: Leaving directory
`
/cwn
/ceph
/ceph
-
0.
60
/man
'
make[1]: Leaving directory `/cwn/ceph/ceph-0.60/man'
这个文件要位于/etc/ceph下面，如果在./configure时没有修改prefix的话，则应该是在/usr/local/etc/ceph下。
//存放log，现在ceph自己还不自动建这个目录
# cat /usr/local/etc/ceph/ceph.conf
;
; Sample ceph ceph.conf file.
;
; This file defines cluster membership, the various locations
; that Ceph stores data,
and any other runtime options.
; If a
'host' is defined
for a daemon, the init.d start
/stop script will
; verify that it matches the hostname (
or
else ignore it). If it is
;
not defined, it is assumed that the daemon is intended to start on
; the current host (e.g.,
in a setup with a startup.conf on each
; node).
; The variables $type, $id
and $name are available to use
in paths
; $type
= The type of daemon, possible values
: mon, mds
and osd
; $id
= The ID of the daemon,
for mon.alpha, $id will be alpha
; $name
= $type.$id
; For example
:
; osd.
0
; $type
= osd
; $id
=
0
; $name
= osd.
0
; mon.beta
; $type
= mon
; $id
= beta
; $name
= mon.beta
; global
[global]
; enable secure authentication
; auth supported
= cephx
; allow ourselves to open a lot of files
max open files
=
131072
; set log file
log file
=
/var
/log
/ceph
/$name.log
; log_to_syslog
= true ; uncomment this line to log to syslog
; set up pid files
pid file
=
/var
/run
/ceph
/$name.pid
; If you want to run a IPv6 cluster, set this to true. Dual
-stack isn
't possible
;ms bind ipv6 = true
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /data/mon$id
; If you are using for example the RADOS Gateway and want to have your newly created
; pools a higher replication level, you can set a default
;osd pool default size = 3
; You can also specify a CRUSH rule for new pools
; Wiki: http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH
;osd pool default crush rule = 0
; Timing is critical for monitors, but if you want to allow the clocks to drift a
; bit more, you can specify the max drift.
;mon clock drift allowed = 1
; Tell the monitor to backoff from this warning for 30 seconds
;mon clock drift warn backoff = 30
; logging, for debugging monitor crashes, in order of
; their likelihood of being helpful 
debug ms = 1
;debug mon = 20
;debug paxos = 20
;debug auth = 20
[mon.0]
host = ceph_mds
mon addr = 222.31.76.178:6789
; mds
; You need at least one. Define two to get a standby.
[mds]
; where the mds keeps it's secret encryption keys
keyring
=
/data
/keyring.$name
; mds logging to debug issues.
;debug ms
=
1
;debug mds
=
20
[mds.alpha]
host
= ceph_mds
; osd
; You need at least one. Two
if you want data to be replicated.
; Define as many as you like.
[osd]
sudo
= true
; This is where the osd expects its data
osd data
=
/data
/osd$id
; Ideally, make the journal a separate disk
or partition.
;
1
-
10GB should be enough; more
if you have fast
or many
; disks. You can use a file under the osd data dir
if need be
; (e.g.
/data
/$name
/journal), but it will be slower than a
; separate disk
or partition.
; This is an example of a file
-based journal.
osd journal
=
/data
/$name
/journal
osd journal size
=
1000 ; journal size,
in megabytes
; If you want to run the journal on a tmpfs (don
't), disable DirectIO
;journal dio = false
; You can change the number of recovery operations to speed up recovery
; or slow it down if your machines can't handle it
; osd recovery max active
=
3
; osd logging to debug osd issues,
in order of likelihood of being
; helpful
;debug ms
=
1
;debug osd
=
20
;debug filestore
=
20
;debug journal
=
20
;
### The below options only apply if you're using mkcephfs
;
### and the devs options
; The filesystem used on the volumes
osd mkfs type
= btrfs
; If you want to specify some other mount options, you can
do so.
;
for other filesystems use
'osd mount options $fstype'
osd mount options btrfs
= rw,noatime
; The options used to format the filesystem via mkfs.$fstype
;
for other filesystems use
'osd mkfs options $fstype'
; osd mkfs options btrfs
=
[osd.
0]
host
= ceph_osd0
;
if
'devs' is
not specified, you
're responsible for
; setting up the 'osd data
' dir.
btrfs devs = /dev/mapper/VolGroup-lv_ceph
[osd.1]
host = ceph_osd1
btrfs devs = /dev/mapper/VolGroup-lv_ceph
[osd.2]
host = ceph_osd2
btrfs devs = /dev/mapper/VolGroup-lv_ceph
4. 配置网络
5.
创建文件系统并启动。以下命令在监控节点进行！
# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
[
/usr
/local
/etc
/ceph
/fetch_config
/tmp
/fetched.ceph.conf.
2693]
The authenticity of host
'ceph_mds (127.0.0.1)' can
't be established.
RSA key fingerprint is a7:c8:b8:2e:86:ea:89:ff:11:93:e9:29:68:b5:7c:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph_mds
' (RSA) to the list of known hosts.
ceph.conf 100% 4436 4.3KB/s 00:00
temp dir is /tmp/mkcephfs.tIHQnX8vkw
preparing monmap in /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool --create --clobber --add 0 222.31.76.178:6789 --print /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: monmap file /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: generated fsid f998ee83-9eba-4de2-94e3-14f235ef840c
epoch 0
fsid f998ee83-9eba-4de2-94e3-14f235ef840c
last_changed 2013-05-31 08:22:52.972189
created 2013-05-31 08:22:52.972189
0: 222.31.76.178:6789/0 mon.0
/usr/local/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.tIHQnX8vkw/monmap (1 monitors)
=== osd.0 ===
pushing conf and monmap to ceph_osd0:/tmp/mkfs.ceph.0b3c65941572123eb704d9d614411fc1
scp: /etc/ceph/ceph.conf: No such file or directory
# cat cp_ceph_conf.sh
cp
/usr
/local
/etc
/ceph
/ceph.conf
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd0
:
/usr
/local
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd0
:
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd1
:
/usr
/local
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd1
:
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd2
:
/usr
/local
/etc
/ceph
/ceph.conf
scp
/usr
/local
/etc
/ceph
/ceph.conf root@ceph_osd2
:
/etc
/ceph
/ceph.conf
# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
temp dir is
/tmp
/mkcephfs.hz1EcPJjtu
preparing monmap
in
/tmp
/mkcephfs.hz1EcPJjtu
/monmap
/usr
/local
/bin
/monmaptool
--create
--clobber
--add
0
222.
31.
76.
178
:
6789
/tmp
/mkcephfs.hz1EcPJjtu
/monmap
/usr
/local
/bin
/monmaptool
: monmap file
/tmp