ceph分布式存储-集群容量评估.docx

?

?

ceph分布式存储-集群容量评估

?

?

1.环境介绍

1.1软件环境

ceph集群：

mon：ceph-xxx-osd02.ys,ceph-xxx-osd03.ys,ceph-xxx-osd01.ys

osd:36

mds：ceph-xxx-osd04.ys=up:active

ceph版本：

cephversion12.2.2.3(277c813c8cdeb79eba8a11bfe08faf1bf7809f07)luminous(stable)

os系统：

CentOSLinuxrelease7.2.1511(Core)

3.10.0-514.16.1.el7.x86_64

1.2硬件环境

cpu型号：Intel(R)Xeon(R)CPUE5-2630v4@2.20GHz

cpu核数：40

硬盘：hdd

2.测试场景

2.1单个磁盘的能力

下表以测试随机写IOPS的命令为例，说明命令中各种参数的含义。

czt01h3dvw

iz6he9e5o8

2.1.1顺序读(blocksize4M)

//blocksize是4M,30个线程并发，持续时间200s

测试结果：带宽：377MB/s平均IOPS：89运行时长：2848ms耗时：1249ms

$fio-direct=1-iodepth=128-rw=read-ioengine=libaio-bs=4M-size=1G-numjobs=1-runtime=1000-group_reporting-filename=/var/lib/ceph/osd/ceph-3/test0012-name=Read_PPS_Testing

Read_PPS_Testing:(g=0):rw=read,bs=(R)4096KiB-4096KiB,(W)4096KiB-4096KiB,(T)4096KiB-4096KiB,ioengine=libaio,iodepth=128

fio-3.1

Starting1process

Jobs:1(f=1):[R(1)][30.0%][r=232MiB/s,w=0KiB/s][r=58,w=0IOPS][eta00m:07s]

Read_PPS_Testing:(groupid=0,jobs=1):err=0:pid=26652:ThuJun716:34:272018

read:IOPS=89,BW=360MiB/s(377MB/s)(1024MiB/2848msec)

slat(usec):min=623,max=54269,avg=10938.52,stdev=8604.56

clat(msec):min=46,max=2750,avg=1238.56,stdev=741.72

lat(msec):min=47,max=2751,avg=1249.50,stdev=743.03

clatpercentiles(msec):

|1.00th=[48],5.00th=[56],10.00th=[65],20.00th=[342],

|30.00th=[802],40.00th=[1200],50.00th=[1401],60.00th=[1586],

|70.00th=[1787],80.00th=[1972],90.00th=[2140],95.00th=[2165],

|99.00th=[2198],99.50th=[2198],99.90th=[2735],99.95th=[2735],

|99.99th=[2735]

bw(KiB/s):min=155648,max=253952,per=57.11%,avg=210261.33,stdev=50053.95,samples=3

iops:min=38,max=62,avg=51.33,stdev=12.22,samples=3

lat(msec):50=1.95%,100=13.67%,250=3.12%,500=5.08%,750=5.08%

lat(msec):1000=5.86%,2000=46.48%,=2000=18.75%

cpu:usr=0.00%,sys=9.41%,c