大数据

YCSB测试hbase-0.96.0

本文介绍了使用YCSB工具对hbase-0.96.0进行性能测试的过程。测试包括了read和update两种事务,详细展示了各项指标,如操作数、平均延迟、95th和99th百分位延迟等。测试结果显示,UPDATE操作具有较低的平均延迟,而READ操作的延迟较高,尤其是在99th百分位上。

1.YCSB简介

YCSB的全称是Yahoo! Cloud Serving Benchmark,这是一个用于测试cloud serving/NoSQL/Key-Value Store的benchmark。 论文发表在SoCC’10, 代码开源。有很多出色的工作(包括SILT@SOSP’11,HyperDex@SIGCOMM’12)使用YCSB进行测试。为了了解hbase-0.96.0的性能,因此学习一下如何使用YCSB。本文主要论述利用ycsb测试hbase-0.96.0。

2、YCSB测试hbase-0.96.0

1)下载ycsb源代码

git clone  https://github.com/brianfrankcooper/YCSB.git

删除原来hbase/pom.xml中的hbase和hadoop依赖项,改为如下:

<dependency>
          <groupId>org.apache.hbase</groupId>
          <artifactId>hbase-it</artifactId>
          <version>0.96.0-hadoop2</version>
</dependency>
<dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.2.0</version>
</dependency>

切换到YCSB目录,并使用mvn编译打包源代码(需要安装maven),如下:

cd YCSB
mvn -DskipTests package

编译结束后,将集群的配置信息写入hbase/src/main/conf/hbase-site.xml文件。例如:

<configuration>
  <property>
      <name>hbase.zookeeper.quorum</name>
      <value>l-hbase1.data.cn8,l-hbase2.data.cn8,l-hbase3.data.cn8</value>
  </property>
</configuration>

务必将hbase.zookeeper.quorum属性配置好,以便 YCSB使用它找到集群的入口。

在运行工作负载之前,必须创建YCSB写入的HBase表。可以在shell中创建表(ycsb默认测试的表名就是usertable):

hbase(main):002:0> create 'usertable ','cf'
0 row(s) in 1.1940 seconds

创建好之后,准备开始测试集群。首先是加载数据,执行如下命令:

bin/ycsb load hbase -P workloads/workloada -p columnfamily=cf

执行结果如下:

YCSB Client 0.1

Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=cf -load

2014-01-14 14:43:28,214 WARN [Thread-2] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

2014-01-14 14:43:28,396 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:host.name=aaron-ubuntu

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.version=1.6.0_37

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.vendor=Sun Microsystems Inc.

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.home=/usr/lib/jvm/java/jdk1.6.0_37/jre

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.class.path=/home/aaron/work/YCSB-master/jdbc/src/main/conf:/home/aaron/work/YCSB-master/gemfire/src/main/conf:/home/aaron/work/YCSB-master/core/target/core-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/target/hbase-binding-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/target/archive-tmp/hbase-binding-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/src/main/conf:/home/aaron/work/YCSB-master/nosqldb/src/main/conf:/home/aaron/work/YCSB-master/dynamodb/conf:/home/aaron/work/YCSB-master/voldemort/src/main/conf:/home/aaron/work/YCSB-master/infinispan/src/main/conf

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=/usr/lib/jvm/java/jdk1.6.0_37/jre/lib/amd64/server:/usr/lib/jvm/java/jdk1.6.0_37/jre/lib/amd64:/usr/lib/jvm/java/jdk1.6.0_37/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2014-01-14 14:43:28,397 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=/tmp

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=Linux

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=amd64

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=3.5.0-45-generic

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=aaron

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/aaron

2014-01-14 14:43:28,398 INFO [Thread-2] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/home/aaron/work/YCSB-master

2014-01-14 14:43:28,402 INFO [Thread-2] zookeeper.ZooKeeper (ZooKeeper.java:(438)) - Initiating client connection, connectString=l-hbase2.data.cn8:2181,l-hbase1.data.cn8:2181,l-hbase3.data.cn8:2181 sessionTimeout=90000 watcher=hconnection-0x3bc0f2e5

2014-01-14 14:43:28,429 INFO [Thread-2] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:(120)) - Process identifier=hconnection-0x3bc0f2e5 connecting to ZooKeeper ensemble=l-hbase2.data.cn8:2181,l-hbase1.data.cn8:2181,l-hbase3.data.cn8:2181

2014-01-14 14:43:28,434 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(966)) - Opening socket connection to server l-hbase3.data.cn8/192.168.44.7:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)

2014-01-14 14:43:28,440 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(849)) - Socket connection established to l-hbase3.data.cn8/192.168.44.7:2181, initiating session

2014-01-14 14:43:28,447 WARN [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxnSocket (ClientCnxnSocket.java:readConnectResult(139)) - Connected to an old server; r-o mode will be unavailable

2014-01-14 14:43:28,448 INFO [Thread-2-SendThread(l-hbase3.data.cn8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1207)) - Session establishment complete on server l-hbase3.data.cn8/192.168.44.7:2181, sessionid = 0x34351eb5ea30c49, negotiated timeout = 40000

[OVERALL], RunTime(ms), 1777.0

[OVERALL], Throughput(ops/sec), 562.7462014631401

[UPDATE], Operations, 1

[UPDATE], AverageLatency(us), 520191.0

[UPDATE], MinLatency(us), 520191

[UPDATE], MaxLatency(us), 520191

[UPDATE], 95thPercentileLatency(ms), 520

[UPDATE], 99thPercentileLatency(ms), 520

[UPDATE], 0, 0

[UPDATE], 1, 0

[UPDATE], 2, 0

[UPDATE], 3, 0

[UPDATE], 4, 0

[UPDATE], 5, 0

[UPDATE], 6, 0

[UPDATE], 7, 0

[UPDATE], 8, 0

[UPDATE], 9, 0

[UPDATE], 10, 0

[UPDATE], 11, 0

[UPDATE], 12, 0

[UPDATE], 13, 0

…

[CLEANUP], 998, 0

[CLEANUP], 999, 0

[CLEANUP], >1000, 0

java -cp /home/aaron/work/YCSB-master/jdbc/src/main/conf:/home/aaron/work/YCSB-master/gemfire/src/main/conf:/home/aaron/work/YCSB-master/core/target/core-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/target/hbase-binding-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/target/archive-tmp/hbase-binding-0.1.4.jar:/home/aaron/work/YCSB-master/hbase/src/main/conf:/home/aaron/work/YCSB-master/nosqldb/src/main/conf:/home/aaron/work/YCSB-master/dynamodb/conf:/home/aaron/work/YCSB-master/voldemort/src/main/conf:/home/aaron/work/YCSB-master/infinispan/src/main/conf com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=cf -load

其中,workloada 为测试的配置文件,详细说明:https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties

然后是执行事务,主要有 read 和update,执行命令如下:

bin/ycsb run hbase -P workloads/workloada -p columnfamily=cf

执行结果如下:

[OVERALL], RunTime (ms), 3060.0

[OVERALL], Throughput (ops/sec), 326.797385620915

[UPDATE], Operations, 475

[UPDATE], AverageLatency(us), 464.33894736842103

[UPDATE], MinLatency(us), 70

[UPDATE], MaxLatency(us), 116412

[UPDATE], 95thPercentileLatency(ms), 0

[UPDATE], 99thPercentileLatency(ms), 0

[UPDATE], Return=0, 474

[UPDATE], 0, 471

[UPDATE], 1, 0

[UPDATE], 2, 2

[UPDATE], 3, 0

[UPDATE], 4, 1

…

[UPDATE], 998, 0

[UPDATE], 999, 0

[UPDATE], >1000, 0

[READ], Operations, 526

[READ], AverageLatency(us), 5223.148288973384

[READ], MinLatency(us), 1439

[READ], MaxLatency(us), 1099990

[READ], 95thPercentileLatency(ms), 4

[READ], 99thPercentileLatency(ms), 5

[READ], Return=0, 526

[READ], 0, 0

[READ], 1, 14

[READ], 2, 232

[READ], 3, 223

[READ], 4, 47

[READ], 5, 6

[READ], 6, 1

[READ], 7, 0

[READ], 8, 1

…

本文首发于微信公众号。原文:查看原文
微信搜一搜 AI拆解猿