July 26, 2012

Some Benchmarks of Hadoop and Hama on Oracle's BDA

The I/O performance of HDFS with TestDFSIO.
% hadoop jar hadoop-test-0.20.2-cdh3u3b.jar TestDFSIO -write -nrFiles 10 -fileSize 1000

----- TestDFSIO ----- : write
           Date & time: Thu Jul 26 18:50:11 PDT 2012
       Number of files: 10
Total MBytes processed: 10000.0
     Throughput mb/sec: 163.4360801490537
Average IO rate mb/sec: 167.77435302734375
 IO rate std deviation: 25.658150459575825
    Test exec time sec: 19.329

hadoop jar hadoop-test-0.20.2-cdh3u3b.jar TestDFSIO -read -nrFiles 10 -fileSize 1000

----- TestDFSIO ----- : read
           Date & time: Thu Jul 26 19:22:14 PDT 2012
       Number of files: 10
Total MBytes processed: 10000.0
     Throughput mb/sec: 374.6721618583739
Average IO rate mb/sec: 375.14581298828125
 IO rate std deviation: 13.625353109608241
    Test exec time sec: 17.311
The communication performance of Apache Hama with Bench tool.
% hama jar hama-examples-0.5.0.jar bench 16 100000 32
...

12/07/26 21:12:16 INFO bsp.BSPJobClient: Current supersteps number: 30
12/07/26 21:12:19 INFO bsp.BSPJobClient: Current supersteps number: 31
12/07/26 21:12:22 INFO bsp.BSPJobClient: Current supersteps number: 32
12/07/26 21:12:25 INFO bsp.BSPJobClient: The total number of supersteps: 32
12/07/26 21:12:25 INFO bsp.BSPJobClient: Counters: 8
12/07/26 21:12:25 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
12/07/26 21:12:25 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=162
12/07/26 21:12:25 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/07/26 21:12:25 INFO bsp.BSPJobClient:     SUPERSTEPS=32
12/07/26 21:12:25 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=5184
12/07/26 21:12:25 INFO bsp.BSPJobClient:     MESSAGE_BYTES_TRANSFERED=10404951552
12/07/26 21:12:25 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=10300386
12/07/26 21:12:25 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=1036800000
12/07/26 21:12:25 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=518400000
12/07/26 21:12:25 INFO bsp.BSPJobClient:     MESSAGE_BYTES_RECEIVED=10404951552
Job Finished in 93.33 seconds

1 comment: