Apache Hama - BSP Job Submission Portal

I just made a simple web portal, where you can submit your BSP job and execute it on my test cluster.

http://hamaroid.com/

I spend 1 hour writing code and didn't test enough. :P

Try to upload hama-0.2.0-examples.jar, and put the args "bench 5 5 1".

Enjoy the cool summer with Apache Hama and GSoC 2011

The Apache Software Foundation was accepted for Google Summer of Code 2011. Enjoy the cool summer with Apache Hama and GSoC!

Introduction

Apache Hama is a Google's Pregel-like, a distributed computing framework based on Bulk Synchronous Parallel.


GSoC 2011 Projects:


1. Evaluation of Hama BSP communication protocol performance

The goal of this project HAMA-358 is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers. By this project, students will have learned how to evaluate components in the design phase.

• A list of prerequisite:
  • Understanding of Bulk Synchronous Parallel model
  • Understanding of RPC (remote procedure call)
• Programming skills:
  • Shell scripts
  • 2d plotting programming
• The estimated duration for this project: 12 weeks
• The level of difficulty: High

2. Development of Shortest Path Finding Algorithm

The goal of this project HAMA-359 is development of parallel algorithm for finding a Shortest Path using Hama BSP. By this project, students will have researched the development of a Message-Passing based parallel algorithm and learned Bulk Synchronous Parallel model.

• A list of prerequisite:
  • Understanding of the Hama BSP programming model
  • Understanding of Graph theory
• Programming skills:
  • Java Programming
• The estimated duration for this project: 12 weeks
• The level of difficulty: High

3. Runtime Compression of BSP Messages to Improve the Performance

In this project HAMA-367, we investigate BSP message data compression in the context of large-scale distributed message-passing systems to reduce the communication time of individual messages and to improve the bandwidth of the overall system.

• A list of prerequisite:
  • Understanding of the Bulk Synchronous Parallel and Message-Passing model
• Programming skills:
  • C or Java Programming
• The estimated duration for this project: 12 weeks
• The level of difficulty: High

Apache Hama 0.2.0 Release Candidates

• http://people.apache.org/~edwardyoon/dist/
• http://people.apache.org/~edwardyoon/documents/

OSS 릴리즈 관리에 GnuPG (그누피지) 사용하기

GnuPG는 GNU Privacy Guard의 약자로 문서나 파일을 암호화할 때 사용되는 소프트웨어이다. 자세한 내용은 공식 웹사이트를 참조:

- http://www.gnupg.org/

일반적으로 오픈소스는 (Apache 프로젝트도 역시) 릴리즈할 때 release tarball 에 GnuPG (그누피지) 를 사용해서 사인을 하는데, 나는 어제 처음 써본것이라 3시간을 이것때문에 삽질했다. :/

어쨌거나 암호학에 생소하면 당췌 문서를 이해하기 쉽지않고 한글 문서는 찾아볼 수도 없어서 이참에 정리를 좀 해보면,

1. 제일 먼저 GPG 키가 없으면 다음과 같은 명령어로 하나 생성해야된다:

[edwardyoon@minotaur:~/public_html/dist]$ gpg --gen-key

2. 질문에 대해서는 대부분 default 로 넘어가면 되고 완료가 되면 다음과 같은 명령어로 ASCII 공개키를 파일로 생성할 수 있다:

[edwardyoon@minotaur:~/public_html/dist]$ gpg --armor --output KEYS --export Edward J. Yoon
Warning: using insecure memory!
[edwardyoon@minotaur:~/public_html/dist]$ cat KEYS 
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.16 (FreeBSD)

mQMuBE1uPgIRCADVN8LNKn8God0EM+PuJ4Y7+YV6FY3R/IFa/W5XbHZpDBWT161S
tZe+w4e4hl+cK5eHDkjvnj+Xx8g9u/ohxF31SOGDrbNBdDpOMJEG6F9vu3TdNdVb
Kz8yLHcjp5v/kQNngyytn/4rVXtYjFCta8Ks5DAHUZCfESoRf8HLvk5fHr4AwXng
vG85xHqQ3axW3SGO4ZDnsaLrofaHazemjR+wxmY7XK6No/aV5gyfTpKgq1sDKzge
NSNxEs99Om1Xfrap1JgWuiXFgTo0mc6DW3j4VCk3DcGqJfKUHEs1EVe4uqC1y/EQ
uA/vuQeMmr6LGV5TTtWplQ/lV96oGosF3LuzAQCiq9fh9a9dgnMVw08Vg4XqVjO+
/Vj72xmdN4ebqUkH0Qf/UBp9xIA2dZ1m62e/Bc9TIlK5oWkYvIjaL++hITD5C7Ca
2qJMXFUo1obI7x726T2SCofluM0ZdU4TUvDn7HtPPmxN/fM8fehjGvYncSbxuUi9
34tSt6NBfbX2ffvn2anD6zPH9IM7RdZEeNI2Kkky87SrnAXr+JLqo2apsokL0Ph6
FxMXR+XmLJfAsvbcVZT23qZ7Qd3ARPR64b9RP+v4AbTe4OuQdPIMtK0Hvx3q3cf/
BfDqpJDvBS57bEhrKJac6yltjuRnoWjnHQ4CL/83rscB+rHWANvt40m8CF6F/bSr
2NeK99q41kRR8E4MtzjjPCdbEUWu4U2F/QvetPDmmgf/QC+l0C7Dt6xvpFpqaDDg
gvz2pbpsJ06Lr0/NICHVRkMBUDl5dFZUSFlh+UxGyzLwKNV3UH5Tiuia6reNM6Ao
Um8mMY2T90up561C0vrpPrkqh6UuyAk19ekRxhoS8vVb51uGWIozRW4MU+joHfY3
YPDD+4D3hx1JIRzATjNV1+Du2DIcjnXyQpiDkMStp9twhiZMqUslZl7lZ0Eud68E
FRqRYsNkPZ5Uzp+52Q3gqOWhoymGGSfgQbaSB3uB4heKWz68M9bU4atQzWFr++TS
+ZFmh9B3ThkpKAcBsGfWqIUVz9ufAVUYcZZFr2/5KDodT7+xZmT/UqZVmmQrwhii
z7QmRWR3YXJkIEouIFlvb24gPGVkd2FyZHlvb25AYXBhY2hlLm9yZz6IeQQTEQgA
IgUCTW4+AgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQsSWxAzxQ6Ne0
PgD/dmwt20c9EHK/3dhRupakg4C5Qwow/8x01Qy2sBL5hIoA93wfgEbVPbObaBtM
7uE+7TBXtFVaAlcmjSm6T9ZnYBm5Ag0ETW4+AhAIALsGf2wj+l9gdZJsxtapCN7p
k5PfdyDOYBWN5sWIm2aTLgxQd+S1sy8FkABxGR1jzaJ5P6+/nw9KiRGXCcXHyjKz
iR3ov4/Dflatkkq14SzpeROYWz8kUG7bARwU9ZQrZjtbp2MHoIH4clFqtv2+8iCS
eV8OJzkSINo0Kwmw5ArmFF2Ps5cP6ybWqDYuJs8S7zGLe/WuLRS4bmcsbt+gPZJ9
2WppvvX2g9C2u+8PtZDIPf0nrh5USQHuAUsQ/HBNvQ10yytMcY2J9L7wgof38rjj
wdijv1pepucMVDErnTBBe2CB1ABM10k45iI52Se0RHOD4LPq5r6FilXyUoy/qtcA
AwcH/jO3AZsShIb5aZ4PH8SQqeX0uphkrWcYy1UKJ3ioYPuG4qlvxnAbyuZcBMcA
Jh+es06ScioBaE6NSA6y81on9pwMJdb9Y63gUB4CjIWzgo4N1+F4bLKMV+wBqocK
S1NpetSHmxik28RmzuVUVloNZGjThcwLAnkbXEn7jm0bUO0xDIYsA5h/4sU2uDqH
Q5lHMwotlYyhTYi43UQX5RLCmBw5neA2UXYNTszn//+5aCmsw9Bl/FE9D2hPUGsh
Q1JQBxGFoQU2y5cEHN8HPN0Fnc0usREpVh49wJQrP5PI+t5YYl1N6ZrNcSfW4ISl
AvH6LdBH+pKkdpznACs5mSulke2IYQQYEQgACQUCTW4+AgIbDAAKCRCxJbEDPFDo
12jUAP4hKXaoSXSkicLtepi9mMl2oy5iYsOjJWbb0IimU5PiQQD+K8dcsL5PCugV
b5uwU0bOa3OtKT32p1QCr6hXsU64b+U=
=qgDs
-----END PGP PUBLIC KEY BLOCK-----

tarball 파일에 사인은 다음과 같은 명령어로 할 수 있다:

[edwardyoon@minotaur:~/public_html/dist/0.2.0]$ gpg --armor --output hama-0.2.0.tar.gz.asc --detach-sig hama-0.2.0.tar.gz 

이렇게 사인을 해두면, 다운로더는 아래와 같이 "Edward J. Yoon <edwardyoon@apache.org>"이 사인한 공식적인 release tarball인지 아닌지를 확인해볼 수 있게 된다:

[edwardyoon@minotaur:~/public_html/dist/0.2.0]$ gpg --verify hama-0.2.0.tar.gz.asc hama-0.2.0.tar.gz
Warning: using insecure memory!
gpg: Signature made Wed Mar  2 13:54:36 2011 UTC using DSA key ID 3C50E8D7
gpg: Good signature from "Edward J. Yoon <edwardyoon@apache.org>"

외국 메일링 리스트를 보면, 이메일 서명에도 GPG를 사용하는 OSS 개발자들이 많더라~

Monitoring and Mining Network Traffic in Clouds

Check out this SlideShare Presentation:

Monitoring and mining network traffic in clouds

View more presentations from Edward J. Yoon.