Edward J. Yoon's Blog: 01/2010

FW: Apache Hama in academic paper

HAMA: An Efficient Matrix Computation with the MapReduce Framework

Sangwon Seoyz, Edward J. Yoon, Jae-Hong Kimy, Seongwook Jiny, Jin-Soo Kimx and Seungryoul Maengy
y Computer Science Division, Korea Advanced Institute of Science and Technology (KAIST)
z Computer Science Division, Berlin University of Technology (TU Berlin)
x School of Information and Communication, Sungkyunkwan University, South Korea
User Service Development Center, NHN Corp., South Korea
fswseo, jaehong, swjin, maengg@calab.kaist.ac.kr, edwardyoon@apache.org, jinsookim@skku.edu

Abstract

Various scientific computations have become so complex, and thus computation tools play an important role. In this paper, we explore the state-of-the-art framework providing high-level matrix computation primitives with MapReduce through the case study approach, and demonstrate these primitives with different computation engines to show the performance and scalability. We believe the opportunity for using MapReduce in scientific computation is even more promising than the success to date in the parallel systems literature.

It is sooner than I'd planned, but nice start. I hope that this project will continue in the future. :)

comScore Reports Global Search Market Growth of 46 Percent in 2009

Google Sites Accounts for Two-Thirds of 131 Billion Searches Conducted Worldwide in December while Introduction of Bing Helps Microsoft Post Significant Gains During the Year

Reston, VA, January 22, 2010 – comScore, Inc. (NASDAQ: SCOR), a leader in measuring the digital world, today released a study on growth in the global search market in 2009. The study revealed that the U.S. remains the largest search market worldwide, while Google Sites retains a commanding position in the global search market.

“The global search market continues to grow at an extraordinary rate, with both highly developed and emerging markets contributing to the strong growth worldwide,” said Jack Flanagan, comScore executive vice president. “Search is clearly becoming a more ubiquitous behavior among Internet users that drives navigation not only directly from search engines but also within sites and across networks. If you equate the advancement of search with the ability of humans to cultivate information, then the world is rapidly becoming a more knowledgeable ecosystem.”

Top Search Markets Worldwide

The total worldwide search market boasted more than 131 billion searches conducted by people age 15 or older from home and work locations in December 2009, representing a 46-percent increase in the past year. This number represents more than 4 billion searches per day, 175 million per hour, and 29 million per minute. The U.S. represented the largest individual search market in the world with 22.7 billion searches, or approximately 17 percent of searches conducted globally. China ranked second with 13.3 billion searches, followed by Japan with 9.2 billion and the U.K. with 6.2 billion. Among the top ten global search markets, Russia posted the highest gains in 2009, growing 92 percent to 3.3 billion, followed by France (up 61 percent to 5.4 billion) and Brazil (up 53 percent to 3.8 billion).

Top 10 Countries by Number of Searches Conducted* December 2009 vs. December 2008 Total Worldwide, Age 15+ - Home & Work Locations Source: comScore qSearch
	Searches (MM)
	Dec-2008	Dec-2009	Percent Change
Worldwide	89,708	131,354	46%
United States	18,688	22,741	22%
China	11,778	13,278	13%
Japan	6,213	9,170	48%
United Kingdom	4,623	6,245	35%
Germany	4,079	5,609	38%
France	3,362	5,425	61%
South Korea	2,796	4,039	44%
Brazil	2,454	3,763	53%
Canada	2,900	3,710	28%
Russian Federation	1,735	3,333	92%

*Searches based on “expanded search” definition, which includes searches at the top properties where search activity is observed, not only the core search engines.

Top Search Properties Worldwide

Google Sites ranked as the top search property worldwide with 87.8 billion searches in December, or 66.8 percent of the global search market. Google Sites achieved a 58-percent increase in search query volume over the past year. Yahoo! Sites ranked second globally with 9.4 billion searches (up 13 percent), followed by Chinese search engine Baidu with 8.5 billion searches (up 7 percent). Microsoft Sites saw the greatest gains among the top five properties, growing 70 percent to 4.1 billion searches, on the strength of its successful introduction of new search engine Bing. Russian search engine Yandex also achieved considerable gains, growing 91 percent to 1.9 billion searches.

Top 10 Search Properties by Searches Conducted December 2009 vs. December 2008 Total Worldwide, Age 15+ - Home & Work Locations Source: comScore qSearch
	Searches (MM)
	Dec-2008	Dec-2009	Percent Change
Worldwide	89,708	131,354	46%
Google Sites	55,638	87,809	58%
Yahoo! Sites	8,389	9,444	13%
Baidu.com Inc.	7,963	8,534	7%
Microsoft Sites	2,403	4,094	70%
eBay	1,327	2,102	58%
NHN Corporation	1,892	2,069	9%
Yandex	992	1,892	91%
Facebook.com	1,023	1,572	54%
Ask Network	1,053	1,507	43%
Alibaba.com Corporation	1,118	1,102	-1%

*Searches based on “expanded search” definition, which includes searches at the top properties where search activity is observed, not only the core search engines.

Responding to JSON in your server-side JSP script

I wanted to implement some open API for JSON clients. It took me an two days to finish it, Javascript&JSP gives me a pain in the neck. :(

Below is the example code to responding to JSON in a server-side JSP script. It shows how to set up the JSON text, HTTP header and jsoncallback. I hope you don't waste time.

<%@ page language="java" import="net.sf.json.JSONObject" %>
<%@page import="java.util.*" %>
<%
    response.setContentType("application/json");
    response.setCharacterEncoding("UTF-8");

    JSONObject object=new JSONObject();
    object.put("key", "value");

    String jsoncallback = request.getParameter("jsoncallback");
    response.getWriter().println(jsoncallback + "(" + object.toString()+");");
%>

CentOS - install rpmforge yum repo

Install DAG GPG key

wget http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt     # download DAG GPG key
rpm --import RPM-GPG-KEY.dag.txt                                # import the key
rm RPM-GPG-KEY.dag.txt                                          # clean up after ourselves

Install yum-priorities

yum install yum-priorities

Download and install package (i386 - 32bit)

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
rpm -Uhv rpmforge-release-0.3.6-1.el5.rf.i386.rpm           # install the rpmforge yum repo
rm rpmforge-release-0.3.6-1.el5.rf.i386.rpm                 # clean up after ourselves

Download and install package (x86_64 - 64bit)

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
rpm -Uhv rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm         # install the rpmforge yum repo
rm rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm               # clean up after ourselves

A short unique string identifier for shorten URL

How to shorten URLs? As far as you know, there is a lot of Short URL Redirection Services, such as "bit.ly", which used to convert a long url to some short url.

Its Mechanism seems simple. The sequential key for long URL is enough.
For example, the "Wa0e" is the key of "http://bit.ly/Wa0e"
for "http://www.beachbody.com/product/fitness_programs/p90x.do?code=P90XDOTCOM".
The mod_rewrite module could be used to remove file extension and parameters (e.g., short.php?key=Wa0e)

Key looks like a random string, but I guess it's just a sequential key. Because, that way is simple, and same with total number of random combination in conclusion.

How to generate sequential key? Below is my example code.

INDEX.length is 62. So, (62^4 -1) URLs could stored in the combination of four ciphers.

private static String[] INDEX = new String[] { "0", "1", "2", "3", "4", "5",
      "6", "7", "8", "9", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j",
      "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x",
      "y", "z", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L",
      "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z" };

  private static String getNextURL(String in) {
    if (in == null) {
      return INDEX[0];
    }

    char[] result = new char[in.length()];
    boolean rounded = false;

    for (int i = in.length() - 1; i > -1; i--) {
      String subStr = Character.toString(in.charAt(i));
      try {
        if (!rounded) {
          result[i] = getNext(subStr).charAt(0);
          rounded = true;
        } else {
          result[i] = in.charAt(i);
        }
      } catch (ArrayIndexOutOfBoundsException e) {
        result[i] = INDEX[0].charAt(0);
      }
    }

    return new String(result);
  }

  private static String getNext(String subStr) {
    if (subStr.equals("Z")) {
      throw new ArrayIndexOutOfBoundsException();
    }

    int x = 0;
    for (int i = 0; i < INDEX.length; i++) {
      if (INDEX[i].equals(subStr))
        break;

      x++;
    }

    return INDEX[x + 1];
  }

Get year from java.util.Date

import java.text.SimpleDateFormat;
import java.util.Date;

public class Test {
  public static void main(String args[]) throws Exception {
    Date date = new Date();
    SimpleDateFormat simpleDateformat = new SimpleDateFormat("yyyy");
    System.out.println("Year: " + simpleDateformat.format(date));
  }
}

Linux Tip, XawTV and Audio driver problem

I recently upgraded my computer by replacing the motherboard, CPU, memory and video card.

Motherboard: GA-P43-ES3G
CPU: Intel(R) Core(TM)2 Quad CPU Q8300 @ 2.50GHz
Audio: Realtek ALC 888
Memory: 2 GB
Video card: NVIDIA GeForce 9600 GT
TV card: VAST bt878
OS: Fedora 12

Damn fast!!. BTW, I just wanted to see TV but It was really hard to solve. :( If you have same problem with Realtek ALC 888 and VAST bt878, below will be helpful for you!

1) XawTV - can't load any font

$ xawtv
....
WARNING: No DGA support available for this display.
/dev/video0 [v4l2]: no overlay support
v4l-conf had some trouble, trying to continue anyway
Warning: Cannot convert string "7x13bold" to type FontStruct
Warning: Missing charsets in String to FontSet conversion
Warning: Missing charsets in String to FontSet conversion
Warning: Missing charsets in String to FontSet conversion
Oops: can't load any font

I installed zvbi-fonts to solve this issue. but didn't solved. So, I tried to install all fonts as below:

$ yum install xorg-x11-fonts**

It'll work well.

2) MIC/Line-In doesn't work

You need to remove the PulseAudio. And then using ALSA mixer, you can enabling the line-in channel.

$ yum install alsa-**
$ yum remove pulseaudio