Daum, the operator of Korea’s second most popular internet portal, www.daum.net, is about to release a new service built around a detailed photographic map of Korea, which covers virtually the entire country including Seoul and the neighboring Gyeonggi Province areas, the six metropolitan cities and Jeju Island. The Street View service provides more than 4 million 360° views of streets across the nation.
See this page.
Distributed matrix multiplication with HAMA
To mutliply two matrices A and B, We collect the blocks to 'collectionTable' firstly using map/reduce.
Rows are named as c(i, j) with sequential number ((N^2 * i) + ((j * N) + k) to avoid duplicated records. Each row has a two sub matrices of a(i, k) and b(k, j) so that minimized data movement and network cost. Finally, We multiply and sum sequentially.
Don't look down upon this uneducated method. There are a lot of duplicated blocks but, it'll be distributed at the hadoop/hbase level. And, Increased node numbers, there is a linear increase of IO channel. So, We can see an approximately linear increase of speed with increment of node number.
Here is my test result with Hama:
* 4 Intel(R) Xeon(R) CPU 2.33GHz, SATA hard disk, Physical Memory 16,626,844 KB
* Replication factor : 3
* Rests are default
= Matrix-Matrix Multiply of 5,000 by 5,000 dense matrix =
Unfortunately, hbase seems not ready to put large columns yet. So, I tested 5,000 by 5,000.
* 3 node : 602 seconds, 400 blocks
* 6 node : 270 seconds, 400 blocks
If you have more good idea, please share with me!! :)
Rows are named as c(i, j) with sequential number ((N^2 * i) + ((j * N) + k) to avoid duplicated records. Each row has a two sub matrices of a(i, k) and b(k, j) so that minimized data movement and network cost. Finally, We multiply and sum sequentially.
Don't look down upon this uneducated method. There are a lot of duplicated blocks but, it'll be distributed at the hadoop/hbase level. And, Increased node numbers, there is a linear increase of IO channel. So, We can see an approximately linear increase of speed with increment of node number.
Here is my test result with Hama:
* 4 Intel(R) Xeon(R) CPU 2.33GHz, SATA hard disk, Physical Memory 16,626,844 KB
* Replication factor : 3
* Rests are default
= Matrix-Matrix Multiply of 5,000 by 5,000 dense matrix =
Unfortunately, hbase seems not ready to put large columns yet. So, I tested 5,000 by 5,000.
* 3 node : 602 seconds, 400 blocks
* 6 node : 270 seconds, 400 blocks
If you have more good idea, please share with me!! :)
Island caretaker job offer: get paid $150,000 to swim, snorkel
Wish you were here? Tourism Queensland is offering a $150,000, rent-free job on Hamilton Island. Wow!! really cool. See : http://www.tq.com.au
Google: statistical machine translation
Google has been investing significant resources in a multi-year effort to develop its statistical machine translation technology. It seems really interesting to me.
The sense of some word (e.g. singular, plural, synonym, polynym, ..., etc) is not clear, and Idioms usually cannot be translated literally in another language. It means it's important to grasp the meaning from the context. So, "statistical machine translation".
Generally, the alignment methodology used for machine translation. Then, We can simply think about statistical alignment and machine translation for a mapreduce model as describe below :
- Compute p(f/e) by summing the probabilities of all alignments.
You may want to see also : Bayesian spam filtering using Map/Reduce
Anyway, We may see the multi-language/semantic-information search engine soon.
References
* “구글의 장담 ‘언어 장벽 없앤다’”
* 자동번역 검색은 '웹 언어장벽 해체'의 시작
* Wikipedia: Example-based Machine Translation
* Google Translation Center, a New Human Translations Service in the Making
* Google Translation Center: The World’s Largest Translation Memory
* More Languages in Goole Translate
* MS 번역 서비스 '윈도 라이브 번역기'
* 구글-인터넷 바벨탑에 도전한다?
* CAT(Computer Aided Translation) tool 관련
* Statistical Machine Translation Org
The sense of some word (e.g. singular, plural, synonym, polynym, ..., etc) is not clear, and Idioms usually cannot be translated literally in another language. It means it's important to grasp the meaning from the context. So, "statistical machine translation".
Generally, the alignment methodology used for machine translation. Then, We can simply think about statistical alignment and machine translation for a mapreduce model as describe below :
- Compute p(f/e) by summing the probabilities of all alignments.
e: English sentence l : the length of e in words f : French sentence m : the length of f fj : word j in f aj : the position in e that fj is aligned with eaj : the word in e that fj is aligned with p(wf/we) : translation prob. Z : normalization constant |  |
You may want to see also : Bayesian spam filtering using Map/Reduce
Anyway, We may see the multi-language/semantic-information search engine soon.
References
* “구글의 장담 ‘언어 장벽 없앤다’”
* 자동번역 검색은 '웹 언어장벽 해체'의 시작
* Wikipedia: Example-based Machine Translation
* Google Translation Center, a New Human Translations Service in the Making
* Google Translation Center: The World’s Largest Translation Memory
* More Languages in Goole Translate
* MS 번역 서비스 '윈도 라이브 번역기'
* 구글-인터넷 바벨탑에 도전한다?
* CAT(Computer Aided Translation) tool 관련
* Statistical Machine Translation Org
Naver, give google korea a sporting chance
The naver.com main page has been changed, Their main idea is to provide users with a personalized homepage to check news, e-mail, blog postings, entertainment features and other content all in one place.
I'm not surprised; This is somewhat like google personal home page (weather forcast, Gmail preview, and RSS feeds, several other module) and There are already many sites on the web that allow the user to set up a personal, customized homepage.
But, almost netizen of korea circumstances are altogether different from me. They are trained by "Stay tuned to this web-site for the latest news and informations". I think it was a key to its popularity in korea.
And now, They will :
be re-trained, or leave to somewhere.
Anyway, the netizen of korea is starting their new life in a far more wide world web.
I'm not surprised; This is somewhat like google personal home page (weather forcast, Gmail preview, and RSS feeds, several other module) and There are already many sites on the web that allow the user to set up a personal, customized homepage.
But, almost netizen of korea circumstances are altogether different from me. They are trained by "Stay tuned to this web-site for the latest news and informations". I think it was a key to its popularity in korea.
And now, They will :
be re-trained, or leave to somewhere.
Anyway, the netizen of korea is starting their new life in a far more wide world web.
Subscribe to:
Posts (Atom)
-
음성 인공지능 분야에서 스타트업이 생각해볼 수 있는 전략은 아마 다음과 같이 3가지 정도가 있을 것이다: 독자적 Vertical 음성 인공지능 Application 구축 기 음성 플랫폼을 활용한 B2B2C 형태의 비지니스 구축 기 음성 플랫폼...
-
개발자 컨퍼런스같은 것도 방문한게 언제인지 까마득합니다. 코로나로 왠지 교류가 많이 없어졌습니다. 패스트캠퍼스로부터 좋은 기회를 얻어 강연을 하나 오픈하였습니다. 제가 강연에서 주로 다룰 내용은, 인터넷 역사 이래 발전해온 서버 사이드 기술들에 대해 ...
-
패밀리 세단으로 새차 구입은 좀 무리일 것 같아서, 중고로 하나 얻어왔습니다. 중고차라고 티 내는건지 :-) 시거잭에 전원이 안들어오더군요. 요즘 참 세상 좋아졌다고 생각드는게, 유튜브에서 시거잭 전원 불량에 대한 단서를 얻었습니다. 바로 퓨즈가 나가... -
무한 집합의 크기 Cardinality , 즉 원소의 개수를 수학에서는 '농도'라고 말한다. 유한 집합의 크기는 그대로 원소의 개수 이지만, 무한 집합의 경우는 원소의 개수를 낱낱이 셈하는 것은 불가능하기 때문에 '농도'라...