바로가기메뉴

본문 바로가기 주메뉴 바로가기

logo

Article Contents

Prev Next
e-Submission

Vol.24 No.4
Citation Share

Improving the Performance of Document Clustering with Distributional Similarities

Journal of the Korean Society for Information Management / Journal of the Korean Society for Information Management, (P)1013-0799; (E)2586-2073
2007, v.24 no.4, pp.267-283
https://doi.org/10.3743/KOSIM.2007.24.4.267

(2007). Improving the Performance of Document Clustering with Distributional Similarities. Journal of the Korean Society for Information Management, 24(4), 267-283, https://doi.org/10.3743/KOSIM.2007.24.4.267
  • Downloaded
  • Viewed
PDF Download

Abstract

In this study, measures of distributional similarity such as KL-divergence are applied to cluster documents instead of traditional cosine measure, which is the most prevalent vector similarity measure for document clustering. Three variations of KL-divergence are investigated; Jansen-Shannon divergence, symmetric skew divergence, and minimum skew divergence. In order to verify the contribution of distributional similarities to document clustering, two experiments are designed and carried out on three test collections. In the first experiment the clustering performances of the three divergence measures are compared to that of cosine measure. The result showed that minimum skew divergence outperformed the other divergence measures as well as cosine measure. In the second experiment second-order distributional similarities are calculated with Pearson correlation coefficient from the first-order similarity matrixes. From the result of the second experiment, second-order distributional similarities were found to improve the overall performance of document clustering. These results suggest that minimum skew divergence must be selected as document vector similarity measure when considering both time and accuracy, and second-order similarity is a good choice for considering clustering accuracy only.

keywords
distributional similarity, divergence, second-order similarity, document clustering, automatic classification, distributional similarity, divergence, second-order similarity, document clustering, automatic classification, 분포유사도, 다이버전스, 2차 유사도, 문헌 클러스터링, 자동분류, distributional similarity, divergence, second-order similarity, document clustering, automatic classification

Reference

1.

정영미. (2005). 정보검색연구. , -.

2.

정영미. (2001). 지식 분류의 자동화를 위한 클러스터링 모형 연구. 18(2), 203-230.

3.

Dagan, Ido. (1999). Similarity-based models of cooccurrence probabilities. 34(1-3), 43-69.

4.

Griffith, A. (1984). Hierarchic agglomerative clustering methods for automatic document classification. , 175 3-11.

5.

Kullback, S. (1951). On information and sufficiency. , -.

6.

Kullback, Solomon. (1968). Information Theory and Statistics. , -.

7.

Lee, Lillian. (1999). Measures of distributional similarity. , 25-32.

8.

Lee, Lillian. (2001). On the effectiveness of the skew divergence for statistical language analysis. , 65-72.

9.

Lin, Dekang. (1998). Automatic retrieval and clustering of similar words. 98, 768-773.

10.

Lin, Jianhua. (1991). Divergence measures based on the Shannon entropy. , 145-151.

11.

Pereira, Fernando. (1993). Distributional clustering of English words. , -.

12.

Theodoridis, S. (2003). Pattern Recognition. , -.

13.

Weeds, J. E.. (2003). Measures and Applications of Lexical Distributional Similarity. , -.

14.

White, H. D. (1981). Author cocitation: a literature measure of intellectual structure. , 163-171.

15.

Griffiths, A. (1986). Using interdocument similarity information in document retrieval systems. , -.

16.

Lee, Lillian. (1999). Distributional similarity models: Clustering vs. nearest neighbors. , 33-40.

17.

Salton, Gerard. (1983). Introduction to Modern Information Retrieval. , -.

Journal of the Korean Society for Information Management

e-Submission OA Policy