基于TFIDF的社区问答系统问句相似度改进算法

Improved TFIDF-Based Question Similarity Algorithm for the Community Interlocution Systems

摘要: 针对社区问答系统问句相似度计算问题，提出了一种改进的TFIDF算法.按照用户的查询意图对问句进行分类，根据特征词在类别中的分布对权值进行调整；将问句的主题词归入特征项进行TFIDF计算.实验结果表明，本文改进的TFIDF算法的<i<P</i<@3比传统的TFIDF算法提高了7.66%，比TFIDF-IG算法提高了5.31%，而且<i<P</i<@5和<i<P</i<@10也有不同程度的提高，与传统TFIDF算法和参考改进算法相比，该算法明显提高了检索性能.

Abstract: To calculate the question similarity in the community interlocution systems, an improved TFIDF algorithm was proposed in this paper. Firstly, the questions were divided into different categories according to the users' retrieval intention, and the weight of every feature word was adjusted based on the distribution in the categories. And then, the topic words were adopted in the feature words for TFIDF algorithm. The experimental results show that, compared with the traditional TFIDF, the <i<P</i<@3 increases 7.66%. Compared with TFIDF-IG, the <i<P</i<@3 increases 5.31%. And different improvements can be obtained in <i<P</i<@5 and <i<P</i<@10. The new algorithm shows better search performance.