Back close

Detection of Duplicates in Quora and Twitter Corpus

Publication Type : Conference Paper

Publisher : Advances in Intelligent Systems and Computing

Source : Advances in Intelligent Systems and Computing, Springer Verlag, Volume 750, p.519-528 (2019)

Url :

ISBN : 9789811318818

Keywords : Artificial intelligence, Big data, Cloud computing, De duplications, It focus, Learning algorithms, Learning systems, Linguistics, Natural language processing systems, Natural languages, Paraphrase, Paraphrase corpus, Quora, Social networking (online), Supervised learning, Supervised machine learning, Twitter

Campus : Coimbatore

School : School of Engineering

Center : Computational Engineering and Networking

Department : Electronics and Communication

Year : 2019

Abstract : Detection of duplicate sentences from a corpus containing a pair of sentences deals with identifying whether two sentences in the pair convey the same meaning or not. This detection of duplicates helps in deduplication, a process in which duplicates are removed. Traditional natural language processing techniques are less accurate in identifying similarity between sentences, such similar sentences can also be referred as paraphrases. Using Quora and Twitter paraphrase corpus, we explored various approaches including several machine learning algorithms to obtain a liable approach that can identify the duplicate sentences given a pair of sentences. This paper discusses the performance of six supervised machine learning algorithms in two different paraphrase corpus, and it focuses on analyzing how accurately the algorithms classify sentences present in the corpus as duplicates and non-duplicates. © 2019, Springer Nature Singapore Pte Ltd.

Cite this Research Publication : S. Viswanathan, Damodaran, N., Simon, A., George, A., M. Kumar, A., and Dr. Soman K. P., “Detection of Duplicates in Quora and Twitter Corpus”, in Advances in Intelligent Systems and Computing, 2019, vol. 750, pp. 519-528.

Admissions Apply Now