Term Frequency – Inverse Document Frequency (TF-IDF)

Xây dựng thư viện TfidfTransformer như mộ hình UML như hình sau

Input

Đầu vào là một tập (corpus) các văn bản

Ví dụ

This I Can't is the first document.
This document is the second document.
And this is the third one.
Is this the first document?

string[] corpus = {
    "This I Can't is the first document."
    "This document is the second document."
    "And this is the third one."
    "Is this the first document?"
};

B1. Thực hiện thống kê tần suất từ input đầu vào (2.5 điểm)

Tạo ra ma trân thống kê tần suất như sau

	this	I	can’t	Is	the	first	document	second	and	third	one
document 1	1	1	1	1	1	1	1	0	0	0	0
document 2	1	0	0	1	1	0	2	1	0	0	0
document 3	1	0	0	1	1	0	0	0	1	1	1
document 4	1	0	0	1	1	1	1	0	0	0	0

B2. Tính Term Frequency (2.5 điểm)

$$tf(t,d) = \frac{f_{t,d}}{\sum_{t'∈d}{f_{t',d}}}$$

Trong đó f_t,d là số lượng của 1 term trong 1 document

	this	I	can’t	is	the	first	document	second	and	third	one
tf(t, d1)	$$\frac{1}{7}$$	$$\frac{1}{7}$$	$$\frac{1}{7}$$	$$\frac{1}{7}$$	$$\frac{1}{7}$$	$$\frac{1}{7}$$	$$\frac{1}{7}$$	0	0	0	0
tf(t, d2)	$$\frac{1}{6}$$	0	0	$$\frac{1}{6}$$	$$\frac{1}{6}$$	0	$$\frac{2}{6}$$	$$\frac{1}{6}$$	0	0	0
tf(t, d3)	$$\frac{1}{6}$$	0	0	$$\frac{1}{6}$$	$$\frac{1}{6}$$	0	0	0	$$\frac{1}{6}$$	$$\frac{1}{6}$$	$$\frac{1}{6}$$
tf(t, d4)	$$\frac{1}{5}$$	0	0	$$\frac{1}{5}$$	$$\frac{1}{5}$$	$$\frac{1}{5}$$	$$\frac{1}{5}$$	0	0	0	0

B3. Tính Inverse document frequency (2.5 điểm)

$$idf(t,D) = log \frac{N}{|{d∈D:t∈d}|}$$

Trong đó: N là số lượng document trong corpus N = |D|, {d 𝜖 D: t 𝜖 d} là số lượng document chứa term t

	this	I	can’t	is	the	first	document	second	and	third	one
Idf(t,D)	$$log\frac{⁡4}{4}$$	$$log\frac{⁡4}{1}$$	$$log\frac{⁡4}{1}$$	$$log\frac{⁡4}{4}$$	$$log\frac{⁡4}{4}$$	$$log\frac{⁡4}{2}$$	$$log\frac{⁡4}{3}$$	$$log\frac{⁡4}{1}$$	$$log\frac{⁡4}{1}$$	$$log\frac{⁡4}{1}$$	$$log\frac{⁡4}{1}$$

B4. Tính Term frequency–Inverse document frequency (2.5 điểm)

$$Tfidf(t,d, D) = tf(t,d) . idf(t,D)$$

	this	I	can’t	is	the	first	document	second	and	third	one
tfidf(t,d1,D)	$$\frac{1}{7} log\frac{4}{4}$$	$$\frac{1}{7} log\frac{4}{1}$$	$$\frac{1}{7} log\frac{4}{1}$$	$$\frac{1}{7} log\frac{4}{4}$$	$$\frac{1}{7} log\frac{4}{4}$$	$$\frac{1}{7} log\frac{4}{2}$$	$$\frac{1}{7} log\frac{4}{3}$$	0	0	0	0
tfidf(t,d1,D)	$$\frac{1}{6} log\frac{4}{4}$$	0	0	$$\frac{1}{6} log\frac{4}{4}$$	$$\frac{1}{6} log\frac{4}{4}$$	0	$$\frac{2}{6} log \frac{⁡4}{3}$$	$$\frac{1}{6} log\frac{4}{1}$$	0	0	0
tfidf(t,d1,D)	$$\frac{1}{6} log\frac{4}{4}$$	0	0	$$\frac{1}{6} log\frac{4}{4}$$	$$\frac{1}{6} log\frac{4}{4}$$	0	0	0	$$\frac{1}{6} log\frac{4}{1}$$	$$\frac{1}{6} log\frac{4}{1}$$	$$\frac{1}{6}$$
tfidf(t,d1,D)	$$\frac{1}{5} log\frac{4}{4}$$	0	0	$$\frac{1}{5} log\frac{4}{4}$$	$$\frac{1}{5} log\frac{4}{4}$$	$$\frac{1}{5} log\frac{4}{2}$$	$$\frac{1}{5} log\frac{4}{3}$$	0	0	0	0

Link tham khảo

https://en.wikipedia.org/wiki/Tf%E2%80%93idf