Report Card for a Classifier

Story

The support team trained a bot that sorts incoming tickets into classes such as billing, bug and feature. Before it goes live, you must grade it: for a log of real tickets you know both the actual class and the class the bot predicted.

Task

Read the log and print, for every class that appears (as an actual or a predicted label), its precision, recall and F1, then the macro F1 and the accuracy.

For a class c:

• TP = tickets that are actually c and predicted c
• precision = TP / (tickets predicted c)
• recall = TP / (tickets that are actually c)
• F1 = 2 · precision · recall / (precision + recall)
• If a denominator is 0, that value is defined as 0 (a class that is never predicted has precision 0; never present has recall 0; F1 is 0 when precision + recall is 0)

Then macro F1 = the plain average of the F1 of all classes, and accuracy = correct predictions / n.

Input

• Line 1: n (1 ≤ n ≤ 3000)
• Then n lines actual predicted. Labels are lowercase letters, digits or underscores, at most 12 characters, without spaces.

Output

One line per class, classes sorted by plain character code (like Python's sorted): <class> <precision> <recall> <f1>. Then the lines macro_f1 <value> and accuracy <value>. Print at least 4 decimals: every number within 0.001 of the exact value is accepted.

Example

12
bug bug
bug billing
bug bug
bug bug
billing billing
billing billing
billing bug
feature feature
feature bug
bug bug
billing billing
feature billing

Output:

billing 0.6 0.75 0.6667
bug 0.6667 0.8 0.7273
feature 1.0 0.3333 0.5
macro_f1 0.6313
accuracy 0.6667

For feature: the bot predicted it once and was right (precision 1), but only 1 of the 3 real feature tickets was found (recall 0.3333).

12
bug bug
bug billing
bug bug
bug bug
billing billing
billing billing
billing bug
feature feature
feature bug
bug bug
billing billing
feature billing