Hello,
I have two questions about the result evalution:
1) How the confusion matrix X is extracted from the predictions?
If I assume that X is similar to the cost matrix C, then X will have 1 in the cell at the intersection of the row of predicted class and the column of the actual class, and 0 everywhere else.
For example, if the predicted class is Buy and the actual one is Hold, then:
0 1 0
X = 0 0 0
0 0 0
Is this assumption true?
If yes, then the cost matrix C has no effect on the accuarcy!
That is, ACC would be 1 if the predicted class matches the actual one and 0 otherwise, whatever the values of C are.
2) While I understand that the cost of a correct prediction (e.g. (Buy, Buy)) is positive and high (e.g. 8), I anticipate the cost of a wrong prediction (e.g. (Buy, Sell) or (Sell, Buy)) should be negative or small.
Could you please explain why the cost of (Buy, Sell) or (Sell, Buy) is positive and high, just like a correct prediction?
Thank you
Hello,
The understanding of the notion of a confusion matrix in the evaluation is pretty standard. You may read more about it, e.g. on Wikipedia:
https://en.wikipedia.org/wiki/Confusion_matrix
Regarding the cost matrix C, you may consider it as a cost and reward matrix. If you take a closer look at the formula for computing a value of ACC (see the bottom of the description https://knowledgepit.fedcsis.org/mod/page/view.php?id=1012), you will notice that values from the diagonal of C appear in the numerator and denominator of a fraction, while the other values appear only in the denominator.
For instance, in the present form, the penalty for wrongly predicting 'Buy' when the true class was 'Sell' is the same as the reward for a correct classification of a 'Sell' instance. However, if you predict 'Buy' and the actual class is 'Hold', then the penalty will be only a half as high as in the first case.
The cost matrix was proposed by experts from mBank.
Best regards,
Andrzej Janusz
