Measures of Skill

Performance evaluation of meteorological algorithms is commonly done using the contingency table, which is a 2$\times$2 table defined as:

$$\left\{ \begin{array}{rr} a & b\\ c & d \end{array}\right\}$$ (2)

where $d$ is the number of correctly detected (or forecast) events (often referred to as ``hits''), $b$ the number of false alarms and $c$ the number of events not detected (``misses''). The number of correctly classified non-events, $a$, is not clearly defined in the analysis presented here. We use, as our performance measure, the critical success index (CSI) which is defined on the above table as  [4,19]:

$$CSI = \frac{d}{b+c+d}$$ (3)

The CSI is zero for an algorithm that possesses no skill and is equal to one for an algorithm that performs flawlessly. At opposite ends of the performance spectrum of meteorological algorithms are the original WSR-88D tornadic vortex signature detection algorithm, with a CSI of 0.02 [14], and the enhanced WSR-88D hail detection algorithm, with a CSI of 0.49 [20].

It is apparent that Equation 3 hinges on a binary classification - either the event occurs or it does not. Yet, marginal events are often observed. There are three ways of dealing with this when designing algorithms:

• Count a marginal event that has been detected as a hit but do not count a missed marginal event as a miss. This approach overstates the performance of an algorithm.
• A slightly better approach is to count marginal events neither as hits nor as misses. However, marginal events are different from non-events in that if detected, they are not counted as false alarms.
• If we accept that an improvement of the performance of the fuzzy logic scheme can be had when its output matches the indecision inherent in that of a human observer i.e. when a marginal BWER is identified as a marginal BWER and not as a BWER, then we can use several degrees of truth for both the real values and the algorithm output.

We decided to use a three-level classification, i.e., regions are BWERs, marginal BWERs or non-BWERs. Scoring the skill of such a classifier can be done based on the $3\times3$ table:

$$\left\{ \begin{array}{rrr} a & b & c\\ d & e & f\\ g & h & i\\ \end{array}\right\}$$ (4)

in which rows represent the truth values 0, 0.5 and 1. The columns represent the output of the scheme. Thus, $h$ would represent BWERs (true value = 1) that were misidentified as marginal BWERs (output = 0.5) by the scheme.

We can then describe a modified CSI based on the $3\times3$ table as:

$$CSI_{mod} = \frac{i+e}{c+g+\frac{b+d+h+f}{2}+i+e}$$ (5)

where the ``misses'' and ``false alarms'' are weighted by the degree to which they are truly ``misses'' and ``false alarms''. The CSI of Equation 3 (where marginal BWERs are counted neither when detected nor when missed) is given in terms of the $3\times3$ table by:

$$CSI = \frac{i}{c+g+i}$$ (6)

The final results in this paper are presented using both the traditional measure in Equation 6 and the truer measure given by Equation 5.