Roebber, P. J., S. L. Bruening, D. M. Schultz, and J. V. Cortinas Jr., 2003: Improving snowfall forecasting by diagnosing snow density. Wea. Forecasting, 18, 264-287. [AMS] [PDF] [HTML]
NOTE: THIS IS AN OLDER VERSION OF THIS MANUSCRIPT, WHICH IS NOW FORMALLY PUBLISHED BY THE AMS. SEE THE LINKS ABOVE FOR THE OFFICIAL VERSION.
Improving Snowfall Forecasting by
Diagnosing Snow Density
*Also affiliated with the NOAA/OAR/National Severe Storms Laboratory
Corresponding author: Paul J. Roebber, Department of Mathematical Sciences, University of Wisconsin at Milwaukee, 3200 N. Cramer Ave., Milwaukee, WI 53211. E-mail: email@example.com
A review of the microphysical literature reveals that many factors may contribute to snow density, including in-cloud (crystal habit and size, the degree of riming and aggregation of the snowflake), sub-cloud (melting and sublimation) and surface processes (compaction and snowpack metamorphism). Despite this complexity, the paper explores the sufficiency of surface and radiosonde data for the classification of snowfall density.
A principal component analysis isolates seven factors that influence the snow ratio: solar radiation (month), low- to mid-level temperature, mid- to upper-level temperature, low- to mid-level relative humidity, mid-level relative humidity, upper-level relative humidity, and external compaction (surface wind speed and liquid equivalent). A ten-member ensemble of artifical neural networks is employed to explore the capability to determine snow ratio in one of three classes: heavy (1:1 < ratio < 9:1), average (9:1<=ratio<=15:1), and light (ratio>15:1). The ensemble correctly diagnoses 60.4% of the cases, which is a substantial improvement over the 25.8% correct using the ten-to-one ratio, 41.7% correct using the sample climatology, and 51.7% correct using the National Weather Service "new snowfall to estimated meltwater conversion" table.
The most critical inputs to the ensemble are related to the month, temperature, and external compaction. Witholding relative humidity information from the neural networks leads to a loss of performance of at least 5%, suggesting that these inputs are useful, if nonessential. Examples of pairs of cases highlight the influence that these factors have in determining snow ratio. Given the improvement over presently used techniques for diagnosing snow ratio, this study indicates that the neural network approach can lead to advances in forecasting snowfall depth.
Forecasting snowfall depth is, at present, a two-step problem. First, an assessment must be made of the amount of liquid water that is to fall, the quantitative precipitation forecast (QPF) problem (e.g., Fritsch et al. 1998). Second, this liquid water must be converted into snow, hereafter the snow-density problem. Historically, very little research has addressed the snow-density problem, as discussed by Super and Holroyd (1997, p. 20) and Judson and Doesken (2000, section 1). As we argue below, current operational practice of forecasting snow density, and hence snowfall depth, is still largely a nonscientific endeavor. Consequently, even if predictions of QPF were accurate and precise, large errors in snowfall forecasts (by factors of 2 to 10) could still occur as a result of inaccurate predictions of snow density. In this paper, we present a method to assess the complex problem of snow-density forecasting.
a. The QPF and snow-density problems
The economic and social value of accurate QPFs is well recognized (e.g., Fritsch et al. 1998). QPFs, however, have historically posed a significant operational forecast challenge (e.g., Charba and Klein 1980) and, in recent years, have exhibited only very modest increases in skill (e.g., Olson et al. 1995; Roebber and Bosart 1998; Fritsch et al. 1998). QPF skill undergoes substantial seasonal fluctuations, with the lowest skills in the warm season in association with the increased potential for convective precipitation (e.g., Olson et al. 1995). Even in the cold season, however, skill remains modest, with mean threat scores for heavy precipitation (>= 25 mm) and heavy snow (>= 100 mm 12 h-1) of 0.27 and 0.20, respectively (Olson et al. 1995).
In the cold season, forecasts of precipitation type are required as well, since large sectors of the economy (e.g., transportation, construction, agriculture, commerce) are affected by snow and freezing rain. The cold-season QPF problem requires (in addition to information concerning the myriad factors that govern precipitation generation, type, and amount) insight into the physical processes controlling the depth of the snowfall via the snow density. Snow-density forecasting is important not only for operational weather forecasts for snowfall, but also for avalanche forecasting, snowmelt runoff forecasting, snowdrift forecasting, and as an input parameter in the snow accumulation algorithm for the WSR-88D radars (Super and Holroyd 1997). Accurate forecasts of the depth of the snowfall are critical to many snow removal operations, since these activities are triggered by exceedances of specific snow-depth thresholds (Gray and Male 1981, 671-706).
b. Forecasting snow density: The ten-to-one rule and current operational practice
Snow density is often assumed to conform to the ten-to-one rule: the snow ratio, defined by the density of water (1000 kg m-3) to the density of snow (assumed to be 100 kg m-3), is 10:1. For example, if a forecaster believes that one inch of liquid water will fall at a given point over a specified time, then the ten-to-one rule implies a total snowfall of 10 inches. As noted by Judson and Doesken (2000), the ten-to-one rule appears to originate from the results of a nineteenth century Canadian study. Potter (1965, p. 1) quotes from this study: "A long series of experiments conducted by General Sir H. Lefroy, formerly Director of the Toronto Observatory, led to the conclusion that this relation [ten to one] is true on the average. It is not affirmed that it holds true in every case, as snow varies in density..." The ten-to-one rule has persisted, however, despite the almost immediate warnings concerning its accuracy (Abbe 1888, p. 386).
More comprehensive measurements of snow density (e.g., Currie 1947; LaChapelle 1962, reproduced in Doesken and Judson 1997, p. 15; Power et al. 1964; Super and Holroyd 1997; Judson and Doesken 2000) have established that this rule is an inadequate characterization of the true range of snow densities. Snow densities can vary from 10 kg m-3 (a snow ratio of 100:1) to approximately 350 kg m-3 (2.9:1). In interviews with current National Weather Service (NWS) forecasters from around the country, we find that the ten-to-one rule may be modified slightly to 12:1 or 20:1, depending on the mean or median value of snow density at a particular station (e.g., Currie 1947; Holroyd and Super 1997, p. 27). The concept of uniformly applying a fixed snow ratio, though, is typically employed. Thus, the persistence of the ten-to-one rule has served merely to exacerbate the snow-forecast problem by oversimplifying the determination of snow density.
Attempts to specify snow density, usually through reference to in-cloud or surface air temperatures, have shown only marginal diagnostic capability, even when using carefully collected measurements (e.g., Grant and Rhea 1974; Super and Holroyd 1997; Judson and Doesken 2000). Super and Holroyd (1997, p. 27) found that the single best prediction method for snow density is persistence, but this still only explains about 30% of the variance. In the absence of explicit snow density forecasts, some empirical techniques have evolved (e.g., Chaston 1989; Garcia 1994; Wetzel and Martin 2001). [See the review in Wetzel and Martin (2001) for a complete list of such techniques.] As argued in Schultz et al. (2002), however, these techniques are inadequate and remain unverified for a large variety of events. To gain further understanding of the snowfall forecasting problem, we next review the physical processes that affect snow density.
c. Factors affecting snow density
The density of snowfall is related to the ice-crystal structure by virtue of the relative proportion of the occupied volume of crystal composed of air. Snow density is regulated by (a) in cloud processes that affect the shape and size of growing ice crystals, (b) subcloud processes that modify the ice crystal as it falls, and (c) ground-level compaction due to prevailing weather conditions and snowpack metamorphism. Understanding how these processes affect snow density is difficult because direct observations of cloud microphysical processes, thermodynamic profiles, and surface measurements are often unavailable.
Cloud microphysical research indicates that many factors contribute to the final structure of an ice crystal. One factor that has been shown to affect snow density is the habit (shape) of the ice crystal. In an environment supersaturated with respect to ice, incipient ice particles will grow to form one of many types of crystal habits (e.g., Magono and Lee 1966; Ryan et al. 1976; Fukuta and Takahashi 1999). The habit is determined by the environment in which the ice crystal grows: specifically, the surrounding air temperature and degree of supersaturation with respect to ice and liquid water. For example, at temperatures of 0° to -4°C, plates dominate; at -4° to -10°C, prisms, scrolls, sheaths and needles dominate, with the specific habit at these temperatures controlled by the degree of supersaturation; at -10° to -20°C, thick plates, sectors, and dendrites are found, depending on supersaturation; and at temperatures less than -20°C, hollow columns and sheaths dominate. Power et al. (1964) showed that dendrites have the lowest density. Although needles would be expected to be dense, they are the next lighter crystal habit since they tend to aggregate at the relatively high temperatures at which they form (Power et al. 1964). Finally, columns and plates were the next densest form. Snow, however, is not a homogeneous collection of ice crystals. As an ice crystal falls, it passes through varying thermodynamic and moisture conditions, with each new crystal habit being superimposed upon the previous structure, such that the final habit is a combination of various growth modes (e.g., Pruppacher and Klett 1997, p. 44).
Another factor affecting density is crystal size. Large dendritic crystals will occupy much empty air space, whereas smaller crystals will pack together into a denser assemblage. The ultimate size of crystals depends on factors that affect the growth rate, such as the residence time in the cloud and the degree of supersaturation with respect to ice. For a given degree of supersaturation with respect to ice, ice-crystal growth by vapor diffusion is dependent primarily on temperature, as well as secondary pressure effects (e.g., Byers 1965; Ryan et al. 1976; Takahashi et al. 1991; Fukuta and Takahashi 1999). At 1000 hPa, the growth rate maximizes near -14°C, whereas at 500 hPa, the rate maximizes near -17°C (e.g., Byers 1965, p. 123). The maximum growth rate is expected to occur near the level of maximum upward air motion within the cloud, where the greatest water vapor delivery occurs (Auer and White 1982). Through natural variations in ice-crystal size, some crystals will grow relative to their neighbors within the cloud and begin to fall, thus promoting the sweepout of smaller particles. If this sweepout occurs in a cloud of ice crystals, then aggregation leads to the formation of snowflakes, and relatively low snow density. An ice crystal falling through a cloud of supercooled water droplets, on the other hand, will lead to rimed crystals (graupel) through accretional growth, and very high snow densities (e.g., Power et al. 1964; Judson and Doesken 2000).
After leaving the cloud, the snow density can also be affected by sublimation and melting. Sublimation occurs when the air surrounding the crystal is subsaturated with respect to ice, whereas melting occurs when the temperature of the air surrounding the crystal is greater than 0°C. Substantial sublimation or melting of a snowflake can occur over relatively short vertical distances (~500 m; Houze 1993, p. 199). Consequently, the liquid water content and the density of a snowflake when it reaches the ground can be strongly dependent upon the low-level air temperature and relative humidity. Hence, the subcloud thermodynamic stratification through which an ice crystal falls also will be a critical factor in determining snowfall density.
Finally, once the snow lands on the ground, compaction of the ice crystals due to prevailing weather conditions and snowpack metamorphism may occur. Wind greater than approximately 9 m s-1 can move ice crystals at the surface, fracturing the crystal during saltation and causing surficial compaction, increasing snow density (e.g., Gray and Male 1981, 345-350). The weight of the snowfall can further compress the snowpack. Within hours of falling, snow metamorphism can occur, in which water sublimates from the sharper edges of the ice crystals and deposits on the more rounded edges, making the snow crystals more rounded and dense (e.g., Gray and Male 1981, 277-285; Doesken and Judson 1997, 18-19). Such destructive metamorphism is accelerated at temperatures approaching freezing.
d. This study
Although the complexity of the snow-density problem is considerable, there is some expectation that the bulk effects of these processes can be assessed through reference to the temperature and moisture profiles within and below the cloud. The existing observational network does not provide such profiles in great temporal or spatial detail (e.g., radiosonde measurements are taken only twice daily with approximate station separations of 300-400 km in the United Sates), however, the regional character of both air masses and measured snow densities (e.g. Potter 1965; Judson and Doesken 2000) suggests the possibility of isolating a useful signal from the existing, albeit deficient, observational record. Such a procedure would clearly have great operational utility. Hence, in this paper, the feasibility of the diagnosis of snow density using only routinely available measurements is addressed.
The difficulty of measuring snowfall and its implications for this study are discussed in section 2. The approach used to diagnose snow density, employing artificial neural networks (ANNs), is discussed in section 3. An interpretation and discussion of these results is offered in section 4. A summary of the findings of this paper is given in section 5.
The purpose of this study is to develop a methodology to diagnose snow density using operational data collected by the NWS over areas of the continental United States receiving frequent, measurable snowfall. The review of factors affecting snow density in section 1 suggests that upper-air observations of temperature and humidity, and surface observations of temperature, wind speed, and precipitation are needed. Thus, snowfall events for this study must be collocated with an upper-air observing site. Sounding data from these sites were extracted from the National Climatic Data Center (NCDC)/Forecast Systems Laboratory North American radiosonde dataset (Schwartz and Govett 1992; http://raob.fsl.noaa.gov/Raob_Software.html).
The new-snow amounts (six-hourly values) come from the United States Air Force DATSAV2 Surface Climatic database, whereas the NCDC hourly precipitation dataset TD-3240 provided the liquid equivalent precipitation. Two different datasets were employed since the DATSAV2 reports generally included only 24-hour precipitation measurements. NCDC Summary of the Day snow reports were examined to ensure consistency between datasets.
Based on potential snowfall events culled from these two datasets, quality control was accomplished in the following manner. First, since temporal variations in density can occur within an event over just a few hours (e.g., Super and Holroyd 1997, 25-27), the reported snow must have occurred within six hours of the nominal time of a radiosonde launch (0000 or 1200 UTC). Second, measuring the depth of new snow is problematic due to metamorphism, compaction, drifting, the frequency of snow-depth measurement (e.g., Doesken and Judson 1997; Doesken and Leffler 2000), and the type of gauge used (e.g., Goodison 1978; Groisman et al. 1991; Groisman and Legates 1994). To mitigate against errors and imprecision in measurement, the depth of new snow must have measured at least 50.8 mm (2 inches) and the melted equivalent precipitation at least 2.8 mm (0.11 inches), this latter threshold being defined by a simple error analysis for snow ratio (e.g., Judson and Doesken 2000, p. 1584). Third, to reduce measurement errors associated with substantial snow drift, only events for which surface winds were less than or equal to 9 m s-1 at the time of the radiosonde report were considered (Gray and Male 1981, 345-350). The quality control measures produced 1650 snowfall events over the 22-y period 1973-1994 for the 28 stations shown in Fig. 1.
The values of the snow ratios for the 1650 events range from 1.9:1 to 46.8:1 (Fig. 2a) and the values for snow densities range from 21.4 to 526.3 kg m-3 (Fig. 2b). The mean and median snow ratios are 15.6:1 and 14.1:1, respectively, whereas the mean and median snow densities are 80.9 and 70.9 kg m-3, respectively. Figure 2a shows that the mode snow ratio is about 10:1 (14% of the events have snow ratios between 9:1 and 11:1), suggesting some weak indication for the validity of the ten-to-one rule. [The mode snow density is between 50 and 60 kg m-3 (Fig. 2b).] These 14% are a subset of 41% classified as average (snow ratios between 9:1 and 15:1; snow densities between 67 and 111 kg m-3; see section 3). Another 14% of the events have snow ratios less than 9:1 and are classified as heavy. The remainder of the events, however, have snow ratios greater than average (45%) and are classified as light. The shape of the curve in Fig. 2b and its skewness towards high snow densities are consistent with previously published studies (e.g., LaChapelle 1962, reproduced in Doesken and Judson 1997, p. 15; Super and Holroyd 1997, p. 23; Judson and Doesken 2000).
The shape of the histograms of snow ratio or snow density from individual observing sites can differ substantially, even for locations in close proximity, as shown by Super and Holroyd (1997) and Judson and Doesken (2000). The variability of the snow ratios from the individual sites in this paper is a subject of continuing research and will not be presented here.
3. Snow-density diagnosis
Despite the quality-control procedures described in section 2, it is expected that the computed snow ratios are subject to imprecision, given the difficulties in measuring snowfall. Consequently, for the purposes of diagnosis, all snow ratios are categorized according to three specified classes that reflect the distinct density characteristics of snowfalls (Fig. 2): heavy (1:1 < ratio < 9:1), average (9:1 <= ratio <= 15:1) and light (ratio>15:1). The average class is defined to include the mode of the snow ratio distribution (Fig. 2a), but extending to higher ratios as suggested by Super and Holroyd (1997), while the heavy and light classes extend outward from either side of that class. Diagnosis of snow ratio then becomes a classification problem and the requirement is to define the boundaries in the input space (composed of the set of snow-ratio predictors) that define the separation between snow-ratio classes. In particular, a discriminant function is sought which evaluates every position in input space and produces a high value for the correct class and low values for the two others.
a. Artificial neural networks (ANNs)
Artificial neural networks (ANNs) are widely used in classification problems (e.g., Bishop 1996; Principe et al. 2000). A general review of applications of ANNs to problems in meteorology and oceanography can be found in Hsieh and Tang (1998). Further information about ANNs can be found in Marzban and Witt (2001), at ftp://ftp.sas.com/pub/neural/FAQ.html, and in the appendix of this paper. An ANN can be defined as a network of many simple processors (processing elements), joined by communication channels (connections) that carry numeric data. The processing elements operate on the inputs they receive via the connections. The architecture of the ANN is defined by the connectivity. ANNs typically follow a training rule by which the weights of the connections are adjusted on the basis of examples provided by training data. In this way, ANNs are said to "learn" and can exhibit some capability for generalization beyond the training data (in other words, the network can produce reliable results for cases not represented exactly in the training data). It is important to recognize, however, that there are no methods for training ANNs that can create relationships not contained within the training data.
In this work, training of the ANNs is accomplished using supervised learning. The inputs (i.e., predictors for snow ratio) and the desired snow-ratio classes are given to the ANN during training so that the network can adjust its connections to match the predicted and desired classes. This phase is distinct from testing, in which the trained network is only given the inputs, not the desired classes. Here, the term "population" is used to describe the set of all cases for which the system is designed, and "sample" (a subset of the population) for the set of cases that are actually available for training and testing. In order to accomplish training and testing, the sample is divided into a training set (here, 60% of the sample), a cross-validation set (20%), and a test set (20%). The training set is used to provide examples to the network to fit the weights and optimize the network snow ratio class (accomplished by minimization of a cost function, in this case, the mean square error). Prior to training, data is reserved for a cross-validation early-stopping procedure, which provides a means to prevent overfitting of the network weights. The test set is used to assess independently the performance of the trained network.
The review of ANNs (see above and appendix) suggests the following salient points. First, the inputs should be selected based upon a priori knowledge of the physical system. Second, the number of inputs should be minimized to constrain the dimensionality of the problem. Third, the training set should be comprehensive (i.e., representative of the population), such that the full range of scenarios to be classified are presented for training. Fourth, training should be accomplished using early stopping to limit overfitting. Fifth, a variety of network architectures should be evaluated and combined into an ensemble to improve results. These criteria are all met in the study methodology as outlined below.
b. Study procedure and results
Given the relationship between thermodynamic profiles and snow density outlined in section 1, the classifier inputs were defined from measurements of temperature and relative humidity obtained from standard radiosonde launches. In order to account for on-the-ground compaction processes (Gray and Male 1981, 275-306), surface wind speed and total melted equivalent precipitation were also used as inputs. Since surface elevation at the sites depicted in Fig. 1 varies from 1.6 km to near sea level, a sigma coordinate system was adopted to define a consistent set of 14 vertical levels for all soundings (Table 1). Further, to constrain the dimensionality of the problem, principal component analysis was used to extract a set of six factors from the original set of 29 inputs (temperature at all 14 levels, relative humidity at the first 13 levels, surface wind speed, total melted equivalent precipitation). [The objective of factor analysis is to reduce a set of correlated variables to a smaller number of factors, which are linear combinations of the correlated variables. In this work, the extracted factors are constrained to be orthogonal (zero correlation between the factors). The factor extraction method used principal components analysis (Hotelling 1933). The initial factoring method, used to determine the number of factors, was a combination method, exploying the larger of the 75% variance rule (Gorsuch 1983) and root curve analysis (Cattell 1966; Cattell and Jaspers 1967). A Varimax transformation method was employed (Kaiser 1958). See Gorsuch (1983) for more details on factor analysis.] The six factors produced by the factor analysis (Table 2) are associated with temperature (F1, F3), relative humidity (F2, F4, F5), and surface compaction (F6). The temperature and moisture factors are further stratified by level, representing upper-level (about 700-500 mb), mid- to upper-level (875-400 mb), mid-level (850-700 mb) and low- to mid-level (below 850 mb). A monthly index meant to represent the effects of solar radiation (January = +1, July = -1) was added as a seventh input (Table 2). This procedure ensures that the inputs are based upon physical knowledge and that the number of these inputs is relatively small.
Analysis of the frequency distributions of the inputs, stratified according to snow-ratio class, is a useful method for assessing their first-order diagnostic capability (Fig. 3). Based upon visual inspection, the greatest separations between the classes are revealed by the month index, F1 (low- to mid-level temperature), F3 (mid- to upper-level temperature) and F6 (compaction). However, these results do not necessarily imply a lack of utility of the remaining three factors, since nonlinear interactions between inputs may enhance diagnostic capability.
Since these seven inputs all have potential discriminatory capability, they were all used in the neural networks. Two types of network architectures were selected to define the ensemble, which was constructed from ten individual networks (e.g., Opitz and Maclin 1999; see appendix). The training datasets for each individual member of the ensemble were distinct, having been obtained from the original training set using the bootstrapping technique (Efron and Tibshirani 1993). This approach has been shown to be effective in producing unique networks (i.e., variable weights), which allows decorrelation of the errors of the individual networks and hence, improved ensemble performance (Breiman 1996; Opitz and Maclin 1999).
The first network type, composing five members of the ensemble, is the multilayer perceptron (MLP) with a single hidden layer (Fig. 4a). MLP networks have been used extensively in classification problems (e.g., Bishop 1996; Principe et al. 2000). Each of the single hidden layer MLPs featured 7 inputs, 40 processing elements in the hidden layer, and 3 outputs (the 3 snow ratio classes). These architectural details were established through trial-and-error (see appendix), maximizing percent correct from the training dataset.
The second network type, also composing five members of the ensemble, is the MLP with two hidden layers (Fig. 4b). The two hidden layer MLPs, whose architectural details were also established through trial-and-error, featured 7 inputs, 7 processing elements in the first hidden layer, 4 processing elements in the second hidden layer, and 3 outputs. For all networks, activations were accomplished using the hyperbolic tangent function everywhere except in the output layer, where the softmax activation function was used (so that the output values would lie between zero and one and sum to one).
When training neural networks, it is sometimes necessary to consider the frequency of occurrence of different classes in the training set. For the 1650 cases composing the entire 22-y dataset, 14% are heavy, 41% are average and 45% are light. For such unbalanced datasets where one class is substantially underrepresented compared to the other classes, the neural networks may simply treat that class as noise. Results from preliminary experiments with the networks for the snow-ratio problem exhibited this effect. Accordingly, a partial balancing criteria (270 heavy, 360 average, and 360 light cases in each training set) was applied to the bootstrap sampling, to improve training results across all three classes. As a result, the network outputs are not interpretable as posterior probabilities (that is, a probability that takes into account the frequency of occurrence of a given class), without application of a renormalization procedure.
Following the training of the ten individual networks, each of the networks was run on the independent test dataset. The individual network outputs were then combined into an ensemble using simple averaging, where the snow-ratio class with the highest ensemble value was compared with the observed class. The probability of detection (POD), false alarm ratio (FAR), bias, and critical success index (CSI) for each snow-ratio class were used as measures of performance. As expected, the ensemble approach resulted in improvement relative to the individual networks. Using the CSI as the metric of effectiveness, the ensemble outperformed the individual networks in 77% of the 30 possible situations (10 networks times 3 CSI per network), with no individual network exceeding the ensemble in all three snow-ratio classes.
Overall, the diagnosis achieves 60.4% correct (Table 3). The performance measures show that the highest forecast quality is generally associated with the light snow-ratio class, with a monotonic drop-off through average to the heavy class. The results suggest that the method provides high-quality diagnoses for all snow-ratio classes. A comparison of the ANN diagnosis scores with those obtained from the ten-to-one rule, observed frequencies (e.g., climatology) of snow-ratio classes, and diagnoses obtained from the NWS "new snowfall to estimated meltwater conversion table" (U.S. Department of Commerce 1996; see also section 4d of this paper; hereafter, NWS Table 4-9) shows that the ANN results are skillful (Table 3). Using the ten-to-one rule achieves 25.8% correct, the climatological estimate 41.7% correct and the NWS Table 4-9 51.7% correct across the three classes. Further, all of the performance measures (save bias) are substantially lower than those obtained from the ANN ensemble. The NWS Table 4-9, which is based upon surface temperature alone, falls short of the ANN ensemble accuracy (as measured by CSI) for all snow-ratio classes.
a. Interpretation of network diagnoses
An important but often overlooked aspect of the use of ANNs is the interpretation of the outputs. Since it is not necessary to have prior knowledge of the precise relationship between the inputs and the desired outputs in an ANN, a trained network that generalizes well may yield new knowledge concerning these relationships.
The importance of inputs could be measured by several methods, including evaluation of the network weights (e.g., size, sums of products, or more elaborate functions), or measuring the gradient of the output with respect to the input (i.e., a vector of partial derivatives that measures the local rate of change of the output with respect to the corresponding input, holding the other inputs fixed). Each of these approaches suffers from deficiencies which may lead to incorrect conclusions concerning importance. A thorough discussion of these issues can be found at: ftp://ftp.sas.com/pub/neural/importance.html.
A more robust method is to compute differences rather than derivatives, an approach that adheres to the strict definition of causal importance. This method is applied here by varying each of the seven inputs (Table 2) in turn across the range of values existing in the test dataset while holding the other inputs to their observed values. The average value of an output snow-ratio class across all the cases (which can range from zero to one) is plotted as a function of the input value, as a means of defining the central tendency of the input importance. Since there may be substantial variations in the response under certain circumstances (consider, for example, cases in which no changes occur until a critical threshold is passed, such as the familiar example of precipitation), the standard deviation in the response across all cases as a function of the input value is also computed.
The results of this analysis are shown in Fig. 5 for the four most critical inputs (month index, F1, F3, F6). The march of the seasons has the most substantial effect on the heavy and light snow-ratio classes, with an increasing likelihood of either class in midwinter, albeit subject to considerable variation in the response (Fig. 5a). Increasing low- to mid-level temperatures have the effect of decreasing (increasing) the possibility of light (heavy) snow ratios, whereas the response of the heavy class is the most variable (Fig. 5b). Increasing temperatures aloft yields a similar response (Fig. 5c). Increasing winds and/or larger liquid equivalent precipitation amounts, both processes tending to produce greater surface compaction, also leads to decreasing (increasing) the likelihood of light (heavy) snow ratios, although in this case the light class response exhibits the greatest variability (Fig. 5d).
The large variation in response (>=0.25) for the heavy and light classes to month index (Fig. 5a), for the heavy class to low- to mid-level temperatures (Fig. 5b) and to mid- to upper-level temperatures (Fig. 5c), and for the light class to surface compaction (Fig. 5d), is an important indicator of the complexity of the problem. A specific comparison of two instances with an average low- to mid-level temperature of -11°C, generally indicative of a light snow ratio (Fig. 5b), demonstrates this dependence in the ANN. In the first (second) example, the profile of the remaining inputs yield the following representative conditions: month April (March), average mid- to upper-level temperature ~-20°C (-7°C), average low- to mid-level relative humidity ~93% (88%), average mid-level relative humidity ~91% (90%), average upper-level relative humidity ~80% (91%), precipitation ~15 mm (38 mm) with no wind. The primary difference between the two examples is in the average mid- to upper-level temperature and precipitation. Both of these factors act to produce higher densities in the second example, which results in a network response of 0.77 for the heavy class. In contrast, the first example exhibits a network response of 0.10 for that same class. Hence, although the general effect of an input can be isolated as depicted in Fig. 5, the actual response in a specifc case can be strongly modified by the conditions represented by the other inputs.
Finally, the importance of the profiles of relative humidity (as represented by the principal component factors F2, F4 and F5) in the diagnosis of snow-ratio class was further examined. Experiments in which the networks were retrained while witholding the relative humidity inputs indicated an overall network performance loss of at least 5% (based upon percent correct), suggesting that these inputs are helpful, if nonessential, diagnostic quantities. A demonstration of this effect is provided by a pair of examples, in which all of the inputs except the average low- to mid-level relative humidities, are identical. In the first (second) example, these humidities are ~40% (100%). The remaining conditions are the following: month (March), average low- to mid-level temperature ~-8°C, average mid- to upper-level temperature ~-5°C, average mid-level relative humidity ~92%, average upper-level relative humidity ~94%, precipitation ~24 mm with no wind. For the first (second) example, the network response for the heavy class is 0.60 (0.82). In both examples, the relative warmth of the sounding and higher precipitation amounts skew the network towards the heavy class. In the second example, the elevated low- to mid-level relative humidity indicates the availability of supercooled liquid water and the likelihood of riming, shifting the network response further towards the heavy class.
b. Case examples
In order to place the network results into synoptic context, a search was conducted for cases in which all but one of the input variables showed considerable agreement (as measured by standardizing the inputs to mean 0 and standard deviation 1, and computing the Euclidean distances for all the inputs relative to the comparison case). Hence, the comparison cases will not necessarily be from the same site or date. This process was repeated for the two temperature inputs (F1 and F3) and the compaction input (F6). Soundings from this analysis for low- to mid-level temperatures (F1) (Figs. 6a,b) document the similarity in thermodynamic structure between Rapid City, South Dakota (RAP) and Peoria, Illinois (PIA), except for the strong surface-based inversion at RAP, consistent with that site being located on the cold side of a frontal boundary (Fig. 7c). For this event at RAP, the snow-ratio class was light (ratio of 18:1). PIA, with warmer low- to mid-level temperatures in association with a site location shifted further towards the warm side of the low level thermal gradient (Fig. 7a) and within the warm-core circulation at 850 hPa (Fig. 7b,d), developed heavy snow-ratio conditions (ratio of 7:1).
The comparison soundings for mid- to upper-level temperatures (F3) (Figs. 8a,b) document the relative warmth at those levels at Huntington, West Virginia (HTS), which led to a heavy snow ratio (8:1), while an average snow-ratio (12:1) developed at Amarillo, Texas (AMA). At the surface, both sites were positioned to the northeast of a cyclone and associated frontal boundary, with arctic air to the north and west (Figs. 9a,c). However, the 500-hPa flow was nearly zonal across eastern North America in the HTS case (562-dam 500-hPa height; Fig. 9d), whereas a trough was positioned across the southern and central United States in the AMA case (548-dam 500-hPa height; Fig. 9b).
Finally, the thermodynamic profiles at Huron, South Dakota (HON) and International Falls, Minnesota (INL) (Figs. 10a,b) exhibit considerable similarity, consistent with the primary differences resulting from surface compaction processes, with stronger winds (8 m s-1 vs 5 m s-1) and higher precipitation amounts (13.0 mm vs 2.8 mm) at HON (average snow ratio of 10:1) than at INL (light snow ratio of 22:1). Interestingly, low-level temperatures at both these sites are fairly warm, which affirms the complexity in some instances in correctly diagnosing snow ratios. Examination of the synoptic charts reveals that both sites were positioned between a downstream anticyclone and an upstream cyclone (Figs. 11a,b), with HON located most directly within the pressure gradient (~5 m s-1 stronger geostrophic southerlies).
Each of these three cases suggest that subtle synoptic-scale differences can be associated with a change in snow-ratio class. However, the results presented in Table 3 indicate that considerable improvement beyond current operational practice can be obtained.
c. Deficiencies of NWS Table 4-9
Finally, the results of Table 3, obtained by application of NWS Table 4-9 to the test data, make clear that serious damage to the climatological record will ensue in the absence of careful direct measurements of snow depth and liquid equivalent. Unfortunately, current NWS practice is not promising in this regard. U.S. Department of Commerce (1996, section 184.108.40.206.2) states, "Report the water equivalent of snow on the ground to the WSFO or WFO at 1800 UTC if the average snow depth (to the nearest inch) is 2 inches or more. . . . Whenever the water equivalent of snow on the ground cannot be measured by melting or weighing of the core sample, enter an estimated water equivalent on the basis of a 1/10 ratio method unless a different ratio is more appropriate. . . . Table 4-9 (New Snowfall to Meltwater Conversion) can only be used as an observing aid in determining water equivalency of newly fallen snow."
Super and Holroyd (1997, p. 24) shed further light on this problem. ". . . personal communication with Grant Goodge of the National Climatic Data Center revealed this table [NWS Table 4-9] was developed as a guide for quality control checking of observations and was not intended to be a substitute for observations of [snow water equivalent]. The table's temperature dependence of density is not based on actual measurements but rather on general impressions in the eastern Tennessee and western North Carolina areas. Hence, the reality of the apparent temperature dependence is uncertain." As was shown in the previous section, this temperature dependence is, in fact, inadequate.
Key findings of this paper are the following. First, seven factors influence the diagnosis of snow ratio: solar radiation (month), low- to mid-level temperature, mid- to upper-level temperature, low- to mid-level relative humidity, mid-level relative humidity, upper-level relative humidity, and external compaction (surface wind speed and liquid equivalent precipitation amount). Second, a ten member ensemble of artificial neural networks can be used to improve the diagnosis of snow-ratio class compared to existing techniques (ten-to-one ratio, sample climatology, NWS new snowfall to estimated meltwater conversion table). Third, the most critical factors are related to month, temperature and external compaction, with relative humidity information providing useful, but less essential input.
Until an integrated and well-verified scheme for forecasting snow is provided as a direct output from numerical weather prediction models, it will be necessary to convert liquid water to snow through the diagnosis of snow-ratio. The method developed in this paper could be applied in the operational environment as follows: (a) QPF techniques, including the output from numerical models, are used to determine the amount of liquid equivalent precipitation; (b) forecast soundings and the QPF obtained from (a) are used within the neural network framework described in this study to diagnose the snow-ratio class; (c) the snowfall forecast is derived from the results of (a) and (b). For example, consider the simple case in which 0.5 inches of liquid equivalent precipitation is forecast and for which the snow-ratio class is diagnosed to be light. Then, the forecast snow depth would be approximately 10 inches (reflecting a median snow-ratio of 20:1 in the light class). It should be noted that this forecast depth is twice that obtained by application of the ten-to-one rule. Probabilities could be generated by using probabilistic QPF, normalizing the network outputs by class frequency and accounting for the frequency distribution of snow ratios within each class.
Although the findings in this paper suggest that improvements in current snow forecasting practice are already obtainable, a key aspect, the role of in-cloud vertical motions (highlighted in the microphysical review of section 1), has been neglected in this study. It is expected that an appropriate vertical motion input (i.e. cloud-scale) would further improve these results, perhaps substantially so. Empirical evidence for the importance of vertical motion is provided at: http://www.nws.noaa.gov/er/hq/ssd/snowmicro. In this preliminary study, the intersection of areas of moderate vertical motion with temperatures favorable for dendritic crystal formation (-12°C to 18°C) within cloudy regions was found to be a useful indicator of snow advisory events. One means to obtain such vertical motions for a longitudinal study of this kind might be to use a mesoscale model and the NCEP/NCAR Reanalysis (Kalnay et al. 1996) within a four-dimensional data assimilation framework. In this approach, the Reanalysis data could be used to provide lateral boundary conditions in a nested mesoscale model configuration, with an interior domain of sufficient resolution to obtain the necessary cloud-scale vertical motions.
Another factor that may negatively affect the performance of the network diagnosis is storm electrification. As reviewed in MacGorman and Rust (1998, 352-355), variations in the electrical state inside the cloud can produce significant changes in the likelihood of aggregation and fragmentation of ice crystals. The range of electric field over which these effects can be important are consistent with such measurements inside winter storms (e.g., Schultz et al. 2002). Unfortunately, as with in-cloud vertical motion, direct measurements of electrical properties of precipitating clouds are not available. Such investigations could form the basis of future research on this problem.
Acknowledgments. The lead authors (P. Roebber and S. Bruening) thank the Department of Mathematical Sciences at the University of Wisconsin - Milwaukee (UWM) for providing some funding support for this work. Funding for D. Schultz and J. Cortinas was provided by NOAA/OAR/NSSL under NOAA-OU Cooperative Agreement #NA17RJ1227. The use of the Storm Prediction Center microfilm collection is gratefully acknowledged. The comments of V. Larson of UWM helped to clarify aspects of the microphysical discussion.
A major ANN architecture is the feedforward type (that is, the connections between processing elements do not form loops) known as the multilayer perceptron (MLP; Figs. 4a,b; Bishop 1996). This architecture can contain one or more hidden layers, i.e. layers containing processing elements that are not directly connected to either the inputs or the desired outputs. It can be shown that an MLP with one hidden layer and an arbitrarily large number of processing elements can approximate any function (e.g., Bishop 1996). In practice, however, it is not possible to specify the number of processing elements that are required, and for more complicated functions, it is often more efficient to have several processing elements in a second hidden layer. Networks with too few hidden processing elements will generalize poorly as a result of underfitting (i.e., insufficient specification of the mapping between the inputs and the outputs), while networks with too many hidden processing elements will also generalize poorly, in this case due to overfitting (which produces a model of the statistical noise as well as the desired signal). Since there is no theoretical basis for defining the number of hidden processing elements, this aspect of the architecture is obtained through experimentation.
The connections between processing elements are constructed using combination and activation functions. In MLPs, each noninput processing element linearly combines values that are fed into it through connections with other processing elements, producing a single value called the net input. An activation function then nonlinearly transforms the net input (necessary to represent nonlinear discriminant functions), yielding an "activation", which is fed through connections to other processing elements. A common activation function, used in this study, is the hyperbolic tangent, which tends to produce fast training. The desired outputs, the snow-ratio classes, are defined using 1-of-C coding, such that three binary output variables (0/1) are specified, one for each snow-ratio class.
A requirement for a network to generalize well is its ability to represent every part of the multidimensional input space. In this way, it is possible to define the mapping from the inputs to the desired outputs. Since the data required to define the mapping is proportional to the hypervolume of the input space, networks with more inputs will require more training examples. Further, the training set must be a sufficiently large and representative sample of the population to promote interpolation (i.e., test cases are in the "neighborhood" of training cases) rather than extrapolation (i.e., test cases lie outside the range of the training cases or are within holes in the training input space). This also suggests that if irrelevant inputs are provided to the network, then performance will tend to be poor, since the network will squander resources in order to represent irrelevant portions of the mapping. Hence, careful selection of inputs is required to improve generalization. Another important generalization consideration relates to noise in the desired outputs. Noise increases the risk of overfitting (Moody 1992), since the network will attempt to approximate artificial boundaries in the training data rather than the desired signal. Problems associated with overfitting can be reduced by using large training sets, applying early stopping and combining networks (into a so-called committee of networks).
The principle of a committee of networks (hereafter, the ensemble network) is tied to the familiar concept of ensemble forecasting. For a set of M networks for which the errors are uncorrelated with each other and for which there is no bias, it can be shown that the error of the ensemble network will be M times smaller than the mean error of the individual networks (Principe et al. 2000). In practice, network errors will tend to be correlated such that the actual advantage of the ensemble is less than that suggested by the theory. By incorporating different network architectures into the ensemble, it is possible to reduce the occurrence of correlated errors and improve ensemble network performance (Opitz and Maclin 1999 and references therein). Further improvements are made possible by altering the training process, for example, by creating different training sets for each individual network through bootstrapping techniques (Efron and Tibshirani 1993; Breiman 1996; Opitz and Maclin 1999). Previous research suggests that ensembles of O(10) members will provide most of the advantage obtainable using this technique (e.g., Opitz and Maclin 1999).
Abbe, C., 1888: 1887 Annual Report of the Chief Signal Officer of the Army Under the Direction of Brigadier-General A. W. Greely, Appendix 46, U.S. Govt. Printing Office, Washington, DC, 385386.
Auer, A. H., and J. M. White, 1982: The combined role of kinematics, thermodynamics, and cloud physics associated with heavy snowfall episodes. J. Meteor. Soc. Japan, 60, 500507.
Bishop, C. M., 1996: Neural Networks for Pattern Recognition. Clarendon Press, 482 pp.
Breiman, L., 1996: Stacked regressions. Machine Learning, 24, 4964.
Byers, H. R., 1965: Elements of Cloud Physics. University of Chicago Press, 191 pp.
Cattell, R. B., 1966: The Scree test for the number of factors. Multivariate Behavioral Research, 1, 245.
Cattell, R. B. and J. A. Jaspers, 1967: A general plasmode (No. 30-10-5-2) for factor analytic exercises and research. Multivariate Behavioral Research Monographs, 67, 211.
Charba, J. P. and W. H. Klein, 1980: Skill in precipitation forecasting in the National Weather Service. Bull. Amer. Meteor. Soc., 61, 15461555.
Chaston, P., 1989: The magic chart for forecasting snow amounts. Nat. Wea. Dig., 14, 2022.
Currie, B. W., 1947: Water content of snow in cold climates. Bull. Amer. Meteor. Soc., 28, 150151.
Doesken, N. J., and A. Judson, 1997: The Snow Booklet: A Guide to the Science, Climatology, and Measurement of Snow in the United States. Colorado State University, Department of Atmospheric Science, 86 pp.
Doesken, N. J., and R. J. Leffler, 2000: Snow foolin': Accurately measuring snow is an inexact but important science. Weatherwise, 53, 3037.
Efron, B., and R. Tibshirani, 1993: An Introduction to the Bootstrap. Chapman and Hall, New York, 436 pp.
Fritsch, J. M., and co-authors, 1998: Quantitative precipitation forecasting: Report of the Eighth Prospectus Development Team, U.S. Weather Research Program. Bull. Amer. Meteor. Soc., 79, 285298.
Fukuta, N., and T. Takahashi, 1999: The growth of atmospheric ice crystals: A summary of findings in vertical supercooled cloud tunnel studies. J. Atmos. Sci., 56, 19631979.
Garcia, C., 1994: Forecasting snowfall using mixing ratios on an isentropic surface An empirical study. NOAA Tech. Memo. NWS CR-105, PB 94-188760 NOAA/NWS, 31 pp. [Available from NOAA/National Weather Service Central Region Headquarters, Kansas City, MO 64106-2897.]
Goodison, B. E., 1978: Accuracy of Canadian snow gage measurements. J. Appl. Meteor.,17, 15421548.
Gorsuch, R. L., 1983: Factor Analysis. L. Erlbaum Associates, 425 pp.
Gray, D. M. and D. H. Male, 1981: Handbook of Snow: Principles, Processes, Management and Use. Pergamon Press, 776 pp.
Groisman, P. Ya., V. V. Koknaeva, T. A. Belokrylova, and T. R. Karl, 1991: Overcoming biases of precipitation measurement: A history of the USSR experience. Bull. Amer. Meteor. Soc., 72, 17251733.
Groisman, P. Ya., and D. R. Legates, 1994: The accuracy of United States precipitation data. Bull. Amer. Meteor. Soc., 75, 215227.
Hotelling, H., 1933: Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417498.
Houze, Jr., R. A., 1993: Cloud Dynamics. Academic Press, 573 pp.
Hsieh, W. W., and B. Tang, 1998: Applying neural network models to prediction and data analysis in meteorology and oceanography. Bull. Amer. Meteor. Soc., 79, 18551870.
Judson, A., and N. Doesken, 2000: Density of freshly fallen snow in the Central Rocky Mountains. Bull. Amer. Meteor. Soc., 81, 15771587.
Kaiser, H. F., 1958: The Varimax criterion for Varimax rotation in factor analysis. Psychometrika, 23, 187200.
Kalnay, E., and coauthors, 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc., 77, 437472.
LaChapelle, E. R., 1962: The density distribution of new snow. USDA Forest Service, Wasatch National Forest, Alta Avalanche Study Center, Project F, Progress Report No. 2, Salt Lake City, Utah, 13 pp.
MacGorman, D. R., and W. D. Rust, 1998: The Electrical Nature of Storms. Oxford University Press, 422 pp.
Magono, C., and C. W. Lee, 1966: Meteorological classification of natural snow crystals. J. Fac. Sci., Hokkaido University, Ser. 7, II, 321335.
Marzban, C., and A. Witt, 2001: A Bayesian neural network for severe-hail size prediction. Wea. Forecasting, 16, 600610.
Matsuo, T., and Y. Sasyo, 1981a: Empirical formula for the melting rate of snowflakes. J. Meteor. Soc. Japan, 59, 18.
Matsuo, T., and Y. Sasyo, 1981b: Melting of snowflakes below freezing level in the atmosphere. J. Meteor. Soc. Japan, 59, 1024.
Matsuo, T., and Y. Sasyo, 1981c: Non-melting phenomena of snowflakes observed in subsaturated air below freezing level. J. Meteor. Soc. Japan, 59, 2632.
Moody, J. E., 1992: The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. Advances in Neural Information Processing Systems, 4, 847-854.
Nakaya, U., 1954: Snow Crystals: Natural and Artificial. Harvard University Press, 510 pp.
Olson, D. A., N. W. Junker, and B. Korty, 1995: Evaluation of 33 years of quantitative precipitation forecasting at NMC. Wea. Forecasting, 10, 498511.
Opitz, D., and R. Maclin, 1999: Popular ensemble methods: An empirical study. J. Art. Intellig. Res., 11, 169198.
Potter, J. G., 1965: Water content of freshly fallen snow. CIR-4232, TEC-569, Meteorol. Branch, Dept. of Transport, Toronto, ON, Canada, 12 pp. [Available from National Snow and Ice Data Center User Services, Campus Box 449, University of Colorado, Boulder CO 80309 0449.]
Power, B. A., P. W. Summers, and J. d'Avignon, 1964: Snow crystal forms and riming effects as related to snowfall density and general storm conditions. J. Atmos. Sci., 21, 300305.
Principe, J. C., N. R. Euliano and W. C. Lefebvre, 2000: Neural and Adaptive Systems: Fundamentals through Simulations. John Wiley and Sons, Inc., 656 pp.
Pruppacher, H. R., and J. D. Klett, 1997: Microphysics of Clouds and Precipitation. 2nd Edition. Kluwer Academic Publishers, 954 pp.
Roebber, P. J., and L. F. Bosart, 1998: The sensitivity of precipitation to circulation details: Part I: An analysis of regional analogs. Mon. Wea. Rev., 126, 437455.
Rogers, R. R., and M. K. Yau, 1989: A Short Course in Cloud Physics. Third Edition. Butterworth-Heinemann, 290 pp.
Ryan, B. F., E. R. Wishart, and D. E. Shaw, 1976: The growth rates and densities of ice crystals between -3C and -21C. J. Atmos. Sci, 33, 842850.
Schultz, D. M., J. V. Cortinas Jr., and C. A. Doswell III, 2002: Comments on ``An operational ingredients-based methodology for forecasting midlatitude winter season precipitation." Wea. Forecasting, 17, 160167.
Schultz, D. M., and coauthors, 2002: Understanding Utah winter storms: The Intermountain Precipitation Experiment. Bull. Amer. Meteor. Soc., 83, 189210.
Schwartz, B. E., and M. Govett, 1992: A hydrostatically consistent North American radiosonde data base at the Forecast Systems Laboratory, 1946-present. NOAA Tech. Memo. ERL FSL-4, 81 pp. [Available from NOAA/FSL, 325 Broadway, Boulder, CO 80303.]
Super, A. B., and E. W. Holroyd III, 1997: Snow accumulation algorithm for the WSR-88D radar: Second annual report. U.S. Dept. of Interior, Bureau Reclamation Tech. Rep. R-97-05, Denver, CO, 77 pp. [Available from National Technical Information Service, Operations Division, 5285 Port Royal Road, Springfield, VA 22161.]
Takahashi, T., T. Endoh, and G. Wakahoma, 1991: Vapor diffusional growth of free-falling snow crystals. J. Meteor. Jap., 69, 1530.
U.S. Department of Commerce, 1996: National Weather Service Observing Handbook No. 7: Surface Weather Observations and Reports, Part IV, Supplemental Observations. National Weather Service, Silver Spring, MD, 57 pp.
Wetzel, S. W., and J. E. Martin, 2001: An operational ingredients-based methodology for forecasting midlatitude winter season precipitation. Wea. Forecasting, 16, 156167.
|Month index||Jan=+1, Jul=-1, 0.33 per month||Solar radiation|
|F1||T(1) to T(8)||low- to mid-level temperature|
|F2||RH(1) to RH(7)||low- to mid-level relative humidity|
|F3||T(6) to T(14)||mid- to upper-level temperature|
|F4||RH(10) to RH(13)||upper-level relative humidity|
|F5||RH(7) to RH(10)||mid-level relative humidity|
|F6||surface wind, precipitation amount||external compaction|
|Predicted Number of Cases|
|POD||0.606 (0.100) [0.000]||0.573 (0.450) [0.467]||0.633 (0.450) [0.680]|
|FAR||0.592 (0.900) [-------]||0.415 (0.550) [0.493]||0.307 (0.550) [0.477]|
|BIAS||1.485 (1.000) [-------]||0.980 (1.000) [0.920]||0.913 (1.000) [1.300]|
|CSI||0.323 (0.053) [0.000]||0.408 (0.291) [0.321]||0.495 (0.291) [0.420]|
Number of hits: Number of hits: