Appears in Preprints, 13
th AMS Conf.
Wea. Analysis and Forecasting, including Symposium on Flash
Floods (Vienna, VA), 02-06 August 1996,
Amer. Meteor. Soc., 403-410.
As part of a two day guest lecture at the Flash Flood Forecasting Course (FFFC) given at the National Weather Service (NWS) Training Center (NWSTC), we have been giving "pretests" that attempt to reveal something about the students' understanding of the topics to be taught. The pretests have remained essentially unchanged for more than 12 years.
Flash flood events have been growing in importance over the years, relative to other convective storm-related phenomena. This growth in importance is due, at least in part, to the effectiveness of the severe thunderstorm and tornado watch-warning program. Deaths due to tornadoes have declined substantially since the late 1940s, while those attributable to flash floods have shown an increase (Maddox et al. 1978). The FFFC was initiated by the NWS soon after the Big Thompson flash flood of 1976; the first class was taught in the Fall of 1978. An important component of the class was the incorporation of guest lectures in special topic areas; owing to the research done by one of us (RAM) into flash flood events, the Environmental Research Laboratories were invited to participate, and we have done so ever since. While we have taught the majority of the classes, other ERL guest instructors include Dr. C.F. Chappell, Mr. B.E. Schwartz, and Mr. D. Rogers.
Once the classes began, it became clear that a significant emphasis on basic concepts of convection was needed in our lectures, since the students appeared not to be comfortable with some very fundamental notions of how convection operates to produce the heavy precipitation that usually creates flash floods. In order to document the state of student knowledge of these basic concepts, the pretests were introduced. Depending on the answers, the course content could be tailored to the level indicated by the performance of each individual class. As we will show, the questions have been designed to highlight certain issues we feel are important. In some sense, all the questions are intended to be "trick questions" that lead the students into the material we feel is important. Clearly, student performance on these pretests is not part of a formal assessment of student learning achievements in the course. It originally was intended to help the instructor aim the lecture portion of the presentation more precisely at student needs.
However, the history of student performance unexpectedly has some implications regarding certain aspects of the National Weather Service Modernization and Restructuring (MAR). These unanticipated results prompted this paper. In Section 2, we shall describe the pretests briefly, including a brief rationale for the questions. Section 3 will concentrate on typical student answers and how those answers have changed during the time we have taught the course, while Section 4 will conclude the paper with our interpretation of these typical results.
2.1 The "convection" pretest
This pretest is given on the first day of our two in the FFFC, and it precedes the presentation about the basics of convection. It includes the questions shown in Table 1.
Questions 1, 2, and 5 are intended to find out if the student has some basic knowledge about deep, moist convection (generally, thunderstorms). Questions 3 and 4 involve some myths about convection. The last question aims to see if the student is aware of basic thunderstorm structure and effects, and is aware of the environment in which ordinary thunderstorms form.
Table 1. Convection Pretest Questions
1. How long do isolated, summertime, afternoon thunderstorms typically last? ____h ___ min
2. Three primary ingredients led to the thunderstorms of #1 above. These are:
1.________ 2.________ 3.________
3. Which of these ingredients was probably most important? ___
4. Thunderstorms are most likely along cold fronts. True ___ False ___
5. What is the approximate terminal velocity of a grapefruit-sized hailstone? ___ m s-1
6. Quickly add the following to the drawing (Fig. 1): 1. Streamlines of the circulation in and near the thunderstorm. 2. Environmental pressure, wind, temperature and RH at the levels and locations indicated (*) on the figure.
Location Pressure (mb) Temp (deg.F at sfc, Wind (dd/ff) RH (%)
deg.C aloft)
A
B
C
D
E
Figure 1. Sketch of an ordinary thunderstorm near Jackson, Mississippi ("JAN, MS"), showing locations in space (A-E) where quantitative values are sought in Question 6. Point A is shown beneath the thunderstorm's rainshaft, while points B-E are at different levels in the near-storm environment.
This pretest is given on the second day of the course, preceding our lectures on how to analyze operational data sets (as described in Maddox 1979 or Doswell and Maddox 1986). It includes the questions shown in Table 2.
Table 2. Meteorological Analysis Pretest Questions
1. 500 mb PVA is related directly to (a) wave movement, (b) vertical motion, (c) clouds and precipitation.
2. Indicate the one region in Fig. 2 in which you would most likely expect heavy (i.e., > 2 inches) convective precipitation to occur.
3. On the diagrams in Fig. 3, draw in what you might expect the vertical wind profile be look like at Wichita, Kansas during midsummer.
4. On the vorticity analyses attached (Fig. 4) shade in red the regions where you would anticipate significant thunderstorm activity to be occurring at the time of the analyses. Refine your areas so that they are as small as is reasonably possible.
Figure 2. Sketch of an extratropical cyclone for Question 2 in Table 2.
Figure 3. Vertical sections for Question 3 in Table 2
Figure 4. Example of 500 mb height and vorticity map used for Question 4 in Table 2.
Questions #1 and #4 are aimed at seeing to what extent the students are willing to accept popular myths about Positive Vorticity Advection (PVA). Question #2, in effect, repeats something of the content of Question #4 in the previous pretest, but also deals with the speed of movement of convection associated with different frontal boundaries. Question #3 attempts to see if the student is aware of the nocturnal boundary layer wind maximum phenomenon.
While it is not possible to detail results over the entire period of our participation in the FFFC, we have tabulated them for the past several years. There is no easy way to display the results to questions requiring some sort of drawing or shading directly on the test pages, so the only results presented will be those which can be tabulated easily. Owing to space limitations, only selected results can be shown.
3.1. The convection pretest
Question 1 (in Table 1) is not entirely clear about whether it applies to the thunderstorm system or to its constituent cells. The most common answer (Fig. 5) suggests students tended to think of the cell, but answers of 60 min or longer indicate a substantial number of students were considering the system, not its constituent cells.
Figure 5. Histogram of answers to Question 1.
Question 2, involving the ingredients for an ordinary, non-severe thunderstorm, revealed that many students are at least able to name the ingredients described in, say, Johns and Doswell (1992): moisture, instability and lift (see Table 3 and Fig. 6). Interestingly, the students tended to consider one of the three primary ingredients most important in rough proportion to the frequency with which that ingredient was mentioned (see Fig. 7). This means that the students choose more or less equally from among the three basic ingredients. The correct answer ("all" - category 4 in Table 3) is not given very frequently. Interestingly, when "trigger" is mentioned as an ingredient, the students choose it as most important 62.5% of the time, whereas "lift" is selected as most important only 30% of the time when it is listed as an ingredient. This underscores the message that "trigger" is an inappropriate word to use in describing the lifting processes by which parcels reach their level of free convection. Its use carries with it too great an emphasis on its importance.
Figure 6. Histogram of answers to Question 2.
Figure 7. Histogram of answers to Question 3.
Table 3. Key to convective storm ingredient numbers used in Figs.
6-7.
Ingredient Ingredient
Number
1 moisture
2 instability
3 lifting mechanism
4 all three1
5 insolation
6 trigger
7 cold aloft
8 differential
heating
9 convection
10 condensation
11 light winds
12 weak cap
13 convergence
14 other
Question 4 was considered to be true by about 25% of the students. This common myth is still widely believed, although in our experience, its acceptance is declining.
Question 5 relates implicitly to the maximum updraft speeds in thunderstorms, in this instance of the severe variety. As can be seen in Fig. 8, the students tend to underestimate this speed (the correct answer is around 75 m s-1), although it appears that there is considerable guessing. It is of some interest to note that answers of "32" and "9.8" show up fairly frequently ... these apparently are values for the gravitational acceleration (g ) in either ft s-2 or m s-2, rather than a fallspeed. Such answers suggest a lack of understanding of the question, since the units for g do not match the units specifically asked for in the question.
Figure 8. Histogram of answers to Question 5.
Finally, the answers to Question 6, involving a great deal of numbers, offer considerable insight into how familiar the students are with the surface effects and environment of ordinary convection. It is impossible in this short paper to provide enough of the details to give a comprehensive view of the results. Even though there was a basic pre-conceived set of prototypical "correct" answers, some variation among the answers is not necessarily wrong. What we hoped to see was consistency; what we found was a disturbingly large range of answers.
The diagram (recall Fig. 1) is scaled so that points C-E are at 850, 500, and 200 mb, respectively. As can be seen in Fig. 9a, Fig. 9b, and Fig. 9c, we obtained quite a range, especially at the top level (E). It should be clear from the distribution of values for Level E that there were 15-20 students out of 148 total who simply failed to understand the nature of the question. Pressures at the top of a thunderstorm exceeding 800 mb are so absurd that it is easier to believe they misunderstood the question than to accept that answer as their best guess.
The distribution for Level D pressures shows that many students are not aware that 500 mb is not even halfway to the top of an ordinary thunderstorm. It is common to think of 500 mb as an "upper level," but relative to a typical thunderstorm, it is not very high.
Figure 9. Histograms of pressure at levels C, D, and E (Figs 9a-c, respectively) for Question 6.
Figure 1 was drawn so that the anvil top of the storm was roughly symmetric in the east-west direction. Therefore, a good answer would be one where the east-west wind component is small. If there is a strong wind in this situation, it must be in the meridional component. As seen in Fig. 10, there is not much consensus about the zonal wind value at Level E, but the vast majority of the answers exceed 25 knots. We interpret this to mean that the model most students in the class have of the environment for thunderstorms is most appropriate for severe thunderstorms rather than the ordinary variety.
Figure 10. Histogram of the zonal component of the answers to Level E wind part of Question 6.
If we consider the average values for the pressure, temperature, and humidity at points A-E (Table 4), they don't appear to be too bad, apart from the high pressure bias at points C-E.
Level Press. Temp. Humid. A 1004.2 69.7F 90.6 B 1002.4 84.0F 65.8 C 883.6 17.7C 61.7 D 634.5 -2.1C 42.0 E 322.6 -33.7C 39.4
However, the averages are not typical of the individual student answers. A substantial number of students either didn't answer Question 6 at all, or only gave answers to part of the table. Many answers showed lower surface pressure under the rainshaft than in the outside environment. We've already mentioned that 15-20 students clearly did not understand that points B-E constituted a vertical profile; nevertheless, they invariably showed decreasing temperatures and increasing wind speed. Some of the temperature and pressure values given were grossly inconsistent. For those students including 500 mb at some level, the temperatures occasionally were above 0deg.C, and sometimes were below -20deg.C. The high values are quite unphysical and the low values are inconsistent with an ordinary thunderstorm. Again, it appears that the only conceptual environment model many students have is a severe thunderstorm environment.
For levels above 500 mb, the temperature estimates were quite wide-ranging, with a large number giving values far too warm. If 250 or 200 mb pressures were given for Level E, the temperatures often were -20deg.C or warmer.
The average values for humidity at the surface do suggest that most students understand that the humidity in rainy areas is higher than in the nearby surroundings. However, the decrease of humidity estimates with height shown in Table 2 is another artifact of the prevalence of the severe thunderstorm environmental model, despite its inapplicability in this case of an ordinary summer thunderstorm.
Now excluding those answers showing the pressure at A to be less than or equal to that at B, the pressure difference between points A and B corresponds to the pressure excess in the thunderstorm mesohigh. As can be seen in
Figure 11. Histogram of mesohigh pressure excess values from those answers showing a positive pressure excess.
Fig. 11, while most of the pressure excess values are in the range from 0-3 mb (which is a reasonable answer for an ordinary summer thunderstorm, a large number of answers are quite high, some even absurdly so. Answers showing a lower pressure at point A than at Point B indicate the belief that a surface low is present under thunderstorms. The origins of this belief are unclear.
3.2 Meteorological analysis pretest
Most of the results of this pretest are not easily tabulated, so most results will be discussed without much documentation. For Question 1 (in Table 2), the majority of the answers indicated that PVA relates most directly to vertical motion (Fig. 12). This is consistent with one of the most common myths in operations. The correct answer is (a), wave movement.
In our presentations over the years only a few students have been able to present a coherent physical argument connecting cyclonic vorticity advection to vertical motion, in spite of the fact that many operational forecasters use this putative relationship as if they understand it. The nearly ubiquitous presence of "PVA" as an "explanation for rising motion in forecasting narratives is a monument to the persistence of mythology and lack of useful training in basic physical processes.
Figure 12. Histogram of answers to Question 1, with "0" corresponding to "none of the choices," "1" to "a," "2" to "b," "3" to "c," and "4" to "all of the choices."
For Question 2, it was found that about 30% of the students answered with something generally correct: an area or point somewhere to the northeast of the low center, on the poleward side of the warm front. Many had the point too close to the low center, which fails to account for probable movement of the system, or too much in the warm sector. The persistent myth of convection along the cold front shows up here, as well (see the discussion of Question 4 in the "Convection" Pretest).
Question 3 was "graded" such that if the wind profile suggested a weakening of the near-surface flow, a low-level but elevated wind maximum somewhere off the surface but below 2 km, and didn't change the flow much above 2 km, it was considered correct. This somewhat liberal interpretation of the answers led to about 40% correct answers. As with the connection between "PVA" and vertical motion, we have found it quite rare for the students to give a good physical explanation of the nocturnal boundary layer wind maximum.
Question 4 is designed to provide students with a direct lesson about the futility of expecting convection to be colocated with areas of "PVA." While there are times when this is true, it is also true that this colocation is not ubiquitous. Virtually all the students put the convection in regions of substantial PVA and so their answers do not match the significant convection in the examples. We are not trying to say that PVA is not at all associated with thunderstorms; we are suggesting that it is not the sole determinant of where substantial convection is likely.
Finally, there are a few observations we should make about the answers we have seen. A number of times, we have seen the use of "+/-" in a context that clearly means the student thinks it means "approximately." In the "convection" pretest, this qualifier was seen preceding the numbers assigned to the table; e.g., at point A, the surface pressure estimate occasionally was seen as something like "+/-1004 mb." This usage suggests a disturbing lack of understanding about the meaning of standard symbols.
In tabulating the results from the convection pretest, a frustratingly large fraction of the students tried to put all the temperatures in Fahrenheit, while a few put them all in Celsius. This suggests that the students are not reading the questions very carefully. It was curious to see how much error was incurred in this way; since the students do not see temperatures aloft in Fahrenheit, this certainly entailed some effort and some of the odd results may have been a result of the conversion.
It also was interesting to see how often non-standard pressure levels were chosen for the levels C-E in Fig. 1. If something unusual, like 558 mb, was chosen for, say, level D, it was going to be difficult to know what to use for a temperature value, because forecasters don't see a 558 mb constant pressure analysis of temperature very often! The students seemed to be going way out of their way to make the questions difficult.
3.3 Evolution of answers with time
Since it presently is not possible to provide direct documentation of how the answers to the pretest questions have changed over the years, we will give our subjective assessment. Overall, the answers to the pretests have not changed significantly since they first were given. The multiple choice questions appear to exhibit some modest improvement. Our subjective evaluation of this apparent improvement is that the students appear to know better which choices to make than they did early in the FFFC history. Unfortunately, an understanding of the concepts associated with the words being selected appears to be roughly as meager as when the course first was being taught.
There is little doubt of the need for extensive training in the NWS (see Doswell et al. 1981, Doswell 1986). One of the primary assumptions of the modernization program of the National Weather Service (NWS) is that by sending selected "top guns" to off-site training programs, they will take back the expertise gained and share it with their colleagues. This would solve the "distance training" problem, since the cost of providing off-site training for the entire staff of the NWS is considered too high.
Based on our experience with the students in the FFFC, it is not at all clear that the students have shared much of their newly-acquired expertise back at their local offices. Successive classes do not seem to be more and more familiar with the basic ideas we cover in our two days. Certainly, the performance on the pre-tests does not indicate much of the information is shared with on-site staff members not attending the FFFC.
There is a number of probable causes for this result: (1) it is not at all clear that the FFFC graduates are qualified in the technical aspects of training, (2) there are relatively few opportunities for sit-down training in the local offices, even if the graduates wanted to pass along their experiences and learning, (3) it is unlikely that the students have mastered the material sufficiently well to go back and act as subject matter "experts" at their offices, and (4) it appears that local management generally does not encourage FFFC graduates to share their experiences with the rest of the staff. In spite of the creation of additional courses for meteorologists at the Cooperative program for Operational Meteorology, Education, and Training (COMET -- a collaboration between the NWS and the National Center for Atmospheric Research), we do not believe that this approach to training is going to be substantive enough to correct the organic problems revealed by the student responses to our pretests.
The results of this pretesting also suggest that another basic assumption (namely, that a four-year undergraduate program gives the incoming forecaster enough basic understanding of important topics in meteorology that specialized training in forecasting [which does not now exist!] can be successful) in operational forecasting may be seriously flawed. It should be clear that successful training requires a firm base in education. However, our interpretation of the pretest results is that the universities generally are not successful in teaching even these very basic concepts in the relatively short span of four years.
For purposes of discussion, even if we assume that newly hired meteorologists somehow have the appropriate education, a substantive (i.e., six months to a year of intensive instruction and rigorous testing) course in forecaster training for every entry-level meteorologist still is not being offered. The United States continues to be the only English-speaking country in the world without such a program. The lack of such training must be at the heart of why student performance in these pretests is so disheartening.
A path being pursued by COMET to improve on-site training is the development of interactive, computer-based training modules. While these may be effective in some cases, we remain quite skeptical that they can replace the value of face-to-face experience in a training session with a human instructor.
With all of this in mind, it is not at all obvious that the NWS is going to find it easy to respond in a meaningful way to the growing threat of flash floods. Technological solutions to forecast problems are not a panacea; at some point, a substantial investment in human resources will need to be made if the flash flood problem is to dealt with effectively.
Doswell, C.A. III, 1986: The human element in weather forecasting. Nat. Wea. Dig., 11, 6-17
Doswell, C.A. III, L.R. Lemon, and R.A. Maddox, 1981: Forecaster training - A review and analysis. Bull. Amer. Meteor. Soc., 62, 983-988.
Doswell, C.A. III, and R.A. Maddox, 1986: The role of diagnosis in weather forecasting. Preprints, 11th Conf. Wea. Forecasting and Analysis (Kansas City, MO), Amer. Meteor. Soc., 177-182.
Johns, R.H. and C.A. Doswell III: Severe local storms forecasting. Wea. Forecasting, 7, 588-612.
Maddox, R.A., 1979: A methodology for forecasting heavy convective precipitation and flash flooding. Nat. Wea. Dig., 4, 30-42.
Maddox, R.A., L.R. Hoxit, C.F. Chappell, and F, Caracena, 1978: Comparision of meteorological aspects of the Big Thompson and Rapid City flash floods. Mon. Wea. Rev., 106, 375-389.
