Budget-Cutting and the Value of Weather Services



NOAA/ERL National Severe Storms Laboratory, Norman, Oklahoma


A contribution to "Forecaster's Forum" section of

Weather and Forecasting


Corresponding author address: Dr. Charles A. Doswell III, National Severe Storms Laboratory, Norman, OK 73069 E-mail: doswell@nssl.noaa.gov

NOTICE: This manuscript appeared in Weather and Forecasting , volume 13, pp. 206-212. As a U.S. Government manuscript, it is not copyrighted by the AMS; however, see their policy statement. There may be slight differences between the paper as it appears here and the final version, owing to small changes by the technical editors.

The authors discuss the relationship between budget-cutting exercises and knowledge of the value of weather services. The complex interaction between quality (accuracy) and value of weather forecasts prevents theoretical approaches from contributing much to the discussion, except perhaps to indicate some of the sources for its complexity. The absence of comprehensive theoretical answers indicates the importance of empirical determinations of forecast value; as it stands, the United States is poorly equipped to make intelligent decisions in the current and future budget situations. To obtain credible empirical answers, forecasters will need to develop closer working relationships with their users than ever before, seeking specific information regarding economic value of forecasts. Some suggestions for developing plausible value estimates are offered, based largely on limited studies already in the literature. Efforts to create closer ties between forecasters and users can yield diverse benefits, including the desired credible estimates of the value of forecasts, as well as estimates of the sensitivity of that value to changes in accuracy of the forecasts. The authors argue for the development of an infrastructure to make these empirical value estimates, as a critical need within weather forecasting agencies, public and private, in view of continuing budget pressures.

1. Introduction

In view of the rising angst over budget-cutting, we believe it to be critically important to determine quantitatively the economic value of weather forecasting services. In the past few years, the Atmospheric Environment Program (AEP) in Canada, the Bureau of Meteorology in Australia, the United Kingdom Meteorological Office, and the Meteorology Service of New Zealand have, among others, experienced unprecedented upheavals in their public forecasting services. Changes such as the effort to "privatize" and to "self-finance" forecasting services have been motivated mostly by budget-reduction drives in their respective federal governments. Although the National Weather Service (NWS) in the United States is also undergoing a major change by implementing its Modernization and Associated Restructuring (MAR) program, to date the impacts on the NWS have remained modest in terms of privatization, at least in comparison to those just mentioned. It remains to be seen how long that relatively small impact will continue.

Draconian budget cutting is, effectively, an externally-imposed constraint on public weather services. Outside of the United States, slashing of the public weather service budget has some of its origins in the inability of public weather service management officials to offer a quantitative estimate of the value of its public forecasting service. In at least one case (which is unidentified to protect the persons who made us aware of it), national-level public weather service officials were asked by policy-makers about the effects of proposed budget cuts and could not produce quantitative documentation of the potential negative impacts of the proposed reductions. This made their service an easy target in the political arena; efforts to balance national budgets eventually will force public weather services in all nations to respond to this sort of question. As it now stands, the United States is poorly equipped to make intelligent decisions in the current and future budget situations We view the inevitability of the inquiry everywhere, including the United States, to be reason enough to try to avert a possible negative outcome of a drastically reduced public forecasting service. In fact, this very issue has already come up in Congressional hearings.[1]

We also should be prepared to admit that the outcome of such an analysis might enhance the chances for a drastically reduced public sector forecasting service. After all, there is a possibility that the study could reveal that it would be more effective to "privatize" large segments of current NWS services (see Brooks et al. 1996; hereinafter BFD96). To mandate in advance the outcome of a quantitative analysis would be unscientific, however. We prefer that when the question is asked, quantitative documentation of forecast value is readily available, to prevent a potentially disastrous impact on the public forecast infrastructure. If it becomes clear afterward that such cuts have been a mistake, it will be a long, expensive process to reconstruct that infrastructure.

We note that, although our emphasis is going to be on economic value, there is a critical aspect of forecast value we are not considering; namely, that associated with reducing human casualties associated with hazardous weather. It is certain that casualty mitigation can have a substantial economic impact in its own right, but we are not prepared to consider such issues as how society values human life or the economic impact of injuries and fatalities. Budget pressure is essentially an economic fact, and so we are emphasizing the importance of assessing the economic value of weather forecasts. Decision-makers need information about economic factors in forecasting, as well as information about the efforts to reduce casualties.

We are not suggesting that casualty mitigation is irrelevant or unimportant. In fact, in BFD96, we recommend an emphasis on hazardous weather in the public weather services. However, we (like most meteorologists) are not well-versed in the literature and techniques of assessing the economic value of forecasting in terms of casualty reduction (see Fisher et al. 1989 for a summary). We believe, further, that the purely economic value of forecasts is sufficiently high that maintaining public weather services makes sense in financial terms, even disregarding contributions to public safety (and the casualty-related economics). The fact that we believe this might not carry much weight in public policy-making, however; what is needed is hard, quantitative information about economic value.

In this essay, we propose a framework for establishing the value of weather forecasting services in section 2. In section 3, we will give some examples of innovative approaches to deriving answers to the question "How valuable are these weather services?" Finally, in section 4, we offer some discussion of how this framework could be applied within the forecasting community of the United States.


2. Forecast value vs accuracy

As Murphy and Ehrendorfer (1987) and Roebber and Bosart (1996) have enunciated so clearly, there is considerable difference between forecast value and forecast accuracy. The relationship between them is not so simple as intuition might suggest. As a simple counter-example to this initial intuitive belief, consider a forecast service that is always wrong: whenever the forecast is for rain, it is dry; whenever the forecast is for warming, it gets colder; whenever the forecast calls for sun, it is cloudy. Clearly, such a forecast service is terribly inaccurate, but a reasonably intelligent user would find it quite valuable! All this intelligent user would have to do is prepare for the opposite of what the forecast calls for and then reap whatever benefits accrue. This example is rather contrived; however, as we shall explain, it is indeed possible for accuracy to increase and yet value to decline in less contrived circumstances.

A factor that makes the connection between accuracy and value so difficult to define is the cost/loss ratio.[2] That is, if the user of a forecast takes some action in response to the forecast, that action has a cost C. If the user fails to take that action, however, there may be a loss L associated with that failure to act. A simple example is a user growing crops that are sensitive to freezing. There are actions that the user can take (e.g., spraying fruit trees with water) to diminish the threat of freezing weather. These actions have a cost that a grower would not want to incur needlessly. However, failing to take those actions in a freeze means some amount of crop loss, creating a proportionate loss of income. Every user of weather information has a cost/loss ratio (C/L) and, generally speaking, that ratio differs for each user. Some users are not knowledgeable about their cost/loss ratio and so are handicapped in determining whether to take a protective action.[3]

Even when C/L is known, however, Murphy and Ehrendorfer (1987) have noted that it is still difficult to be precise about the relationship between accuracy and value. They point out that it typically is possible to obtain a single-valued relationship between accuracy and quality only when making a number of simplifying assumptions about the problem. Of particular import is the process by which forecast accuracy is specified; generally, this is not completely determined by single scalar measures of accuracy (e.g., a Brier score or a skill score).

It is through the choice of an accuracy measure that the counterintuitive result of value declining as accuracy increases can arise. Consider another simple example, this time for dichotomous (i.e., yes/no) forecasts of a dichotomous event. This problem can be described in terms of a simple 2 x 2 contingency table (Table 1).

Table 1. The contingency table for the case of dichotomous forecasts (i.e., yes/no) and dichotomous events.


















One accuracy measure is the probability of detection [POD, defined as x/(x+y) ]. Another is the false alarm ratio (FAR, defined as z/(x+z) ]. If the POD is selected as an accuracy measure, then a user whose value depends on FAR might well find value declining as POD increases, since POD can be increased simply by predicting the event more frequently, without regard for the FAR. As noted by Murphy and Ehrendorfer, no single accuracy measure can describe all aspects of accuracy,[4] even for this simple 2 x 2 case. For the same reason, no accuracy measure can be appropriate for all users. Thus, a comprehensive theoretical expression for the relationship between accuracy and value can be hard to find. Value is a complex, user-dependent function and although purely prescriptive studies (Murphy 1994) should continue, they cannot provide definitive answers.

Since statistical theory offers only limited insight into the value of weather forecasting, it is virtually inevitable that we will have to develop some sort of empirical basis for estimating the value of the forecast product (i.e., descriptive studies; Murphy 1994). Note that the value contributed by weather forecasting is not one of generating income for users. Users certainly can improve their profits by making use of weather information, but they generate income from selling their product or service. The only people who generate income from forecasts, per se, are forecasters who charge users directly for their services. Instead, users of weather information benefit from forecasts by minimizing their costs and losses on the basis of the forecasts. Instead of being able to point to income generated, we usually can only hope to offer information about resources saved. Such information often is difficult to document. In effect, we have to know what the costs and losses would have been in the absence of the forecasts, or in comparison to some other source of weather information (e.g., climatology or persistence forecasts). It is very useful, as well, to know the sensitivity of the value to the accuracy. It may be that costs and losses are very sensitive to weather, but not very sensitive to the weather forecast . For example, a hailstorm can cause tremendous crop losses, but there is very little at present that can be done to save the crops from those losses, given the relatively short lead times (generally less than one hour) for severe thunderstorm warnings. A farmer might not be able to protect the crops, but investing in crop hail insurance is a decision that must be made. That decision is not particularly sensitive to the accuracy of hailstorm forecasts, however. Deciding to buy crop hail insurance is more a matter of knowing the climatology of hailstorms than one of forecast accuracy.

Presumably, budget-cutting might lead eventually to accuracy losses, and it is important to understand how and to what extent this affects the value. Although the relationship between accuracy and value is not simple, as we have noted, it is unlikely that forecast value is going to improve for most users when forecast accuracy (no matter how this is measured, within reasonable limits) declines. If we have difficulty measuring quantitatively the precise value of the forecasts, we are further challenged to determine the tendencies in the accuracy-value relationship.

In his report on the cost-benefit of the NWS MAR, Chapman (1992) derived an estimate of the value of forecasts to users by citing a 1980 survey[5] that asked what the respondents were willing to pay for weather services. The "resultant per capita figure" (presumably, some measure of central tendency of the responses) was $24.15 per year. Through an adjustment "for changes in purchasing power and population" this base figure became $35.50 per year. The benefit of the MAR was assumed to accrue through the absence of a 5% per annum "erosion of confidence" attributed to the obsolescence of the current system if it remained unchanged.

Chapman's study also mentioned a questionnaire-based survey of "major agricultural, industrial, and commercial organizations," asking the 250 respondents to estimate their weather-related losses, as well as to decide what fraction of those weather-related losses are preventable. It is not clear from Chapman's report how or when this survey was carried out, how the results were extrapolated to the entire nation from this sample of commercial interests, or how the potential annual savings were broken down further into those due to "operational improvements" and to "scientific advances." We are not prepared to dispute Chapman's figures, but it should be clear that his report is not sufficient to address our concerns.

To obtain hard quantitative information, then, forecasters are going to have to develop working relationships with forecast users. In the National Weather Service, efforts along these lines are already underway, including the creation of Advisory groups of users. However, those efforts generally are associated with obtaining user input about forecast products. Although we certainly favor seeking user input regarding forecast products, public weather services also need to find out from their users how to measure the economic value of the forecast products.

Forecasts can be of relatively low accuracy and still have value for an intelligent user, as we will show later. Whereas it should be obvious to public policy-makers that perfect forecasts would be quite valuable for all users, such perfection is unattainable. The public perception of weather forecast accuracy typically is more negative than the objective reality of forecast verification statistics. Many weather forecast-sensitive users have learned how to take at least some advantage of existing forecasts, even with something less than perfect accuracy (as we discuss later). Historically, forecasters generally have not paid much attention to how their forecasts are being used, preferring instead to focus their attention on the meteorological aspects of the forecasting task.

It is our belief that this historical lack of attention to the economic value of forecasting is hurting the weather forecasting services (public and private) and, ultimately, the users of forecasts, as well. In the public sector, which is under considerable budget pressure, the resource base for forecasting is being threatened; this may or may not have a negative impact on forecast quality. If forecast quality is perceived by the policy-makers to be diminishing or not worth the investments, then further budget cuts are likely. Thus, a downward spiral could occur, possibly leading to an evisceration of public forecasting services (e.g., as in New Zealand).

In any case, both public and private sector forecasters can benefit from having an accurate knowledge of the value of their products, at least to a representative cross-section of important users. We certainly are not alone in these concerns. Recently, Emanuel et al. (1995) have noted that "At present, we have very little understanding of the costs and benefits of weather data and forecasting information." They recommended "... the commission of a collateral economic analysis of the costs and benefits of weather forecasting improvements ..." and "... the establishment of an independent scientific committee for providing guidance to the National Weather Service ... ."

The American Meteorological Society has established a committee on Societal Impacts and the National Center for Atmospheric Research now has a working group to address the societal impacts of weather, in collaboration with a diverse group of sponsors, including the U.S. Weather Research Program. They recently completed a workshop and a summary of that meeting[6] can be found in Pielke and Kimpel (1997). We are wholehearted supporters of these efforts. A key to their success will be multidisciplinary involvement, with economists, social scientists, private sector business, etc.

Establishment of the accuracy-value relationship is complex and begins with a clear knowledge of forecast accuracy. Unfortunately, most forecasters at present have a less-than-adequate understanding of the accuracy of their forecasts, because most forecast verification has focused on measures-oriented schemes. Again, this is outside the domain of this essay, and we have written about this elsewhere (Brooks and Doswell 1996). We note that the issue of multi-faceted accuracy measurement is touched upon by Murphy and Ehrendorfer (1987) in their discussion, and it adds considerably to the complexity of establishing the accuracy-value relationship.


3. Some approaches

A recent study by Robinson (1989) has shown that it is possible to isolate the economic impact of weather events on weather-sensitive users. However, Robinson's work stops short of finding out what the actual user costs are; rather, those costs have been estimated. Robinson's interesting study also does not assess what the impacts might have been had the forecasts not been available. There are good reasons for a meteorologist's inability to determine all the direct and indirect impacts of weather-related costs and losses for even a representative sample of users. It is also easy to understand the difficulty in estimating the user response in the absence of the forecasts. Nevertheless, these issues must be explored if we are to obtain credible information about the value of forecast products.

If establishing forecast value with some accuracy is to be accomplished, how might this be done? We believe that simply polling users with a question like "How valuable are weather forecasts to your operation?" or "How much are you willing to pay for weather information?" is not adequate. Even if users have clear subjective opinions about the value of forecasts they are receiving, this type of information is not going to be enough to satisfy the budget-cutters. What is needed is a objective, quantitative knowledge of what the negative impacts on users would be if the forecasts were not available or were reduced in accuracy. That is, we need to document the costs and losses incurred as a direct consequence of being without the forecast information and/or if the forecast accuracy declines.

At the moment, the forecasting community is only beginning to develop an infrastructure to acquire such data, so it is important to consider how to go about acquiring it. Moreover, not all business users are interested in sharing information about the quantitative value of forecasts to their operation, since they believe it gives them a competitive edge. Fortunately, some creative and innovative ideas are already in the literature to point the way, and we are going to suggest some other ideas, as well


a. Estimating impacts

As an example of what it takes to determine actual costs, Leigh (1995) has demonstrated that by working directly with a user of forecast information, it is possible to determine credible cost/loss estimates. Leigh considered the impact of terminal aerodrome forecasts (TAFs) on a single airline, at a single airport; only international flights were considered and only a single beneficial effect of TAFs on aircraft operations was included. The airline in question uses TAFs to make the decision whether or not to take on additional fuel, to be used if weather conditions might make it necessary to divert to an alternate airport. Clearly, if the extra fuel turns out to be unnecessary, this is an added cost, and the added weight contributes to increasing the fuel burn during the flight and reducing the useful payload. If the additional fuel is not aboard and the conditions at the airport become adverse, the flight must divert en route to another airport. A scenario whereby the aircraft is forced to land under adverse conditions and the landing is unsuccessful was not considered.[7] For the airline under study, the extra fuel is mandatory whenever TAFs call for weather conditions failing to meet certain criteria. If the TAF is for conditions meeting or exceeding the criteria, the decision is at the pilot's discretion. Since the number of cases when the pilot chose to carry alternate fuel loads in spite of favorable TAFs is small, Leigh assumed that the airline's pilots always follow the TAFs in that decision-making.

TAFs are notoriously difficult to accomplish successfully, and there is considerable uncertainty associated with them. Leigh's contribution is to develop a method for estimating costs and losses using uncertain forecasts within an analytic decision-making framework. The key assumption in this framework is that in the absence of TAFs, the airline would require the alternate fuel on all flights, in order to determine the expected costs in the absence of the forecasts.

Leigh finds that the value of this single application of TAFs to forecast operations, for a single airline's international flights to one airport were about $7 million Australian (~$5.5 million U.S.) yr-1. Moreover, he estimates that if TAF accuracy (according to a single measure) were to increase by 1%, the added value would be worth more than $1 million Australian yr-1. These results suggest that the uncertain TAFs are still of considerable value to aircraft operations nationwide; however, Leigh points out that there is a threshold TAF accuracy value below which they have no value in this context. Obviously, the TAF accuracy exceeds that threshold and knowledge of that threshold is of considerable importance. Since the airline pays the Australian Bureau of Meteorology for all of the forecast services it receives, it is noteworthy that this single service has a value to the airline exceeding those payments. Thus, the costs to the airline for their support of forecast services is outweighed by the benefits accrued. Leigh points out that most airlines at present are operating under policies that always require the extra fuel, irrespective of the TAFs, thereby negating the value (and benefits) offered by TAFs.


b. "Before and after" situations

In certain instances, it is possible to work with users of weather information to develop new forecasting services and therefore, after some time has elapsed from the inception of the service, it becomes relatively straightforward to determine its economic impact. That is, since there is a period without the service with which to compare, value estimates become relatively simple. An example of this can be found in the study by Anaman and Lellyett (1996), looking at a new service introduced in 1992 to benefit the Australian cotton industry. They found by surveying the users that the benefits to the cotton producers amount to about a 1% decrease in their cotton production costs, corresponding to approximately $400, 000 Australian. In view of the costs incurred by the producers (~$30, 000 Australian) for this service, this was deemed to be a favorable benefit-cost ratio. Factors exist that affect the validity of such "before and after" comparisons; for example, growing conditions could vary in a systematic way during the first few years after introducing the service, introducing a bias in the outcome. Forecast methods could be improving (or worsening) overall, affecting any comparison of the value of different weather services from one period to another. We recommend that any such comparison should be carried out in an analytic decision-making framework, comparable to that used by Leigh (1995), with an aim at establishing clearly the value and benefits of the new products.


c. "Missing forecast" situations

Another possibility for investigating value that has not been pursued, to our knowledge, is considering the impacts on users when the forecasts are missing. No public weather service would (or should) withhold forecast information deliberately as part of an experiment, but for a variety of reasons, public weather forecasts may fail to go out (e.g., communications outages), or may go out only after considerable delay. It might be possible to interact with users about how they deal with situations where the forecast is missing or delayed, and what the consequences to their operations are in those situations. There are relatively few such situations in any given year at a given office, of course, but there might be enough "missing forecast" cases nationwide to attempt to derive some useful information from this otherwise negative event. The challenge is to interact with a broad enough cross section of users to exploit the limited number of situations.


4. Discussion

Given that there is a high probability of large budget cuts being imposed on the NWS in the next 5-10 years ($37 million in fiscal year 1997), it is imperative to develop quantitative estimates of the value of forecasts and the sensitivity of that value to the quality of the forecasts. This information can be used in a variety of positive ways, not simply to argue for maintenance of the status quo. Obviously, forecaster-user collaboration can be beneficial to both sides.

The important issue is the development of effective, meaningful bases for making cost-cutting decisions. If imposition of budget cuts is inevitable, then it behooves public decision-makers to be selective about applying those cuts, rather than simply imposing uniform budget reductions across the board. If the value of the forecasts suffers as a result of quality changes imposed by resource reductions, concern for public welfare demands that the least valuable components of the forecasting system need to be the primary targets of the budget-cutters. Naturally, in this area, casualty reduction might take precedence over pure economic value; we are not prepared to make this sort of decision, but it is typical of the problems faced by public decision-makers.

In anticipation of cost-cutting exercises to come, we want to emphasize the multidimensional nature of forecast quality. As of this writing, there is a serious exercise underway to determine the impact of removing half of the nation's rawinsondes, as a way to save resources. If the measure of the impact is the 500-mb anomaly correlation score, the assessment might have a much different implication than if the measure is the equitable threat score for precipitation or the area CSI (Critical Success Index) for tornado watches. The impact of an observing system can be global, national, regional, and local, simultaneously. Who is to decide what is the most important? If the proposed changes affect the global or national forecasts very little but have a huge impact on the mesoscale or convective storm scale forecasts, how is the decision to be made? Murphy and Winkler (1987) and Brooks and Doswell (1996) have emphasized the pitfalls of single scalar measures for forecast quality evaluation. Exercises to measure impacts of cost-cutting proposals need to consider the full range of potential adverse effects.

Moreover, it has been argued elsewhere (see BFD96) that a key issue is forecaster performance on the relatively few days when the weather attains hazardous proportions. The impact on forecast quality of a hypothetical reduction in rawinsonde flights on a day like the so-called Superstorm of March 1993 might be much larger than on a set of more or less "average" weather days. Day-to-day performance is certainly a substantial issue, but it would be a serious mistake not to consider the impacts of any particular budget choice on a set of days when significant hazardous weather developed. History offers some examples. As a cost-saving measure, routine ship rawinsondes in the northern Pacific were terminated in 1981, in spite of the study by Spagnol et al. (1980) showing a negative impact. In their paper, Spagnol et al. (1980) indicated that some measures might be useful to compensate for the loss of the soundings. It seems now that those compensating procedures may not have been successful; Lord (1996) has shown recently that there can be considerable numerical weather prediction model forecast sensitivity to the initial conditions in just this northeastern Pacific Ocean area where the ship formerly took observations.

Cost-cutting measures can take a variety of forms, some of which include large reductions in staffing. As noted in BFD96, it is possible to envision revised structures in the public forecasting system that would have reduced staffing but with a considerably changed overall character to the public forecasting task and the skills needed by the staff. In view of the unknown nature of the effects of budget reductions even within the present system, it is difficult at present to anticipate what the effect on forecast accuracy of any proposed changes might be, including the NWS's MAR. Any accuracy changes are likely to be important economically, but in the absence of documented evidence for expected changes in either accuracy or value, it is hard to imagine how our policy-makers can make intelligent decisions in the current and future budget situations. Impacts on the quality of the public forecasting service can have a large "ripple" effect since, ultimately, weather forecasting is a primary reason for supporting much of the basic and applied meteorological research in all sectors. This is a situation we consider to be important to the entire meteorological community and, ultimately, for the nation as a whole.

The significance of this issue seems to call to a wide range of interests to support the burgeoning efforts already underway. We will need a coalition to develop among meteorologists (forecasters and researchers, public and private), users, and representatives from related fields (economics, policy makers, etc.). We believe that, although the entire meteorological community ought to be concerned with the outcome of that decision-making process, we cannot and should not try to do this in meteorological terms only. We meteorologists tend to be poorly-versed in techniques outside our particular domain of expertise that are going to be needed to carry out this effort.

Public policy-makers must make difficult economic decisions that include issues of human safety, as well as purely economic factors. We hope those decisions will be made in the United States with as comprehensive a knowledge of the economic impacts of weather forecasts as possible, rather than without that quantitative information.

Acknowledgments. Our views on this subject have been influenced heavily by discussions with our colleagues; notably, Drs. R. Davies-Jones, J. M. Fritsch, J. Cortinas, R. Maddox, as well as many others too numerous to mention. Constructive input from an anonymous reviewer, as well as from Drs. Roger Pielke, Jr. and James F. Kimpel has been very beneficial. The lead author is grateful to the Australian Bureau of Meteorology for supporting a visit to the Sydney office, where the articles about the economic value of forecasts were brought to his attention.


Anaman, K. A., and S. C. Lellyett, 1996: Assessment of the benefits of an enhanced weather information service for the cotton industry in Australia. Meteor. Appl., 3, 127-136.

Brooks, H. E., and C. A. Doswell III, 1996: A comparison between measures-oriented and distributions-oriented approaches to forecast verification. Wea. Forecasting, 10, 288-303.

______, J. M. Fritsch, and C. A. Doswell III, 1996: The future of weather forecasting: The eras of revolution and reconstruction. Preprints, 15th Conf. on Weather Analysis and Forecasting, Norfolk, VA, Amer. Meteor. Soc., 523-526.

Chapman, R. E., 1992: Benefit-cost analysis for the modernization and associated restructuring of the National Weather Service. U. S. Department of Commerce, NOAA/National Weather Service, 1325 East-West Highway, Silver Spring, MD 20910, 118 pp.

Emanuel, K., and Coauthors, 1995: Report of the first prospectus development team of the U. S. Weather Research Program to NOAA and the NSF. Bull. Amer. Meteor. Soc., 76, 1194-1208.

Fisher, A., L. G. Chestnut, and D. M. Violette, 1989: The value of reducing risks of death: A note on new evidence, J. Policy Anal. Manage., 8, 88-100.

Leigh, R. J., 1995: Economic benefits for terminal aerodrome forecasts (TAFs) for Sydney Airport, Australia. Meteor. Appl., 2, 239-247.

Lord, S. J., 1996: The impact on synoptic-scale forecasts over the United States of dropwindsonde observations taken in the northeast Pacific. Preprints, 11th Conf. on Numerical Weather Prediction(Norfolk, VA), Amer. Meteor. Soc., 70-71.

Murphy, A. H., 1994: Assessing the economic value of weather forecasts: An overview of methods, results, and issues. Meteor. Appl., 1, 69-73.

______, and M. Ehrendorfer, 1987: On the relationship between the accuracy and value of forecasts in the cost-loss ratio situation. Wea. Forecasting, 2, 243-251.

______, and R.L. Winkler, 1987: A general framework for forecast verification. Mon. Wea. Rev., 115, 1330-1338.

 Pielke, R. A., Jr., and J. Kimpel, 1997: Societal aspects of weather: Report of the sixth prospectus development team of the U. S. Weather Research Program to NOAA and NSF. Bull. Amer. Meteor. Soc., 78, 867-876.

Roebber, P. J., and L. F. Bosart, 1996: The complex relationship between forecast skill and forecast value: A real-world analysis. Wea. Forecasting, 11, 544-559.

Robinson, P. J., 1989: The influence of weather on flight operations at the Atlanta Hartsfield International Airport. Wea. Forecasting, 4, 461-468.

Spagnol, J., M. Horita, and P. Haering, 1980: Analyses of the eastern Pacific without Ship Papa data. Preprints, 8th Conf. onWeather Forecasting and Analysis, Denver, CO, Amer. Meteor. Soc., 150-155.