**5. Techniques used to improve the quality control procedures during the extreme events.**

Quality of data during the extreme events such as strong cold fronts and hurricanes may de‐ crease resulting in a higher number of "true" outliers than that during the normal climate conditions. (28) carefully analyzed the sample examples of these extreme weather conditions to quantitatively demonstrate the causes of the outliers and then developed tools to reset the Type II error flags. The following discussion will elaborate on this technique.

#### **5.1. Relationship between interval of measurement and QA failures**

Fig. 1 shows the result for maximum temperature at Silver Lake Brighton, Utah. The IDW approach results in a large bias. The best fit line for IDW indicates the estimates are system‐ atically high by over 8 F (8.27); the slope is also greater than one (1.0684). When the best fit line for IDW estimates was forced through zero, the slope was 1.2152. On the other hand the estimates from the SRT indicate almost no bias as evidenced by the best-fit slope (0.9922).

For the minimum temperature estimates a similar result was found (Fig. 2). The slope of the best-fit line for the SRT indicates an unbiased (0.9931) while the slope for the IDW estimates indicates a large bias on the order of 20% (slope = 1.1933). The reader should note the SRT unbiased estimators are derived every 24 days (see ) and that applying the SRT only once

**Figure 2.** The results of estimating minimum temperature at Silver Lake Brighton, UT for both the IDW and the

**5. Techniques used to improve the quality control procedures during the**

Quality of data during the extreme events such as strong cold fronts and hurricanes may de‐ crease resulting in a higher number of "true" outliers than that during the normal climate conditions. (28) carefully analyzed the sample examples of these extreme weather conditions to quantitatively demonstrate the causes of the outliers and then developed tools to reset the

Type II error flags. The following discussion will elaborate on this technique.

for the entire period will degrade the results shown (7).

10 Practical Concepts of Quality Control

SRT methods.

**extreme events.**

Analyses were conducted to prepare artificial max and min temperature records (not the measurements, but the values identified as the max and min from the hourly time series) for different times-of-observation from available hourly time series of measurements. The ob‐ servation time for coop weather stations varies from site-to-site. Here we define the AM sta‐ tion, PM station, and nighttime station according to the time of observation (i.e. morning, afternoon-evening, and midnight respectively). The cooperative network has a higher num‐ ber of PM stations but AM measurements are also common; the Automated Weather Data Network uses a midnight to midnight observation period.

The daily precipitation accumulates the precipitation for the past 24 hours ending at the time of observation. The precipitation during the time interval may not match the precipi‐ tation from nearby neighboring stations due to event slicing, i.e. precipitation may occur both before and after a station's time of observation. Thus, a single storm can be sliced in‐ to two observation periods.

**Figure 3.** Example time intervals for observations at Mitchell, NE (after 28).

The measurements of the maximum and the minimum temperature are the result of making discrete intervals on a continuous variable. The maximum or minimum temperature takes the maximum value or the minimum value of temperature during the specific time interval. Thus the maximum temperature or the minimum temperature is not necessarily the maxi‐ mum or minimum value of a diurnal cycle. Examples of the differences were obtained from three time intervals (see Fig 3) after28)). The hourly measurements of air temperature were retrieved from 1:00 March 11 to 17:00 March 13, 2002 at Mitchell, NE. The times of observa‐ tion are marked. Point *A* shows the minimum air temperature obtained for March 11 for AM stations, and *B* is the maximum temperature obtained for March 13 at the PM stations. The minimum temperature may carry over to the following interval for AM stations and the maximum temperature may carry over to the following interval for PM stations. We have therefore marked these as problematic in Table 4to note that the thermodynamic state of the atmosphere will be represented differently for AM and PM stations. Through analysis of the time series of AM, PM and midnight calculated from the high quality hourly data we find that measurements obtained at the PM station have a higher risk of QA failure when com‐ pared to neighboring AM stations. The difference in temperature at different observation times may reach 20 o F for temperature and several inches for precipitation. Therefore the QA failures may not be due to sensor problems but, to comparing data from stations where the sensors are employed differently. To avoid this problem AM stations can be compared to AM stations, PM stations to PM stations, etc. Note this problem will be solved if moderniza‐ tion of network provides hourly or sub-hourly data at most station sites.

precipitation events, in which only a single station received significant precipitation. Higher precipitation entries occurring in isolation are more likely to be identified as potential outli‐ ers. These problems were expected to be avoided by examining the precipitation over larger

Toward a Better Quality Control of Weather Data

http://dx.doi.org/10.5772/51632

13

No significant relationship is found between the topography and the fraction of flagged re‐ cords. Some clusters of stations with high flag frequency are located along the mountains; however, other mountainous stations do not show this pattern. Moreover, some locations with similar topography have different patterns. For the State of Colorado, a high fraction of flags occurs along the foothills of the Rocky Mountains where the mountains meet the high plains. A high fraction was also found along interstate highways 25 and 70 in east Colorado. These situations may come about because the weather stations were managed by different organizations or different sensors were employed at these stations. These differences lead to

**Figure 4.** Time series of Stratton and a neighboring station during 2002 droughts. a) The daily time series of Tmax for

Stratton and Stratton AWDN station (a058019). b) Hourly time series at Stratton AWDN station. (after 28).

intervals, e.g. summing consecutive days into event totals.

possible higher fraction of flagged records in some areas.

**5.3. 2002 drought events**


**Table 4.** Time interval and possible performance of three intervals of measurements.

#### **5.2. 1993 floods**

Quality control procedures were applied to the data for the 1993 Midwest floods over the Missouri River Basin and part of the upper Mississippi River Basin, where heavy rainfall and floods occurred (28). The spatial regression test performs well and flags 5~7 % of the data for most of the area at *f*=3. The spatial patterns of the fraction of the flagged records do not coincide with the spatial pattern of return period. For example, the southeast part of Ne‐ braska does not show a high fraction of flagged records although most stations have return periods of more than 1000 years. While, upper Wisconsin has a higher fraction of flagged records although the precipitation for this case has a lower return period in that area.

The analysis shows a significantly higher fraction of flagged records using AWDN stations in North Dakota than in other states. This demonstrates that the differences in daily precipi‐ tation obtained from stations with different times of observation contributed to the high fraction of QA failures. A high risk of failure would occur in such cases when the measure‐ ments of the current station and the reference station are obtained from PM stations and AM stations respectively. The situation worsens if the measurements at weather stations were obtained from different time intervals and the distribution of stations with different time-ofobservation is unfavorable. This would be the case for an isolated AM or PM station.

Among the 13 flags at Grand Forks, 9 flags may be due to the different times of observation or perhaps the size and spacing of clouds (28). Four other flags occurred during localized precipitation events, in which only a single station received significant precipitation. Higher precipitation entries occurring in isolation are more likely to be identified as potential outli‐ ers. These problems were expected to be avoided by examining the precipitation over larger intervals, e.g. summing consecutive days into event totals.

#### **5.3. 2002 drought events**

maximum temperature may carry over to the following interval for PM stations. We have therefore marked these as problematic in Table 4to note that the thermodynamic state of the atmosphere will be represented differently for AM and PM stations. Through analysis of the time series of AM, PM and midnight calculated from the high quality hourly data we find that measurements obtained at the PM station have a higher risk of QA failure when com‐ pared to neighboring AM stations. The difference in temperature at different observation

failures may not be due to sensor problems but, to comparing data from stations where the sensors are employed differently. To avoid this problem AM stations can be compared to AM stations, PM stations to PM stations, etc. Note this problem will be solved if moderniza‐

Time intervals (e.g.) ~7:00 ~ 17:00 ~midnight

Precipitation Good Good Good

records although the precipitation for this case has a lower return period in that area.

observation is unfavorable. This would be the case for an isolated AM or PM station.

Among the 13 flags at Grand Forks, 9 flags may be due to the different times of observation or perhaps the size and spacing of clouds (28). Four other flags occurred during localized

The analysis shows a significantly higher fraction of flagged records using AWDN stations in North Dakota than in other states. This demonstrates that the differences in daily precipi‐ tation obtained from stations with different times of observation contributed to the high fraction of QA failures. A high risk of failure would occur in such cases when the measure‐ ments of the current station and the reference station are obtained from PM stations and AM stations respectively. The situation worsens if the measurements at weather stations were obtained from different time intervals and the distribution of stations with different time-of-

Quality control procedures were applied to the data for the 1993 Midwest floods over the Missouri River Basin and part of the upper Mississippi River Basin, where heavy rainfall and floods occurred (28). The spatial regression test performs well and flags 5~7 % of the data for most of the area at *f*=3. The spatial patterns of the fraction of the flagged records do not coincide with the spatial pattern of return period. For example, the southeast part of Ne‐ braska does not show a high fraction of flagged records although most stations have return periods of more than 1000 years. While, upper Wisconsin has a higher fraction of flagged

tion of network provides hourly or sub-hourly data at most station sites.

Maximum temperature Problematic

**Table 4.** Time interval and possible performance of three intervals of measurements.

Minimum temperature Problematic

F for temperature and several inches for precipitation. Therefore the QA

**AM station PM station Nighttime station**

**(AWDN)**

times may reach 20 o

12 Practical Concepts of Quality Control

**5.2. 1993 floods**

No significant relationship is found between the topography and the fraction of flagged re‐ cords. Some clusters of stations with high flag frequency are located along the mountains; however, other mountainous stations do not show this pattern. Moreover, some locations with similar topography have different patterns. For the State of Colorado, a high fraction of flags occurs along the foothills of the Rocky Mountains where the mountains meet the high plains. A high fraction was also found along interstate highways 25 and 70 in east Colorado. These situations may come about because the weather stations were managed by different organizations or different sensors were employed at these stations. These differences lead to possible higher fraction of flagged records in some areas.

**Figure 4.** Time series of Stratton and a neighboring station during 2002 droughts. a) The daily time series of Tmax for Stratton and Stratton AWDN station (a058019). b) Hourly time series at Stratton AWDN station. (after 28).

Instrumental failures and abnormal events also lead to QA failures. Fig. 4 shows the time series of the Stratton Station in Color adooperated as part of the automated weather net‐ work. This station has nighttime (midnight) readings while all of the neighboring sites are AM or PM stations. Stratton thus has the most flagged records in the state (6): the highlight‐ ed records in Fig. 4 were flagged. We checked the hourly data time series to investigate the QA failure in the daily maximum temperature time series for the time period from April 20 to May 20, 2002. No value was found to support a Tmax of 88 for May 6 in the hourly time series, thus 88 o F appears to be an outlier. On May 7 a high of 85 o F is recorded for the PM station observation interval, in which the value of the afternoon of May 6 is recorded as the high on May 7. The 102 o F observation of May 8 at 6:00 AM appears to be an observation error caused by a spike in the instrument reading. The observation of 93 o F at 8:00 AM May 17 is supported by the hourly observation time series (see Fig. 4 (b)) and is apparently asso‐ ciated with a down burst from a decaying thunderstorm.

!

!

!

!

!

!

!

!

!

!

!

!! ! !

!

!

!

! !

!

28

!

!

!

!

!

!

!

!

!

!

! !

!

!

!

! !

!

! ! !

!

!

!

! !

!

! !! !

!

!

!

!

!

!

!

! !

!

!

!

!

! !

!

!

! ! !

!

27

!

!!

!

! ! ! ! ! !

!

!

!

!

! !

! !

!

!

!

!

!

!

! !

!

!

!

! !

!

!

!

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

! !

!

!

!

! ! ! !

!

!

!

!

!

!

! ! !

! !

!

!

!

!

!

! !

!

! ! ! !

!

!

!

! ! ! !

!

25

!

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

! ! !

!

! !

!

28

!

!

!

!

!

!

!

!

!

! !!

!

!

! !

!

! !

!

!

!

!

!

!

!

! !

!

!

! ! !

!

!

!

!

!!

! ! !

!

!

!

!

!

!

!

!

27

!

! ! !

! ! ! ! ! !

!

!

! ! !

! ! ! !

!

! ! !

!

!

!

!

! ! ! ! ! !

!

!

!

!

! !

! !

!

!

!

!

25

! !

!

!

!

! !

!

! ! !

!

!

!

!!

!

!

!

!

!

!

! !

!

!

28

!

25

!

!

!

!

!

!

!

!

!

!

! !!

!

!

!

!

!

!

! !

!

!

! !

!

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

! !

! !

!

!

!

!

!

!

!

! ! ! !

!

!

!

!

! !

!

!

!

! ! ! !

!

! !

!

!

!

!

!

! !

!

!

!

!

!

!!

!

!!

!

! !

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

! ! !

!

!

!

!!

!

! !

! ! ! ! ! !

27

!

! ! ! ! ! !

!

!

! ! !

!

!

!

! ! ! ! ! !

!

!

!

! !

! ! !

! !!

!

!

! ! !

!

! !

!

!

!

!

!

!

!

!

!

!

28

!

!

!

!

!

!

!

!

!

!

!

!

!

! !

!

!

! !

!

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

! ! !

! !

!

!

!

!

!

!

!

!

! ! ! !

!

!

!!

!

!

! !

!!

!

!

!

! !

!

!

! ! !

!

! !

! !

!

! !

!

! !

!

!

!

!

!!

!

!

!

!

!

! !

!

!

! !

!

!

!

!

!

! !!

!

!

!

!

! !

!

! ! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! !

!

27

! !

! !

! ! ! ! ! !

!

! ! ! !

!

!

!

! ! ! ! ! !

!

!

!

!

! !

! ! !

!

!

!

! ! ! ! ! !

! !

!! ! !

!

!

!

!

!

!

!! !

! !

!

!

!

! !

!

! !

!

! !

!

!

!

!

. **Number of flags**

!

!

!

! ! !

!

!

! ! !

!

!

!

!

!

!

!

!

! !

! ! ! !

!

!

!

! !

!

! ! ! !

!

!

!

! !

!

!

! ! !

! !

!

! !

! !

!

!

! !

!

!

!

! ! ! !

!

! !

!

!

!

!

!

! !

! ! ! !

!

! !

! ! ! !

> 25

!

!

!

!

!

! !

!

! !

and tropical storm events. These differences are:

!

!

!

! !

!

!

!

!

!

!!

!

!

!

!

!

!

!

! ! ! ! ! !

!

!

!

!

!

! !

!

!

!

!

!

!

! !

!

!

!

! ! !

!

!

! !

!

!

!

!

! !

24

!

!

!

!

23

**Figure 5.** Daily weather maps and spatial pattern of flagged records for 1992 Andrew Hurricane events. (after 28).

The spatial patterns of flagged records are significant for both the spatial regression test of the cold front events and the tropical storm events. However, most of these flagged records are type I errors, thus we tested a simple pattern recognition tool to assist in reducing these flags. Differences still exist between the distribution patterns of the flagged records for the cold front event and the tropical storm events due to the characteristics of cold front events

**•** Cold fronts have wide influence zones where the passages of the cold fronts are wider and the large areas immediately behind the cold front may have a significant flagged frac‐

!

!

!

!

!

!

!

!

!

!

!

!

! ! ! ! ! !

!

24

!

!

!

! ! !

! !

!

!

! !

!

!

!

!

!

! !

! !

! !

!

! !

! !

! ! !

!

!

!

!

!

!

! !

!

!

!

!

!

!

! !

!

!

! !

!

!

! !

! ! ! !

24

!

!

!

! !

!

!

!

!

!

!

!

! !

!

! ! ! ! ! ! ! ! ! !

!

!

! !

!

!

! ! ! !

!

!

! ! ! ! !

!

!

! ! ! ! ! ! ! ! ! ! ! !

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

!

! !

! !

!

!

!

! ! !

!

! ! ! ! ! !

!

!

!

!

!

!

! !

!

!

!

!

!

!

!

24

!

! !

!

!

!

!

!

!

!

!

!

!

!

! !

!

! !

!

!

!

! ! ! !

!

! ! !

!

!

!

24

!

!

!

!

!

! ! ! !

!

! !

!

!

!

!

!

!

! !

!

! ! !

!

! !

!

!

!

! !

!

!

!

!

!

!

!

!

!

! !

!

!

! !

!

!

!

!

!

!

! ! !

!

! !

!

! ! ! !

!

!

!

!

!

! !

!

August 23 August 24

! ! ! ! !

! !

!

!

!

! !

!

! !

!

!

!

! ! ! ! !

!

! !

!

!

!

!

! ! ! !

! !

!

!

! !

!

! !

! !

! !

! ! !

!

! !

!

!

!

! ! ! ! ! !

!

!

!

!

! ! !

!

!

!

!

! !

!

!

!

! !

!

!

!

! !

! ! !

!

!

!

!

!

! !

!

!

!

!

! !

!

!

!

!

!

!

!

!

! ! ! ! ! !

!

!

!

!! ! ! !

! ! !

!

! !

!

!

!

!

!

!

! !

!

!

!

! !

! !

!

! !

!

!

!

August 25 August 26

! !

! ! !

!

! ! ! !

!

!

!

!

!

! !

!

!

!

!

!

!

!

!

!

!

! !

!

! !

!

!

!

!

!!

!

! !

!

! ! !

!

!

!

! ! !

!

!

!

24

!

!

!

!

! ! !

!

! !

! ! ! !

!

!

!

!

!

! !

!

!

! !

! !

! ! !

! ! !

!

! !

!

!

! ! ! ! ! !

!

!

!

!

August 27 August 28

!

!

!

!

! !

! ! !

!

! ! !

!

! ! !

!

!

!

!

!

! ! !

!

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

!

! ! ! !

!

!

!

!

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

! !

! !

! !

!

! ! !

!

!

!

! !

! !

! ! !

!

!

! ! !

!

! ! ! ! ! ! !

!

!

!

! !

!

! !

! !

!

!

!

! !

!

!

! !

!

!

!

! ! !

!

!

!

26

!

!

!

!

!

!

!

!

! !

! ! ! !

!

! !

! !

! ! ! ! ! ! !

! ! !

!

!

!

! !

! ! !

!

!

!

!

! !

! !

!

!

!

!

!

! ! !

!

! ! !

! !

!

!

! !

!

!

!

!

!

!

! !

!

! ! ! !

!

! ! !

!

! !

!

! !

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! !

! !

!

!

!

!

!

!

!

!

!

! ! !

! ! ! ! ! !

!

!

!

! ! !

! !

! !

! ! !

! ! ! !

! ! !

!

!

!

!

!

! ! !

! ! ! ! ! !

!

!

!

! !

! ! !

! !

! ! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! !

! ! ! !

! ! !

!

! !

!

!

! !

!

!

! !

! !

! !

! ! !

!

!

!

!

! ! ! ! !

! !

! !

! !

!

!

!

!

!

!

! !

!

!

!

!

!

!

!

!

! ! ! ! ! !

!

!

!

! !

!

!

! ! !

!

!

!

!

!

! ! ! ! !

! ! !

!

!

!

!

!

!

! !

!

!

!

!

!

!

! ! !

!

!

! !

!

! ! !

!

!

!

! ! ! !

!

!

!

! ! !

!

! ! !

!

!

!

! ! !

!

!

!

!

!

! !

!

!

! !

!

!

!

!

! ! ! ! ! !

!

!

26

!

!

! ! !

!

! ! ! !

!

!

!

!

!

!

!

!

!

!

! !

! !

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

! !

!

!

!

!!

!

!

!

!

!

!

!

!

! ! !

!

!

! !

! ! !

! ! !

!

!

!

!

!

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

!

!

!

!

!

!

! !

!

!

!

! !!

!

!

!

!

! !

! ! ! ! !

!

!

!

!

!

!!

! !

!

!

!

!

!

!

!

! ! ! ! ! !

!

24

!

!

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! ! ! !

!

!

!

!

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

! !

!

!

! !

!

!

!

! ! !

!

!

! !

!

!

!

!

!

!

!

!

!

!

! !

!! ! ! ! !

!

!!

!

! ! !

!

!

!

24

!

!

! ! ! !

!

!

!

!

!

!

!! !

!

!

!

! !

!

!

! ! ! !

!

!

!! ! !

!

!

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

!

! !

!

!

!

!

! ! !

!

!

!

!

!

!

!

!

!

! !

!

! !

!

!

!

!

23

!

!

!

!

!

!

!

!

!

! ! ! !

!

!

!

!

!

! !

!

!

! !

!

!

!

! !

!

! ! !

!

!

!

!

!

!

! ! !!

!! ! !

!

!

!

! !

!

28

!

!

!

!

!

!

!

!

!

!

! !!

!

!

!

! !

!

! !

!

! ! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! !

!

!

! ! !

!

27

! ! !

!

! ! ! ! ! !

!

!

!

!

! !

! !

!

!

!

!

!

!

!

!

!

!

! !

!!

! ! ! ! !

!

!

!

!

!

! !

! !! !

!

!

!

!

!

!

!

!! !

!

! ! !

!

!

!

!!

!

27

!

! ! !

!

!

!

!

!

!

!! ! !

!

!

28

!

!

!

!

!

!

!

!

!

!

!! !

! !

!

!

!

! !

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

! !

!

!

!

! ! ! !

!

!

!

!

!

!

! ! !

! !

!

!

!

!

! !

!

!

!

! ! ! !

!

!

!

!

!

!

25

!

!

!

!

! !

!

! !

!

! ! !

!

!

!

!

!!

!

!

!

!

!

!

!

!

!

!!

!

! !

! ! ! ! ! !

!

!

!

!

!

! ! ! ! ! !

!

!

!

! !

! ! !

!

!

!

! ! ! !

!

!

!

!

!

!

!

28

! ! !

!

!

!

!

!

!

!

!

! !

!

!

! !

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

! !

!

!

!

! ! ! !

!

!

!

!

!

! ! !

! !

!

!

!

!

! !

!

!! !

!

!

! !

!

!

! ! !

!

! !

!

!

!

!

! ! ! !

!

!

! !

!!

!

! !

!

! !

!

!

!

! ! ! ! !

!

! !

!

!

!

!

!

! ! ! ! !

!

! !

! !

!

!

! !

24

!

! !

! !

!

! !

!

!

!

! ! !! !

!

! !

!

! ! ! !

!

!

!

!

!

!

! !

!

!

!

! ! ! ! !

!

!

! !

!!

! !

!

!

!

!

!

!! !!

!

!

!

!

25

!

!

!

!

!

! !

!

!

!

!

! ! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

!

!

!

!

!

!

!

!

!

27

!

! ! !

! !

! !

! ! ! ! ! !

!

!

! ! !

! ! !

!

!

!

!

! ! ! ! ! !

!

!

!

! !

!

!

!

! ! !

! ! ! ! ! !

!

!

!

! ! ! ! !

!

!

! !

!

!

!

!

!!

!

!

!

!

!

!

!

!

!

!

!!!

! !

!

!

!

! !

!

! !

!

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

!

! ! ! !

!

! !

!! !

!!

!

!

! !

!

! ! ! !

!

!

!

!

! !

!

!

! ! !

! !

!

!

!

!

! ! ! !

! !

!

! !

! !

!

!

! !

! !

!

!

!

!

! !

!

!

! !!

! ! !! !

!

! !

!

!

! ! ! !

> 25

!

!

! !

!

! ! !

!

!

!

!

!

!

!

!

! ! !

!

!

28

! !

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

!

! ! !

!

!

!

!

!!

!!

!

!

!

!

!

!

!

!

!

!

! !

27

! ! ! ! ! !

!

!

!

!

! ! ! ! ! !

!

!

!

! !

! !

!

!

!

! ! !

! ! ! ! ! !

!

! !

!

!

!

! !

!

!

!

!

!!

!

!

!

!

!

! !

!

!

!

!

!

!

!

! !

!

!

!

!

! !

!

! !

!

!

! !

!

!

!

!

!

500 250 0 500 Kilometers

!

!

! ! !

!

!

!

!

!

! ! ! !

!

! !

!

!

!

! !

!

!

!

! !

!

!

! ! !

!

! !

!

!

!

! !

! ! !

!

!

!

! !

! !

! !

!

!

! ! !

!

!

!

!

! !

!

!

!

! ! ! ! !

!

! ! ! !

!

25

!

!

!

!

!

! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! !

!

!

!

!

!

! !

!

!

!

!

!

! !

!

!

! !

!

!

!

! ! !

!

!

!

! !

!

!

!

!

!

!

!

24

!

!

!

!

23

!

! !

!

!

!

!

!

!

!

!

! ! ! ! ! !

!

24

!

!

!

!

!

!

!

!

!

!

!

23

23

!

!

!

!

! ! !

!

!

!

! !

!

!

! !

!

!

!

!

!

!

!

! !!

!

! !

!

!

!

!

!

! !

!

! ! !

!

!

!

! ! ! ! ! ! ! ! ! ! !

> ! !

!

!

!

! !

!

!

! !!

!

!

! !

!

!

! !

! ! ! !

24

!

!

!

! !

!

!

!

!

!

!

! ! ! ! !

! !

! !

! ! !

! ! !

!

!

! ! !

!

! !

! ! ! !

!

!

!

! !

!

!

! !

!

!

!

!

! ! ! !

!

!

!

!

! ! ! !

24

!

!

!

!

!

! !

!

!

!

!

!

!

! !

! !

!

!

!

!

!

!

!

! ! !

!

!

!

!

!

!

!

! !

!

!

! !

!

!

!

! ! ! ! ! !

!

24

!

!

!

! ! !

! !

!

!

!

! !

!

! ! ! ! ! !

!

!

! !

!

!

!

!

!

!

!

!

!

! !

! !

! ! !

!

! ! ! !

!

!

!

!

!

! !

! !

! !

!

!

!

! !

!

!

!

!

!

!

!

! !

!

!!

! !

!

!

!

!

! ! ! !

!

!

!

!

! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! !

!

!

!

!

!

! !

!

!

!

!

!

!

! !

! !

!

!

!

! ! !

!

!

!

! !

!

!

!

!

!

!

!

24

!

!

!

!

23

!

!

!

!

!

!

!

!

!

! !

!! ! ! ! !

!

! !

!

!

! ! ! !

!

!

!

!

!

! ! ! ! ! !

!

!

!

!

! !

!

!

!

!

!

!

! !

! !

!

!

!

!

! ! !

!

Toward a Better Quality Control of Weather Data

!

! !

!

!

!

!

!

!

!

!

!

!

!

http://dx.doi.org/10.5772/51632

15

!

!

!

!

!

!

!

!

!

! !

!! ! ! ! !

!

! ! ! ! ! ! ! ! ! !

! !

!

!

! ! ! !

!

!

!

!

!

!

!

! !

!

! ! ! !

!

!

!

!

! ! !

!

! !

!

! ! ! !

!

!

!

!

! !

!

! !

! !

! ! !

! ! !

!

! !

!

!

!

! ! ! ! ! !

!

!

!

!

!

! ! !

!

!

! !

!

!

!

!

!

! ! ! ! ! !

!

!

!

!

!

!

! ! ! !

!

! ! !

!

!

!

!

!

!

! !!

!

!

!

!

!

!

!

! !

!

! ! !

! !

!

!

!

!

!

!

!

! !

!

!

! !

!

!

!

! !

! ! !

!

!

!

!

!

!

! ! !

!

!

!

!

!

! !!

!

! !

!

!

! !

!

!

!

!

!

!

!

! ! !

!

! !

!

! !

!

!!

! ! ! !

!

!

!

!

!

! !

!

! !

!

!

! !

! !

! ! !

! ! !

!

! !

!

!

! ! ! ! ! !

!

!

!

!

!

!

!

!

!

! ! !

! !

!

!

!

! ! ! ! ! !

!

!

!

!

!

!

!

!

! !

! ! !

! !

!

!

!

!

!

!

!

! !

!

! !

!

!

!

! !

! ! !

!

!

!

!

!

!

!

! !

!

!

! ! !

!

!

!

24

!

!

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

!

! ! !

!

! !

! ! ! !

!

!

!

!

!

!

!

!

! !

!

!

! !

! !

! ! !

! ! !

!

! !

!

!

!

! ! ! ! ! !

!

!

!

!

!

! !

!

!

!

! !

! ! !

!

!

!

!

!

!

! !

!

!

! !

!

! !

!

! !

! !

!

! !

! !

!

!

! !

! !

! !

!

! !

! !

! ! !

!

!

! ! !

!

!

! ! !

!

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

!

!

!

! !

! !

!! !

!

!

!

!

!

! !

!

!

!

!

!

!

26

!

!

!

!

! !

!

!

!

!

!

! !

! ! ! !

!

! ! ! ! ! !

! !

! ! !

! !

!

! ! !

! !

! !

! ! !

! ! ! !

! ! !

!

!

! !

!

! !

!

! !

!

!

!

! !

! ! !

!

!

!

!

! !

! !

!

!

!

!

!

!

!

!

! ! !

! ! ! ! ! !

!

!

! ! !

!

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

!

!

! !

!

! !

!

!

!

! !

! ! !

!

!

! ! !

!

!

!

! !

!

!

! !

! !

!

!

!

!

!

! !

!

! !

! ! ! ! ! !

> ! ! !

! !

!

! ! !

! !

! !

! ! ! !

! ! !

!

!

!

!

!

!

!

!

!

!

!

!

!

! ! ! ! !

! ! !

!

!

!

! !

!

!

!

! !

! ! !

! !

! !

!

!

!

!

! !

!

!

!

!

!

! ! ! ! !

!

!

!

!

!

!

!

!

!

!

! ! ! ! !

! ! !

!

!

!

!

! !

!

! !

!

!

!

!

! !

!

!

! ! !

!

! !

!

!

!

!

!

!

!

! !

!

!

! ! ! ! !

! !

!

! ! !

23

!

! !

!

!

!

! !

!

! !

!

!

! !

!

!

!

!

!

!

!

!

!

!

!

!

! !

> ! !

! !

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

! ! ! ! !

! ! !

!

!

!

! !

!

!

! !

!

!

!

!

!

!

!

! !

!

!

! ! ! ! !

! !

!

! ! !

!

! !

! !

!

!

!

!

! ! ! ! !

!

!

!

!

!

!

!

!

!

!

!

! ! ! ! !

! ! !

!

!

!

! !

!

!

! !

!

!

!

! ! !

!

!

!

!

!

! ! !

!

23

!

!

!

!

! ! !

! ! ! ! ! !

!

!

! !

! !

!

!

!

!

!

! !

!

! !

! ! ! ! ! !

! ! !

! !

!

! ! !

! !

! !

! ! ! !

! ! !

! !

!

!

! !

!

! !

!

! !

!

!

!

!

!

!

!

!

!

!

!

! !

> ! !

! !

!

!

!

!

!

!

!

!

! ! !

! ! ! ! ! !

!

!

!

!

! !

!

! ! !

! !

! !

! ! !

! ! ! !

! ! !

!

!

! ! !

!

! !

!

!

!

! ! !

! !

!

!

!

!

!

! !

!

! !

! !

! ! ! ! ! !

> ! ! !

> > !

!

!

!

!

!

!

! !

!

! !

!

!

!

! !

! ! !

!

!

! ! ! ! ! !

!

!

!

!

!

!

!

!

!

!

!

! ! !

!

! ! !

!

!

!

!

!

!

!

! ! ! !

!

! ! !

!

!

! ! ! ! !

! ! !

!

! ! ! !

!

!

!

! ! ! !

! !

!

!

!

! ! !

!

!

!

!

!

!

!

! !

!

!

!

!

! ! ! ! ! !

!

26

!

!

! ! !

!

! ! ! !

!

!

!

!

!

! ! !

!

!

! ! !

! !

!

!

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

!

! ! !

!

!

!

!

!

!

!

! !

!

!

!

!

!

26

!

!

! ! !

!

! ! ! !

!

!

!

!

!

!

! !! ! ! ! ! !

!

! ! !

! !

!

!

!

!

!

!

!

! !

! !

!

! !

!

! ! ! ! !

!

!

! ! !

!

!

!

! !

!

!

!

! ! ! !

!

! !

!

!

!

!

26

!

!

!

!

!

! !

!

!

! !

!

! ! !

!

!

!

!

!

!

!

! ! ! !

! ! ! ! ! ! !

!

! !

! ! !

!

!

!

!

!

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

!

!

! ! !

!

! ! ! !

!

!

!

!

!

! ! !

!

!

!

!

!

!

!

!

!

!

! ! ! ! ! !

!

! !

!

! ! ! ! !

! !

! !

!

! !

!

! !

!

! ! ! !

!

!

!

!

!

! !

!

!

! !

! !

! ! !

!

!

! ! !

!

! ! ! ! ! ! !

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

! !

! !

! !

!

! ! !

!

!

! !

!

! !

!

! !

! !

!

! ! ! !

!

!

!

!

!

! !

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

!

! ! !

!

!! !

!

!

! !

! ! !

!

!

! ! !

!

! ! ! ! ! ! !

!

!

!

! !

!

!

!

!

!

!

!

!

!

!

! !

!

! ! ! ! ! ! ! ! ! !

!

!

! !

!

!

! !

!

! ! ! !

!

!

!

!

! ! !

!

! !

! ! ! !

!

!

!

!

!

! !

! !

! ! !

! ! !

!

! !

!

!

!

! ! ! ! ! !

!

! ! !

!

!

!

!

!

!

!

! ! !

!

!

! !

!

!

! !

!

!

!

!

!

!

! !

! !

!

! !

!

!

!

! !

! ! !

!

!

!

!

!

!

!

!

!

!

!

! !

!

!

! !

!

!

!

! !

!

! ! ! !

!

!

!

! ! ! ! ! ! ! ! ! ! !

! !

! !

! !

! ! !

!

! !

!

!

!

!

! ! !

!

! !

! ! ! !

!

!

!

!

!

!

!

!

August 21 August 22

! ! ! ! !

! !

!

!

!

! !

!

! !

!

!

!

! !

!

! !

!

!

!

!

!

! ! ! !

! !

!

! !

!

! !

! !

!

!

!

!

!

!

!

!

! ! ! ! !

! !

!

! !

!

! !

!

!

! !!

! !

!

! !

!

! ! ! ! ! !

!

! !

! !

!

!

!

!

!

! !

! !

! !

!

! !

! !

!

!

! ! !

!

!

! ! !

! ! !

!

! ! ! !

!

!

!

!

!

! !

!

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

! !

! !

! !

!

! ! !

!

!! !

! !

!

!

!

!

!

! !

!

!

! ! !

!

!

!

!

!

! !

!

! !

! !

! ! ! ! ! ! ! !

! ! !! !

!

!

!

!

!

!

!

! !

!

!

! !

! !

!

!

!

!

!

! ! !

!

! ! !

!

! ! !

! !

!

! !

! ! !

! ! ! !

! ! !

!

! !

!

!

!

!

! !

! ! !

! ! ! ! ! !

!

!

! ! ! !

!

!

!

!

!

!

! !

!

! !

! !

! !

!

!

! ! !

!

!

! ! !

!

! ! ! ! ! ! !

! ! !

!

!

!

!

! !

!

! ! !

!

! ! !

! !

!

! ! !

!

!

!

! !

!

!

!

!

! ! !

!

! ! !

! !

! !

! ! !

! ! ! !

! ! !

!

!

! ! !

!

! !

!

!

!

! ! !

!

!

!

!

!

!

! !

!

! !

! !

! ! ! ! ! ! ! !

! ! !

!

!

!

!

!

!

!

!

!

!

! !

!

!

! !

!

!

!

! !

! ! !

!

! ! ! ! ! !

!

!

!

! !

! ! !

! !

! !

!

!

! !

!

! !

!

!

!

!

!

!

!

! !

!

!

!

!

!

! ! ! ! ! !

! ! !

!

!

!

! ! ! !

!

!

! !

!

!

!

! !

!

!

!

!

!

! ! !

!

! !

! ! !

! !

! ! !

!

!

!

!

!

! !

!

!

!

!

!

! !

!

!

!

!

!

!

!

!

!

! ! ! ! ! !

! ! !

!

!

!

! !

!

!

! !

!

!

!

!

!

! !

!

!

! ! ! ! !

! ! !

!

!

!

!

!

!

!

! !

!

! ! ! ! ! !

!

!

!

!

!

! !

!

! !

!

!

!

!

! ! ! ! !

! ! !

!

! ! !

!

!

!

!

! ! ! !

! !

!

!

!

! ! !

!

!

!

! !

!

!

!

!

! !

!

!

!

!

!

26

!

!

! ! ! !

!

!

! ! ! !

! !

!

! ! !

!

!

! ! !

!

!

!!

!

! !

!

!

!

!

! !

!

!

!

!

! ! ! ! ! !

!

26

!

!

! ! !

!

! ! ! !

!

!

!

!

!

!

!

!

! ! !

!

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

! !

! !

!

! ! ! ! ! ! !

!

!

!

! ! !

!

! ! ! !

!

!

!

!

!

!

!

! !! ! ! ! ! !

!

! ! !

!

!

!

!

!

!

!

! !

!

! !

!

! ! ! ! !

!

#### **5.4. 1992 Andrew Hurricane**

In Fig. 5 the evolution of the spatial pattern of flagged records from August 25 to August 28, 1992 during Hurricane Andrew and the corresponding daily weather maps shows a heavy pat‐ tern of flagging.. The flags in the spatial pattern figures are cumulative for the days indicated. The test shows that the spatial regression test explicitly marks the track of the tropical storm. Starting from the second land-fall of Hurricane Andrew at mid-south Louisiana, the weather stations along the route have flagged records. The wind field formed by Hurricane Andrew helps to define the influence zone of the hurricane on flags. Many stations without flags have daily precipitation of more than 2 inches as the hurricane passes, which confirms that the spa‐ tial regression test is performing reasonably well in the presence of high precipitation events.

#### **5.5. Cold front in 1990**

Flags for the cold front event during October, 1990 were examined. The maximum air tem‐ perature dropped by as much as 40 o F during the passage of the cold front. Spatial patterns of flags on October 6 coincide with the area traversed by the cold front and many stations were flagged in such states as North Dakota, South Dakota, Iowa, and Nebraska. On Octo‐ ber 7, the cold front moved to southeast regions beyond Nebraska and Iowa. Of course near‐ by stations on opposite sides of the cold front may experience different temperatures thus leading to flags. This may be further complicated when different times of observation are involved. The cold front continues moving and the area of high frequency of flags also moves with the front correspondingly.

A similar phenomenon can be found in the test of the precipitation and the minimum tem‐ perature. A spatial regression test of any of these three variables can roughly mark the movements of the cold front events. The identified movements of the cold fronts and associ‐ ated flagging of "good records" may lead to more manual work to examine the records. Simple pattern recognition tools have been developed to identify the spatial patterns of these flags and reset these flags automatically (see Fig. 6).

!

!

!

!

!

!

!

!

!

!

!

!

! !

!

!

!

!

!

!

! !

!

! !

! !

!

! !

!

!

!

Instrumental failures and abnormal events also lead to QA failures. Fig. 4 shows the time series of the Stratton Station in Color adooperated as part of the automated weather net‐ work. This station has nighttime (midnight) readings while all of the neighboring sites are AM or PM stations. Stratton thus has the most flagged records in the state (6): the highlight‐ ed records in Fig. 4 were flagged. We checked the hourly data time series to investigate the QA failure in the daily maximum temperature time series for the time period from April 20 to May 20, 2002. No value was found to support a Tmax of 88 for May 6 in the hourly time

station observation interval, in which the value of the afternoon of May 6 is recorded as the

17 is supported by the hourly observation time series (see Fig. 4 (b)) and is apparently asso‐

In Fig. 5 the evolution of the spatial pattern of flagged records from August 25 to August 28, 1992 during Hurricane Andrew and the corresponding daily weather maps shows a heavy pat‐ tern of flagging.. The flags in the spatial pattern figures are cumulative for the days indicated. The test shows that the spatial regression test explicitly marks the track of the tropical storm. Starting from the second land-fall of Hurricane Andrew at mid-south Louisiana, the weather stations along the route have flagged records. The wind field formed by Hurricane Andrew helps to define the influence zone of the hurricane on flags. Many stations without flags have daily precipitation of more than 2 inches as the hurricane passes, which confirms that the spa‐ tial regression test is performing reasonably well in the presence of high precipitation events.

Flags for the cold front event during October, 1990 were examined. The maximum air tem‐

of flags on October 6 coincide with the area traversed by the cold front and many stations were flagged in such states as North Dakota, South Dakota, Iowa, and Nebraska. On Octo‐ ber 7, the cold front moved to southeast regions beyond Nebraska and Iowa. Of course near‐ by stations on opposite sides of the cold front may experience different temperatures thus leading to flags. This may be further complicated when different times of observation are involved. The cold front continues moving and the area of high frequency of flags also

A similar phenomenon can be found in the test of the precipitation and the minimum tem‐ perature. A spatial regression test of any of these three variables can roughly mark the movements of the cold front events. The identified movements of the cold fronts and associ‐ ated flagging of "good records" may lead to more manual work to examine the records. Simple pattern recognition tools have been developed to identify the spatial patterns of

F observation of May 8 at 6:00 AM appears to be an observation

F during the passage of the cold front. Spatial patterns

F is recorded for the PM

F at 8:00 AM May

F appears to be an outlier. On May 7 a high of 85 o

error caused by a spike in the instrument reading. The observation of 93 o

ciated with a down burst from a decaying thunderstorm.

series, thus 88 o

high on May 7. The 102 o

14 Practical Concepts of Quality Control

**5.4. 1992 Andrew Hurricane**

**5.5. Cold front in 1990**

perature dropped by as much as 40 o

moves with the front correspondingly.

these flags and reset these flags automatically (see Fig. 6).

**Figure 5.** Daily weather maps and spatial pattern of flagged records for 1992 Andrew Hurricane events. (after 28).

The spatial patterns of flagged records are significant for both the spatial regression test of the cold front events and the tropical storm events. However, most of these flagged records are type I errors, thus we tested a simple pattern recognition tool to assist in reducing these flags. Differences still exist between the distribution patterns of the flagged records for the cold front event and the tropical storm events due to the characteristics of cold front events and tropical storm events. These differences are:

**•** Cold fronts have wide influence zones where the passages of the cold fronts are wider and the large areas immediately behind the cold front may have a significant flagged frac‐ tion of weather stations. The influence zones of the tropical storms are smaller where only the stations along the storm route and the neighboring stations have flags.

**5.6. Resetting the flags for cold front events and hurricanes**

standard deviation of the temperature change can be calculated as:

Some measurements during the cold front and the hurricane were valid but flagged as outliers due to the effect of QC tests during times of large temperature changes caused by the cold front passages and the heavy precipitation occurring in hurricanes. A simple spatial scheme was de‐ veloped to recognize regions where flags have been set due to Type I errors. The stations along the cold front may experience the mixed population where some stations have been affected by the cold fronts and others have not. A complex pattern recognition method can be applied to identify the influence zone of the cold fronts through the temperature changes (e.g. using some methods described in Jain et al, 2000). In our work, we use the simple rule to reset the flag given that significant temperature changes occur when the cold front passes. The mean and the

> 1 1 *<sup>n</sup>*

<sup>1</sup> \* \*

change at the*i* th station for the current day, *n* is the number of neighboring stations, and *σΔ<sup>T</sup>* is the standard deviation of the temperature change for the current day. A second round test

where*ΔT* is the difference between maximum/minimum air temperature for the current day and the last day. The cutoff value *f'* takes a value of 3.0. The test results with this refinement for Tmax are shown in Fig. 7 for Oct. 7, 1990. The results obtained using the refinements de‐ scribed in this section were labeled "modified SRT" and the results using the original SRT were labeled "original SRT" in Fig. 7 and 8. Of the 291 flags originally noted only 41 flags

For the heavy precipitation events, we compare the amount of precipitation at neighboring stations to see whether heavy precipitation occurred. We use a similar approach as for tem‐ perature to check the number of neighboring stations that have significant precipitation,

( ) *i threshold*

*i T T n* <sup>=</sup>

( ) <sup>2</sup> 0

*T* is the mean temperature change of the reference stations, *ΔTi*

' ' D - £D £D + *Tf T Tf*

s

remain after the reset phase. The daily temperature drops more than 20 o

*n T i i i*

=

*n*

sD

is applied to records that were flagged in the first round:

where the flags were reset and the largest drop is 55 o

z

where *<sup>Δ</sup>*¯

*i*

*TT TT*

 s

F.

= ³ *count p p* (10)

D= D å (7)

Toward a Better Quality Control of Weather Data

http://dx.doi.org/10.5772/51632

17

= D D -D D å (8)

D D *T T* (9)

is the temperature

F at most stations

**•** Cold fronts exert influences on both the air temperature and precipitation. The temper‐ ature differences between the regions immediately ahead of the cold fronts and regions behind can reach 10~20 o C. The precipitation events caused by the cold fronts may be significant, depending on the moisture in the atmosphere during the passage. The trop‐ ical storms generally produce a significant amount of precipitation. A few inches of rainfall in 24 hours is very common along the track because the tropical storms general‐ ly carry a large amount of moisture.

**Figure 6.** Spatial patterns of flagged records for cold front events and related fronts. The temperature map is the in‐ terpolated maximum temperature difference between October 6 and October 7, 1990. The color front is on October 7, and the black one is on October 6. The flags are the QA failures on that day.

#### **5.6. Resetting the flags for cold front events and hurricanes**

tion of weather stations. The influence zones of the tropical storms are smaller where only

**•** Cold fronts exert influences on both the air temperature and precipitation. The temper‐ ature differences between the regions immediately ahead of the cold fronts and regions

significant, depending on the moisture in the atmosphere during the passage. The trop‐ ical storms generally produce a significant amount of precipitation. A few inches of rainfall in 24 hours is very common along the track because the tropical storms general‐

**Figure 6.** Spatial patterns of flagged records for cold front events and related fronts. The temperature map is the in‐ terpolated maximum temperature difference between October 6 and October 7, 1990. The color front is on October

7, and the black one is on October 6. The flags are the QA failures on that day.

C. The precipitation events caused by the cold fronts may be

the stations along the storm route and the neighboring stations have flags.

behind can reach 10~20 o

16 Practical Concepts of Quality Control

ly carry a large amount of moisture.

Some measurements during the cold front and the hurricane were valid but flagged as outliers due to the effect of QC tests during times of large temperature changes caused by the cold front passages and the heavy precipitation occurring in hurricanes. A simple spatial scheme was de‐ veloped to recognize regions where flags have been set due to Type I errors. The stations along the cold front may experience the mixed population where some stations have been affected by the cold fronts and others have not. A complex pattern recognition method can be applied to identify the influence zone of the cold fronts through the temperature changes (e.g. using some methods described in Jain et al, 2000). In our work, we use the simple rule to reset the flag given that significant temperature changes occur when the cold front passes. The mean and the standard deviation of the temperature change can be calculated as:

$$\overline{\Delta T} = \frac{1}{n} \sum\_{l=1}^{n} \Delta T\_l \tag{7}$$

$$\left| \sigma\_{\Delta T} \right|^2 = \frac{1}{n} \sum\_{l=0}^{n} \left( \Delta T\_l \, \text{\*} \, \Delta T\_l \right) - \overline{\Delta T} \, \text{\*} \, \overline{\Delta T} \tag{8}$$

where *<sup>Δ</sup>*¯ *T* is the mean temperature change of the reference stations, *ΔTi* is the temperature change at the*i* th station for the current day, *n* is the number of neighboring stations, and *σΔ<sup>T</sup>* is the standard deviation of the temperature change for the current day. A second round test is applied to records that were flagged in the first round:

$$
\overline{\Delta T} - f' \sigma\_{\Lambda \overline{I}} \le \Delta T \le \overline{\Delta T} + f' \sigma\_{\Lambda \overline{I}} \tag{9}
$$

where*ΔT* is the difference between maximum/minimum air temperature for the current day and the last day. The cutoff value *f'* takes a value of 3.0. The test results with this refinement for Tmax are shown in Fig. 7 for Oct. 7, 1990. The results obtained using the refinements de‐ scribed in this section were labeled "modified SRT" and the results using the original SRT were labeled "original SRT" in Fig. 7 and 8. Of the 291 flags originally noted only 41 flags remain after the reset phase. The daily temperature drops more than 20 o F at most stations where the flags were reset and the largest drop is 55 o F.

For the heavy precipitation events, we compare the amount of precipitation at neighboring stations to see whether heavy precipitation occurred. We use a similar approach as for tem‐ perature to check the number of neighboring stations that have significant precipitation,

$$
\zeta = \text{count}(p\_i \ge p\_{\text{threshold}}) \tag{10}
$$

where the *p <sup>i</sup>* is the daily precipitation amount at a neighboring station, and *p threshold* is a threshold beyond which we recognize that a significant precipitation event has occurred at the neighboring station, e.g. 1 in. When *ζ* ≥2and*p phigh* , we reset the flag. Here *p* is the precip‐ itation amount of the current station, and *p high* is the upper threshold beyond which the threshold will flag the measurement. Fig.8 shows maps of flags after the reset process. Of the 78 flags originally noted only 41 flags remain after the reset phase. Most of the remain‐ ing flags are due to the precipitation being higher than the upper threshold.

Flags for the Andrew 1992 hurricane. The flags are the cumulative flags starting from Aug. 20 to Aug. 29, 1992. The flags by the modified SRT method overlay the flags by the

Toward a Better Quality Control of Weather Data

http://dx.doi.org/10.5772/51632

19

**6. Multiple interval methods based on measurements from reference**

based on the empirical statistical distributions underlying the observations.

precipitation and is inefficient in identifying the outliers.

sults in fewer Type I errors than the other methods.

One QC approach involved developing threshold quantification methods to identify a sub‐ set of data consisting of potential outliers in the precipitation observations with the aim of reducing the manual checking workload. This QC method for precipitation was developed

The search for precipitation quality control (QC) methods has proven difficult. The high spatial and temporal variability associated with precipitation data causes high uncertainty and edge creep when regression-based approaches are applied. Precipitation frequency dis‐ tributions are generally skewed rather than normally distributed. The commonly assumed normal distribution in QC methods is not a good representation of the actual distribution of

The SRTmethod is able to identify many of the errant data values but the rate of finding er‐ rant values to that of making type I errors is conservatively 1:6. This is not acceptable be‐ cause it would take excessive manpower to check all the flagged values that are generated in a nationwide network. For example, the number of precipitation observations from the co‐ operative network in a typical day is 4000. Using an error rate of 2% and considering the type I error rate indicates that several hundred values may be flagged, requiring substantial

(29) found the use of a single gamma distribution fit to all precipitation data was ineffective. A second test, the multiple intervals gamma distribution (MIGD) method, was introduced. It assumed that meteorological conditions that produce a certain range in average precipita‐ tion at surrounding stations will produce a predictable range at the target station. The MIGD method sorts data into bins according to the average of precipitation at neighboring stations; then, for the events in a specific bin, an associated gamma distribution is derived by fit to the same events at the target station. The new gamma distributions can then be used to establish the threshold for QC according to the user-selected probability of exceed‐ ance. We also employed the *Q* test for precipitation (20) using a metric based on compari‐ sons with neighboring stations. The performance of the three approaches was evaluated by assessing the fraction of "known" errors that can be identified in a seeded error dataset(18). The single gamma distribution and *Q*-test approach were found to be relatively efficient at identifying extreme precipitation values as potential outliers. However, the MIGD method outperforms the other two QC methods. This method identifies more seeded errors and re‐

original SRT method.

**stations for precipitation.**

personnel resources for assessment.

**Figure 7.** All points shown were flagged by the original SRT method while the red points were those that are flagged by the modified SRT method for maximum daily Temperature. Blue symbols are those that are reset by the modified SRT method.

**Figure 8.** This is the reset of flags for the Andrew 1992 hurricane. The flags are the cumulative flags starting from Aug. 20 to Aug. 29, 1992. The flags by the modified SRT method overlay the flags by the original SRT method.

Flags for the Andrew 1992 hurricane. The flags are the cumulative flags starting from Aug. 20 to Aug. 29, 1992. The flags by the modified SRT method overlay the flags by the original SRT method.

where the *p <sup>i</sup>*

18 Practical Concepts of Quality Control

the modified SRT method.

is the daily precipitation amount at a neighboring station, and *p threshold* is a

threshold beyond which we recognize that a significant precipitation event has occurred at the neighboring station, e.g. 1 in. When *ζ* ≥2and*p phigh* , we reset the flag. Here *p* is the precip‐ itation amount of the current station, and *p high* is the upper threshold beyond which the threshold will flag the measurement. Fig.8 shows maps of flags after the reset process. Of the 78 flags originally noted only 41 flags remain after the reset phase. Most of the remain‐

**Figure 7.** All points shown were flagged by the original SRT method while the red points were those that are flagged by the modified SRT method for maximum daily Temperature. Blue symbols are those that are reset by

**Figure 8.** This is the reset of flags for the Andrew 1992 hurricane. The flags are the cumulative flags starting from Aug.

20 to Aug. 29, 1992. The flags by the modified SRT method overlay the flags by the original SRT method.

ing flags are due to the precipitation being higher than the upper threshold.
