Following is a brief survey of some technical problems with the paper of Chuine et al. [Nature, 2004]. This survey is additional to the remarks on the note that I published about the paper in Theoretical and Applied Climatology [2007].
One obvious problem with the paper of Chuine et al. is that the confidence intervals for vineyard differences are based on 11-year smoothing (see their Figure 1). The authors give no reason for this smoothing. Using an 11-year average will misleadingly rid the error bars of large peaks, giving the impression that the errors are smaller than they really are.
A second problem is with the use of standard errors. The Supplementary Information for Chuine et al. states that the “delay between véraison and harvest is also very much constant (standard error of this delay is 0.74 days in the Colmar dataset, n=26)”. What is relevant for the analysis, however, is not the standard error, but the standard deviation. In this case, the standard deviation will be 0.74*sqrt(26) = 3.8 days. Given that Chuine et al. are analyzing data over 634 years, we should expect that there will be a couple of years where the error in estimating the delay between véraison and harvest is about three standard deviations, i.e. 11 days, which is substantial for the analysis.
A third problem is related to how the model was calibrated: the model's parameters were selected by comparing the model-estimated temperatures with actual temperatures; the parameters were chosen to minimize the sum of squared errors (SSE). That is reasonable if we are trying to see how the model performs on average. But the question being asked by Chuine et al. is not about averages. Instead, the question is this: is there any other single year whose summer was nearly as warm as, or warmer than, the summer of 2003? Thus the correct metric is not the SSE, but instead the maximum squared error, or the maximum positive error, or similar.
A fourth problem concerns the model inversion. The model is first run normally (i.e. without inversion), so as to calibrate it. When doing this, the inputs are (recent) daily temperatures, and the outputs are estimated harvest dates (which are then compared with actual harvest dates, as part of the calibration procedure). Afterwards, the model is inverted; so the inputs are harvest dates and the outputs are estimated temperatures. The inverted model, though, outputs annual temperatures (strictly, an average temperature for each year's grape-growing season). Thus, the normal model should have, as its inputs, annual temperatures. Having the normal model defined to input annual temperatures would mean that the model had much less information input to it, strongly indicating that it would be less accurate during calibration (as well as verification).
A fifth problem concerns the parameterization of the flowering date model. The parameters were chosen to minimize the SSE of the model during the calibration period (which comprised 17 years). Those parameters are listed by Chuine et al. as being <t0, Tb, F*> = <92, 8.63, 412.97>. Using those parameters, the SSE is 186. Using instead <t0, Tb, F*> = <101, 10.3, 302>, the SSE is 140. Hence, contrary to the claim of the authors, the parameters that were chosen are nowhere near optimal.
(There are other problems with the parameterization of the flowering date model, as well as with the véraison model. Those problems are not described here; they should be clear to someone with a background in numerical global optimization.)
Note that the first three problems would be clear to anyone with a knowledge of data analysis as taught at the second-year undergraduate level. The last two problems are also undergraduate level, albeit not second year. And only the fifth problem requires having the raw data. The conclusion is unpalatable, but clear: either the peer reviewers did not have an appropriate background in mathematical data analysis—upon which the paper heavily relied—or they did not honestly check the paper.
Addendum. W.W. Eschenbach kindly pointed out to me that the modelled temperatures of Chuine et al. are no more accurate than the temperatures that can be estimated via a simple linear regression of temperature on harvest date. Indeed, the SSE for the straight line is 35, whereas for the model, it is 54. I found that similar results hold even when only the warmest (50%, 25%, etc.) years are considered.
The figure below illustrates how well the straight line and the model fit the data.
Click to enlarge.
Open blue boxes represent observed temperatures (taken to be averages of adjusted monthly extrema, as per Chuine et al.). The turquoise line is a linear polynomial, fitted to the observations. The blue line is a lowess curve, fitted to the observations (generated using Maple, with default settings, omitting the left-most point). Solid red circles represent model-estimated temperatures. The red line is a cubic polynomial, fitted to the estimations. All temperatures are in °C; harvest dates are defined via 1 = September 1st. Years are 1883–2003, with 1978 missing.
Additionally, it seems clear from the figure that the model is essentially just a cubic polynomial. The underlying reason for that is plain from the model (detailed in the Supplementary Information). The model comprises three phases. The first phase describes the plant flowering; this phase is modelled by a linear polynomial. The second phase describes elongation of grape cells; this phase is empirically short and is modelled as exponentially small. The third phase describes photosynthate accumulation; this phase is modelled as a cubic polynomial. A linear polynomial is a cubic polynomial; hence the three phases together can be modelled by a cubic polynomial with some exponentially-small variations. The figure is consistent with that.
The essentially-cubic nature of the model also explains why the model substantially overestimates the temperature in 2003: the figure shows how the model curves above the observational data for early harvest dates. For early harvest dates, the straight line lies below the cubic. And the lowess curve is the lowest of the three; the lowess, moreover, is insignificantly different from horizontal prior to September 18th.
The figure also shows that only 3 (out of 120) harvest dates are prior to September 12th. Given the temperature variation around later harvest dates, 3 is too few to develop a reliable model, with error estimates, for early harvest dates