Rating Wines Controversy: It's ba-a-a-a-a-ck!

Hoke, Hi...

Taking your points in the order in which you wrote them:

If you are operating on a clear, "precise" ordinal system, you can't put unequal weight on the ordinations you use. The nature of numbers is that they are all equal---therefore the space between the integers is also precise, so there is no allowance for the significance of "breaking points". Not in the numbers themselves; that can exist only in the subjective mind of the observer.

- Methinks claiming great precision for this particular ordinal system would be foolish at best. It is most admittedly an interpretation and in that a bit fickle and the "breaking points" are truly projections of how this critic or that defines them. The only trick to decoding is becoming accustomed to the idiosyncrasies of the particular critic. The issue is thus not necessarily precision but consistency in the scoring system of a particular critic.

I see no possible way to posit an actual---versus projected---"breaking point" between 89 and 90. Certainly no more than there is between 88 and 89, or between 90 and 91. The breaking point exists only in your mind, not in the numbers you use. You create and impose it, by putting weight on specific numbers. And that takes away any sense of objectivity

- When it comes to deciding on breaking points, I compare those to the declaration of a hurricane as Force 3 or 4. Unlike the speed of sound which has a fixed measure (you are either under it, at it or below it), even the determination of the strength of the hurricane has a subjective element.

- Returning to the word rather than the number, even the determination of whether a wine is described as, for example, "superb", "outstanding" or "excellent in every way" is laded with interpretation both on the part of the writer and the reader. No more and no less than with scores.

How can you possibly apply the same numerical rating scale to three entirely different and contrasting wines? I know we have a desire to rank things, but how can you rank those three in any meaningful way? [/i]

- In the words of the King of Siam, this indeed "is a puzzlement" but it is a puzzlement with a two-part solution for, in the scoring of any wine one must consider some ideal maximum standard to which all all wines must be compared. As much as a vinho verde or a Beaujolais Villages may give enormous pleasure, such wines will never have the depth, breadth and length of a wine made from say Cabernet Sauvignon, Pinot Noir or Sangiovese. Nor, as another example, will a wine made from Emerald Riesling grapes ever match the flavour profile of a White Riesling. Unlike Will Rogers who never met a man he did not like, I have never met an off-dry Emerald Riesling that has earned more than 85 points.

- First, after having developed this standard (admittedly a partly objective/partly subjective task on the part of each critic), it becomes apparent that with very, very rare exceptions that the same scale can hold throughout.

- Second, although one strives to taste wines among their peers, one way check on one's loyalty to his/her standard, one should periodically do what I refer to as "mixed-group" tastings (e.g. a mixed bag of Beaujolais Cru wines, Burgundy, Bordeaux and Tuscan reds), those to see whether one's definitions and even stereotypes hold firmly. Another way of course is to have a "kicker" or two set into one's tastings.

- As I have said in earlier posts even on this thread, no-one should claim that scores are fully objective. Like the tasting notes themselves, scores are subject at least in part to the personal interpretation of those writing the notes or giving the scores.

… you put great care and thought into putting together the right words to describe and detail each and every wine you critique? Why in the world would you negate that care and concern and artistry by reducing it all down to a simple number score, merely for the convenience of a lazy reader to whom such a wine is meaningless anyway?

When talking with you I am never in a defensive mode. Our discussions offer far too much pleasure, thought and not infrequently positive challenge to feel a need for defensiveness. Despite that (and with a smile at the choice of my words in this case), I will defend myself by saying that a good deal of thought, some formal, some informal, goes into the awarding of a score.

As much as I would prefer not to print scores, I can understand that for many they do have value. Because I write tasting notes and award scores for all of the wines I taste, and because those scores in my case do range from 50-100, they have value at different levels for people who have either a minimum of knowledge or a great deal of wine savvy. On the one hand, the reader who enjoys off-dry Emerald Riesling or White Zinfandel wines for example is well aware of the difference between a wine that earns 65 points and one that earns 82 points. On the other hand the person who is deeply into and knowledgeable about the wines of Tuscany clearly recognizes, albeit on a shorthand basis, that the Brunello di Montalcino that earns 80 points is, at least in my opinion, qualitatively different from that which earns 92 points.


On somewhat of a tangent, there is one aspect to the issues scores that we have not discussed to this point but which has taken a great deal of thought on my part over the last nine-ten months. In my ponderings, I have consulted with several respected colleagues, winemakers and others in the wine trade as well as readers of my forum and column. This issue relates to the reality that nearly every wine producing region of the world is now giving us wines that have improved and continue to improve in quality on a regular basis. Because of that, the time has come to somewhat revise not my scores but the meanings of those scores. Whereas, for example, in the past a wine earning 75-79 points might have been considered average in quality, that "average" has now risen to include wines that earn 80-84 points and wines in the lower scoring category might now best be defined as mediocre.

There are indeed problems with "shifting" one's definitions, not least of which is that the revised definitions should not reflect negatively on scores awarded before the shift was made. Because of various logistical considerations, I will publish my somewhat revised definitions only on 1 January 2011. If my fully objective calculations are correct that leaves me another 81 days on which to continue pondering and formalizing my thoughts.

Very Best
Rogov

I think that a lot of misunderstandings re wine scores emerge from the wrong concept of the score. Many of those who criticize scores are, in fact, attacking the following implicit idea: there exists an objective linear ordering between the wines that can be expressed in two-digit numbers(so to say, "objective" scores), and different critics produce good or bad --- depending on their skills --- approximations of those "objective" scores. This idea is very easy to attack indeed, and I will be the first to reject it.

However, the true nature of scores is very different: they are nothing but encoded descriptions of the overall impression that a specific critic got while tasting the wine. As such, they are, first of all, not numbers! They only look like numbers, but any attempt of performing arithmetic operations with them is absolutely meaningless. One of the most common mistakes of this kind are so-called panel scores calculated as the average of the individual scores. Second, there is no sense in comparing scores of individual wines given by different critics. A 90-point wine is just a nonsense, whereas "Rogov's 90-point wine" and "Parker's 90-point wine" both have certain meaning, albeit different. Finally, concerning the mysterious one-point difference: 89-90 vs. 91-92. Here one again falls into the trap of thinking of scores as numbers and performing arithmetics (this time simply subtraction). Whereas 89 might mean (as Rogov explains in detail) something like "among the best in the category of very good wines", while 91 means "slightly better than the entry level in the category of excellent wines". With this understanding the difference between 92 and 91 becomes incomparable with the difference between 89 and 90.

I am in enthusiastic agreement with Alek on all that he says above. My thanks to him for forumlating those thoughts so very well.

Best
Rogov

My antipathy to the point system, in your case, is due to another reason: you put great care and thought into putting together the right words to describe and detail each and every wine you critique? Why in the world would you negate that care and concern and artistry by reducing it all down to a simple number score, merely for the convenience of a lazy reader to whom such a wine is meaningless anyway?

My thoughts exactly. The rest of this discussion will probably be a repetition of old arguments. Reading Rogov's side of the forum, I see too many cases of his readers making purchase decisions on the basis on 1-2 points either way, so his own reservations about the fallacy of scores are either not clear enough to his readers or are simply being disregarded.

One short anecdote: A few months ago, I was shopping at Gordon's at Waltham and debating between two Chevillon Premier Crus. Cheryl Lechan offered to print out the Meadows reviews and I wound up buying the lower scoring wine because it read like a more minerally wine. She liked my decision to ignore the score and we started talking and I made a new wine friend that way. By ignoring the scores. So it is like a personal vindication of my beliefs.

[James Roscoe]

This controversy is great! It's the great mental m_______n of the wine world (along with cork and terrior). Thanks Hoke!

As a basic neophyte I doubt I would see the difference between an 87 point wine or a 91 point wine. Depending on the point-giver, I might actually like the 87 point wine better! Personally, I tend to ignore the points and read the reviews.

Perhaps worth keeping in mind that a great many of the most respected wine critics in the world are not based in the United States and most surely do not appreciate the "enormous, over-oaked, jammy reds" and wines of that ilk. Indeed, not all critics, even in the United States have the power of Robert Parker. Regardless of their power, however, certainly not all appreciate or are biased toward such wines.

It's too bad that those critics are not more prominently voiced in the US, because they are not the ones that are in the popular wine periodicals and make the retail stores (which, for better or worse, steer the uneducated consumer to a certain wine style based on their scores). Unfortunately that has shaped the culture and has determined what the mass public thinks is a "good wine". I hear so many consumers rave about such and such a wine getting 94 points, and I want to shake them and tell them that none of that really matters!!! That is just one person's (or a group of critics) opinion, not an actual measure if the wine is good or not. But that brings me back to a point I raised before - one person's 90 is another's 70. It is all based on personal taste and preference. Heck, there are people out there who love Yellowtail and the like!

I do my best to seek out those who I know share my palate preference, and look for their tasting notes and recommendations. I find that infinitely more helpful than numerical scoring.

This brings up another issue in my mind. Has anyone ever tasted a wine one day and loved it and opened another bottle down the road and been surprised at how different it seemed? Assuming there were no flaws in the wine, how to we explain this? There could be dozens of different factors affecting this. Perhaps your tastes have changed, or the setting you are tasting it in, serving temperature, physiological factors such as being able to breathe clearly, etc. All these things could influence a critic's score - one day they might score a wine an 88; another day a 90. I'm sure this has been the case, especially when tasting blind. Because of these factors, I have a hard time taking one's numerical score at the definitive answer on a particular wine. It would be so much more helpful to review tasting notes in this case.

Just my 2 cents!

Cheers!
Joy

Joy, Hi...


Before starting, let me state loudly and clearly that I am not in the least bit anti-American. Having so stated, there is no-one to blame or praise for American culture other than Americans. That is fair enough, though, for the same is true of Europeans.

Democracy does, however, have its place in the process of selection, for, as French and English critics are not well known by consumers in the USA, I can comfortably assure you that American wine critics are neither well known nor well appreciated by wine consumers in the UK and throughout most of Europe. Even Robert Parker, who may have his influence in the trade in Bordeaux and other places, is very little known by wine consumers on the "other side" of the Atlantic Ocean.

And, for the most part, English and European critics do rate wines but, as a result of cultural differences do not use the 100 point scale. Americans take easily to the 100 point scale because that is that same system used in awarding grades in school. The English and European standard for grades is 20 points and many critics therefore use that as their base. True, some use a one-five star rating system.

I would suggest, not only to you but to all, that scores have something in common with either post-modern literature or pornography. Simply stated, if you don't like scores, post-modern literature or pornography, ignore them. They won't "go away" but at least they will not bother you.

As to your other issue, indeed many people taste a wine one day or one year and love it and then later on open another bottle to realize how different it seemed. Agreed with you that there are many factors that might influence one's reactions to a wine but when it comes to writing criticism, whether the critic "likes" or "dislikes" a wine is not at issue. The issue is only whether the wine meets the standards that were set out for it. As a personal example, I so dislike wines made from the Emerald Riesling grape that if I live to be 120 years old (ask Hoke, I'm not far from that) I will never walk into a wine shop to buy a bottle of it. On the other hand, as a professional I must taste and evaluate such wines. That is no problem as my personal likes do not come into question. I taste the wine and evaluate it on one basis and one only – standards. Nor in my criticism will I say or even imply anything negative about those people who enjoy Emerald Riesling. It is both the right and privilege of any person to enjoy or not enjoy any wine of his/her choice.

As to the same critic tasting the same wine more than once and giving it widely varying scores. The two major reasons for this with a professional are either bottle variation or taster variation. The manner in which the critic can avoid such variation influencing his/her notes is quite simple. First of all, one of the advantages of tasting blind is that one can "double up" wines in a tasting…that is to say, having the same wine inserted into the tasting in differently numbered glasses. When that is revealed at the end of the tasting if the tasting notes and scores vary too widely, that means that either the critic's attention was not fully focused on the wines or that his/her palate was having an "off" day. In cases were the difference is anything more than minimal, the results of the entire tasting should be discarded and those wines re-tasted on a "better day". As to bottle variation, that is one of the reasons why professional critics try as often as is feasible to re-taste, re-taste and re-taste yet again, that both to check the accuracy of one's original notes and to follow a wine as it develops in the bottle.

Once again, we are all in agreement that tasting notes are far, far more important than scores. More than that, the tasting notes that should be most important to any given wine lover are those of that critic or those critics who most closely give direction to their own palate.

As to the issue of panel scores – for the moment let it merely be said that I am opposed to both tasting notes and scores derived by panels.

As you may by now be realizing, I am not a man of few words. I apologize for that.

Best
Rogov

Daniel:

Okay, the riposte in re the concept of the breaking point and use of numbers as more like a Richter Scale is a good one. I can accept that concept at least---though I seriously doubt any of your readers and point-followers have even thought of it, and for them the numbering sequence is strictly linear and mathematical.

The horrifying thought in this vein, though, is that people like Monkton lawyers, having exhausted the limitations of 1-100, will now proceed to use numbers algebraically, and soon we will inevitably see squared scores.

However, you have not as yet convinced me( or even come close, really) to the concept that the point scale includes a mythical master wine of consummate greatness, the Holy Grail of Winedom, that encompasses all wines.

I cannot conceive of one scale that would encompass the three wines that I proferred as examples. As simple as that.

I can conceive, at least, of a system that ranks all wines of type, yes. So a ranking system of all Orange wines, or all Bordeaux wines, or all Burgundy wines is at least theoretically possible for me. But a singular system that encompasses the three wines mentioned is impossible.

As always, respectfully and affectionately, Hoke

Perhaps worth keeping in mind that a great many of the most respected wine critics in the world are not based in the United States and most surely do not appreciate the "enormous, over-oaked, jammy reds" and wines of that ilk. Indeed, not all critics, even in the United States have the power of Robert Parker. Regardless of their power, however, certainly not all appreciate or are biased toward such wines.

It's too bad that those critics are not more prominently voiced in the US, because they are not the ones that are in the popular wine periodicals and make the retail stores (which, for better or worse, steer the uneducated consumer to a certain wine style based on their scores).

What about Allen Meadows? He uses points, seems to be pretty influential for Burgundy, yet does not like "enormous, over-oaked, jammy reds".

two points that have not been mentioned:

one major rating system is ignored. wine 'competitions' give Gold, Silver, etc. medals to wines. this is a rating system. some ratings are given by individuals(parker, wine spectator, et. al.); others are given by panels(wine and spirits, competitons, et. al.). i think that competitions are the best at handling the next point. they can consider categories other than simple 'quality'.

but my major problem with this entire discussion is the notion that 'quality' is the one and only valid criteria for judging a wine. the main contender in my view is the category of 'delicious'. could a simply delicious pinot grigio ever be a 'quality' wine - could it be a 100pt wine? even a 90pt wine? probably not. would a critic who gave an unoaked and absurdly delicious wine a score of 100pts. ever be taken seriously again. would a critic who gave a pinot grigio of any sort 100pts. ever be taken seriously? and yet, perhaps like Hoke, i find myself drawn more an more to simply delicious wines - red, white, rose, whatever. i also love high quality wines - but they are far more rare than the critics tell us.

other categories other than 'quality' and 'delicious': superbly representitive of the region or type; historically accurate(seeing how it used to be done); and there may be others.

David, Hi...

As to competitions, I am firmly of the school that believes such events so seriously faulted that with very few exceptions their awards are beneath meaningless. I recently posted about my problems with competitions at viewtopic.php?f=29&t=35139

With re quality - indeed that is not the only factor in awarding a wine a score. Every critic and every scoring sheet that I have ever seen includes a category of "overall impression". More than that, I agree with one of those in the article to which Hoke pointed, that although a wine may earn a critical score of 85 it may still earn "pleasure points" of 97 (e.g. a vinho verde or a Beaujolais Village, or even the simplest country-style wine when served at a harvest festival in the village from where the grapes come). A critical score may indeed have a partly subjective aspect but the pleasure score is entirely subjective and will vary not according to the wine but to the individual drinking it, the circumstances in which it is consumed, etc....

Best
Rogov

Here's my answer to the poll...

I want a complete review of the wine with whatever system the critic prefers, numerical, starred or othwerwise. It is not an either/or proposition.

ok, mr. rogov - i've read your article about wine competitions. what you report bears absolutely no resemblance to any competition at which i have ever judged. and, so far as i can tell from talking to respected judges - berger, frost, peterson, heald, puchalowski, salvi and others - the shanigans you describe would never be tolerated here. we regularly evaluate judges on nearly all the problem points you mention and 'throw the bums out' if necessary.

in addition, american run competitions usually require consensus from the judges following a discussion of each wine - not an averaging of privately arrived at scores. this way, a judge who has found a virtue or flaw in a particular wine can be asked to reevaluate. sometimes they still find it; sometimes they admit they misjudged; but each wine gets a pretty fair hearing based on a frank and honest discussion; and judges are actually forced to defend their positions. those who can't usually quit of their own volition.

i can see that if competitions in general suffer from the flaws you mention, then the (hopefully) honest appraisal of a single person will be more useful than that of a panel. but all competitions are NOT like that.

in any event, the results of competitions would belong on the list of options for what information one would prefer: stars, 100 points, no points, whatever. certainly as we have seen all these things have problems; but they don't disappear from the list because of them.

What about Allen Meadows? He uses points, seems to be pretty influential for Burgundy, yet does not like "enormous, over-oaked, jammy reds".

If only he were the norm rather than the exception! The problem is that he is limited to one region of wines. If you are looking for great Burgundy, then go to him. But that doesn't do anyone good if they are looking for anything else. The issue it that generally US critics favor a certain style of wines and that is what makes the major wine publications in this country. You have to go to trade journals such as the Sommelier Journal to get decent reviews from professionals who you know are educated and generally not favoring that style of wine.

This controversy is great! It's the great mental m_______n of the wine world (along with cork and terrior). Thanks Hoke!

Better not let Christine O'Donnell read you said that, James. She'll put a spell on you!

David, Hi....

If the truth be told, I am largely unaware (use the word "ignorant" if you will and I would not be offended) of American wine competitions. If you would be kind enough to provide a list of some of those to which you refer, I would much appreciate that and would do a studied follow-up. If those competitions prove as you say, without shenanegans, I would be more than pleased to so report.

Thanks
Rogov

How much “weight” are they putting on the category of “overall impression”? Even so, “overall impression” is/ and should (IMO) be included in the evaluation, it should however not be “weighted” (and I believe you pointed that out in your last post) as much as the other categories used for evaluation. In the AWS (American Wine Society) UC Davis Modified 20 point scale; appearance (max 3 points); aroma and bouquet (max 6 points); taste and texture (max 6 points); aftertaste (max 3 points) and overall impression (max 2 points). The two points awarded for overall impression will, IMHO be similar to what you called the “breaking point” decider. Maybe moving a wine from very good to excellent (just because you liked it a lot and enjoyed it).

Salute

Victor, Hi....

There is a certain variation to what proportion of a total score is left to "ovrall impression", that depending on the individual critic's scoring system. To the best of my knowledge that varies between 10-20% of total score.

Best
Rogov

David
What's your view on the influence of the senior judge, chair of judges? There's seems to be quite some talk in Australia over what's claimed to be excessive influence, e.g. instructions that wines showing green aspects to be marked down, or wines showing over-ripeness to be marked down. i.e. personal agenda of the lead judge overpowering other opinions.

regards

Ian

If only he were the norm rather than the exception! The problem is that he is limited to one region of wines. If you are looking for great Burgundy, then go to him. But that doesn't do anyone good if they are looking for anything else..

What else are you looking for?

Outside of the very expensive regions, do you really need a critic to tell you what wines to buy? Just talking to folks around here or to a smart retail clerk seems to be way more than sufficient.

What else are you looking for?

Outside of the very expensive regions, do you really need a critic to tell you what wines to buy? Just talking to folks around here or to a smart retail clerk seems to be way more than sufficient.

I'm not saying that I am always directed by critics as to what to buy, but I think it is fair to say that the general population is. If not, than what is the point of a critic? Unless you are talking about bloggers who do it for fun, professional critics are paid to influence people's buying decisions. Whether it be a wine critic, food critic, movie critic, etc, their whole purpose is to give their opinion and people follow that.

If I am an average consumer who doesn't have the luxury of being in a large city with great wine shops full of educated employees, and am not familiar with many of the regions of the world, how am I supposed to know where to start? Sure, there is always trial and error, but that can get enormously expensive. I think critics have their place, and although I'm not a fan of the points scale, I can appreciate tasting notes and learn to read the styles they prefer and so be more informed when stepping into a wine shop.

I think it is safe to say that most people are influenced by others' recommendations at times in their life when buying wine, especially those of like-minded and palated individuals. I had the privilege of working with a Master Sommelier at one of the restaurants I worked at, and he steered me towards a number of amazing wines that the average retail employee would be clueless about. So is the job of the quality professional critic - someone who gets paid to sift through the "crap" and call out the good stuff (depending on their palate preference). And it is nice to read reviews on wines not readily available in my area to know what to seek out when buying online as well.

If I am an average consumer who doesn't have the luxury of being in a large city with great wine shops full of educated employees, and am not familiar with many of the regions of the world, how am I supposed to know where to start?

Google.

(Like for everything these days).

I think there is a fair amount of diversity out there in terms of opinions/palates/perspectives.

But I agree, it's not easy to understand the obscure wines without any guidance. That's why they're still obscure!

As is known, when I publish wine reviews they are accompanied by a score and I use the 100 point system. Despite that I voted that I prefer wine reviews without scores.
My logic is quite simple...when I first started using the 100 point system (indeed a 50 point system) those scores were entirely a reference for my personal benefit and were not published.

There is something intellectually repugnant about a so called 100 point system that has one half the scale automatically banned from use. If it is truly a 50 point scale, say so and rate out of 50 points.

Hoke, Hi...

You talk about a friend who sends you nothing but the score and as what that should mean to you. If I may be so bold, it means that your friend is a schmuck. After all, what he has done is the equivalent of having sent you the hub cap or hood ornament of a Lamborghini and asked you to use that in making your judgement about the car.

Best
Rogov

Actually, Lamborghinis don't have hub caps. The early ones with Borrani wire wheels and the intermediate modeals (like mine) with spline drive magnesium Campagnolo wheels use a 3 eared knock off (which do indeed cost something upward of $1000 a set). The cars after the early 70s use bold on mags that also lack any form of hub cap other than a small centre finisher to cover the central hole. Sorry, I couldn't let that one pass uncommented upon - I gave in to my inner pedant!

FWIW, I think that even the pros, although they may not admit it, use a sliding scale. I have a very hard time thinking that a Beaujolais that gets a 90 point score is really as much a wine as the vintage of Petrus or Haut Brion that got the same score. A 90 point French Colombard isn't going to be a better wine than an 89 point Grand Cru Burgundy, IMHO, even though that first wine may be a true prodigy in it's own class.

[James Roscoe]

This controversy is great! It's the great mental m_______n of the wine world (along with cork and terrior). Thanks Hoke!

Better not let Christine O'Donnell read you said that, James. She'll put a spell on you!

You know witch rhymes with bitch. I don't want to get too close to either type of person (whether actually female or.....? ).

Now back to our regularly scheduled program.