Rating Wines Controversy: It's ba-a-a-a-a-ck!

I think the more knowledge one has about wine, the less value point scales have. In 1985, when I was just learning about wine, I depended on Parker's (et al) reviews and the 100 (or 20 or 5) point scales - they were a great way to understand many things about wine: I learned what some folks thought was good and not so good, I learned how different one wine can be to different people and about climate, soil and wine makers - all the things that go into a bottle of wine. But now, after 25 years of tasting and developing my own beliefs, the scales mean the least of all. Reading an analysis about a wine, reading weather reports and picking up a subtle detail about rainfall or getting recommendations from friends I have confidence in will trump a number anytime.

hello Ian - i haven't seen much of that here. again, i don't think the judges would put up with it. in my experience, the chief judge gets called in to help settle disputes - or more often to provide additional information that will help the judges make a more reasoned decision. i have heard that the chief judge can require that a wine cannot receive a bronze medal if it gets even one vote for no medal. this has the effect of holding down the usually high number of bronzes. this is about as intrusive as i have seen.

Rogov - here is a link to the only major wine competition i judge at:

http://www.fliwc.com

you will notice one Isreali fellow - Moises Spak - who appears to also run his own competition. perhaps you know him or could find him and talk to him about this Finger Lakes competition. or maybe you've run into John Salvi over the years and could ask his opinion. Dan Berger runs the Riverside International Wine Competition - again, maybe you could contact him or find their website. I am supremely confident he would not tolerate any monkey business in his domaine. Doug Frost is an MS and MW and runs a couple of competitons. he is pretty accessible as well. google can track him down i'm sure. the san francisco competition is pretty well regarded but i don't know much about it other than that.

i used to run a smaller competition for wines from my state only - Michigan - and now judge for it. Berger judges for us every year and Frost has several times. Any hint of anything amiss and our judges would yell bloody murder. there are very few sheep around here.

http://www.michiganwines.com

Bob Whitley is another good contact, since he runs a few different wine competitions.

David has cited a few competitions and, while it's true that many of the top-level competitions do operate in a generically similar way...the devil is often in the details.

First thing you have to do is separate the wheat from the chaff---lots and lots and lots of competitions out there and some of them are run by very strict guidelines. Some of them, however, are run like local mom-and-pop back yard barbecues: they're fun for those who attend but not of much interest to anyone else.

Some competitions require "judge testing" and periodic re-testing during the competition to see if there is notable variance in the judge's assessments. CA state fair competition is one such. Others are simply at the whim of the person who designs the competition.

Some competitions make sure to invite certain specific people; some have loose categories that can include "judges in training" (i.e., often a "we couldn't find a reputable proven judge so we filled in at the last moment" or "this person has a lot of clout so let's massage them by letting them pretend to be a wine judge", but just as often legit too).

Many competitions insist on mixing up the judging panel, getting people from all walks of life---but usually the trade or academic. One of the general rules of competitions: never do a competition with an all winemaker panel. Winemakers often (not always, but often) are the worst judges, since they tend to focus (and be in favor of) their own style and are blind to others.

Balanced judging panels is an interesting dynamic, and often the determining point in whether a competition is good or not.

Another determinant---and also a very important factor---is what the categories are, and how finely they are parsed. Competitions can divide by variety/country/region/style/price point (and all sorts of gradations thereof)/ sugar levels, etc.

So you can have "Riesling"....or you can have seven or more different classifications of Rieslings, depending upon how you parse it with style/sugar classifications, or with price gradations. That gives you the potential, in Riesling alone, of multiplying your medal score by a significant amount (say, seven Gold Medals instead of one)....and to some wineries, it's all in the total number of medals that counts, so the "high medal" comps get more entrants than the more stringent ones. Because all the public sees is a group of medals of some sort.

Then there's 'grade inflation'. Some of the older competitions, or competitions that are trying to make a big splash, steadily inflate the number and type of medals...so you get Bronze/Silve/Gold...then Platinum, then Double Gold, then Best of Class, the "Sweepstakes", all in a bid to trump up the flummery; make something "bigger and better".

I've done quite a few competitions. Some of them I would advise people NOT to pay attention to. Some I would recco strongly. Some of them the results were not particularly meaningful to me; some of them were particularly meaningful in that I learned something about the state of the biz and I thought the results were worthwhile indicators.

But one thing I always remember. Always! Any and every competition is a promotional event and it is done to generate a profit of some sort. The judges usually work either free or for a pitifully small compensation, so the experience of tasting broadly and the "prestige" within trade circles is usually their greatest reward---they're certainly not making money from it.

And as with anything in wine, a competition can only be an indicator, just as a critic is an indicator. It still, and always has been, caveat emptor and consider the source.

David, Hi Again...

Thanks for posting the information on the Finger Lakes Competition. Before my comments, please do not take anything I have to say as antagonistic towards or aimed at you. My comments are aimed directly at competitions, those including not only competitions such as the Finger Lakes or Israeli held Terravino but also major competitions such as at VinItaly and Bourg en Blaye which is associated with VinExpo. More than this, the comments that follow are in addition to what I have said in the earlier article to which I linked.

1. With regard to consulting before awarding final scores, it has been clearly demonstrated that in such consultations there is a dominant personality, perhaps the chairperson or perhaps another on the panel and that at least several of the members of each panel will be at least unconsciously influenced by that dominance in their own scores.

2. For many years, competitions awarded Gold, Silver and Bronze medals. To that has now been added the category of "Double Gold". One of the things that this means is that wines that formerly would have been awarded Gold now move up to Double Gold, and so forth down the line to wines that might have been given no award whatever now earning a Bronze Medal. I recently wrote, somewhat sarcastically, that in the near future we will be seeing Platinum awards. I dread the day we will attain Uranium 238 awards and that those wines attaining scores of 65 will be awarded a bronze medal. Something akin perhaps to the American College Entrance Exam Boards where you start with a score of 400 simply for having shown up and signing your name.

3. I agree wholeheartedly that some of the judges at respected competitions are indeed well qualified to serve as judges. My problem comes in when even the best of judges serves as a member of a panel, not only the first problem coming into play but also a good many statistical problems.

4. I also have a problem with some of those judges who spend anywhere from 30-50% of their time traveling from competition to competition in order to serve as judges. For not a few of these, this is the only way they can have their expenses covered for visits outside of their own country or venue. More than that, in not a few competitions judges are invited only if they can bring in a certain number of wines from their own country. There is an obvious conflict of interest in this.

5. I do indeed know Moise Spaak and several of the other judges you named and I have respect for some of those. Spaak organizes not one competition but more than ten annually, those including one in Israel as well as quite a few in some pretty obscure places. With all due respect to him, Spaak is the perfect example of how too many wine competitions have become businesses in and of themselves, having nothing to do with improving the image of wine but in turning a good profit.

6. I am often frustrated by competitions that claim to be "international" in scope but where a huge percentage of the wines being judged come from a very focused and relatively small area (e.g. Israel, Finger Lakes). It never fails to amuse me that in an Israeli or Finger Lakes based "international competition", Israeli or Finger Lakes wines will walk away with 75-85% of the awards. Are we to then assume that Israeli or Finger Lakes wines represent 75-80% of the world's greatest wines?

7. I am not saying that all judges would allow hanky-panky. I am, however saying that in many cases judges have no control over whatever hanky-panky takes place in back rooms.

8. With regard to scores, in many competitions a score of 85 is quite enough to win a Bronze Medal. Now a wine with a score like that may be pleasant enough, but enough to garner a medal that will be awarded at a gala dinner?

9. I also question the validity of competitions in which many of the very best wineries would not dream of entering their best wines because they have nothing to gain by winning a gold or platinum medal for a wine already acknowledge to be superb and everything to lose if the judges decide that their wine placed fiftieth after the equivalent of a Two Buck Chuck.

9. As to the results and those little gold stick-on badges that wineries can place on their wines, when I hear that a wine earns "double-gold" I am expecting wines that can compete with the best of Bordeaux, Burgundy, Tuscany, Sauternes, etc, etc…. As a single example of how this can be problematic – the awarding in 2009 of Double Gold to Gallo's "Sherry"

As can be seen, and perhaps as a pleasant change, I am in full accord with everything that Hoke posted in regard to competitions (above).

In short, I continue to feel that with perhaps only one or two competitions held worldwide, competition results are without meaning at best and misleading to consumers at worst.

Best
Rogov

Rogov and Hoke - i'll try to take a few of both your points; starting with Hoke. nothing down to the parsing issue seems to me to constitute a per se problem with competitions and many would surely look to balance and to make sure to invite one or two more competant judges than actually needed in case someone comes down sick or misses a flight - as Pooch did at last years FLIWC. as to the Riesling example: i simply don't understand how having say seven categories would yield seven times as many gold medals. a group of 50 rieslings no matter how divided might merit 10 gold medals or none. the chips fall where they may. is someone assuming that the highest scoring wine is automatically a gold medal? not so. then there is the subject of grade inflation. one needs actual evidence here; and it can't be hard to come by. is it in fact true that the SF or Riverside competittions have awarded more total and gold medals as a percentage of entries each year of their existence? that information is available. what does it show?

on Rogov's point #1 - my experience is that this does not seem to be big factor; though i would love to have an observer assess that independently. typically, there is not discussion until each judge has given their own choice of medal. to avoid dominance at this level competitions typically begin taking these assessments with a new person each time. the captain may begin the first flite by giving his own score first; but then the second flite is started off by the person on his left and continues in this fashion the whole day. if there is an obvious consensus, no discussion actually takes place - the award is recorded. further, in competitions where a great variety of wines are included, any dominance often passes from person to person. if i am the only one who has ever tasted a wine from the Diamond variety, i may have more influence for that reason. and so on with the speciaties of other judges. i am hard pressed to see this as a problem. on #2- simply put, this is NOT the case. Double Gold is always a sub category of gold not a new category. it does not in any way increase the total number of gold medals nor the number of medals. since it is quite uncommon for four judges to be in complete agreement, some competitions create a name for gold medal wines that were awarded all gold scores - as opposed to say three golds and a silver. as to #6 - i suspect you would rather take this one back. if the competition is in Israel, most of the wines entered will be from Israel. Same with Finger Lakes - though to a lesser extent. are you really surprised that the percentage of medals bears a similarity to the percentage of entries? and are they therefore the worlds greatest wines? well, we can only judge what we have in front of us - not all the worlds wines. Really! #7 what hanky panky CAN take place in the back room - adulterating a sample? they open bottles, pour samples in glasses and tag them with the number of the sample. #8 - this isn't how many if not most US competitions work. each judge arrives at a judgement about the merit of each wine by whatever means he or she wishes. they have been invited for their expertise and need not be told how to arrive at their opinion. some judges score - many do not. they announce not scores; but vote for a medal - e.g. Silver - perhaps low or high Silver. taken together, these assessments may give a clear consensus - or may require discusssion whereby the low vote tries to convince others and the high vote tries the same. retaste, rethink and try again. and i am completely mystified by #9. maybe this is because you think that medals are based on scores and only wines the equal of the worlds greatest wines could possibly get the highest medal. consumers can't generally afford the worlds greatest wines even if they could find them. this is a way of categorizing the 'best of the rest' if you will or finding the gems amongst the swill - wines that consumers can actually find and buy that will give them a great deal of pleasure and are arguably superior to others they could spend their money on.

anyway, this is an interesting discussion.

David, Hi Yet Again....

As to hanky-panky in the back room, quite simple. At truly serious competitions the tasting sheets of each judge are given to members of a firm of certified public accountants or attorneys and they do the summations. I have seen competitions where the tasting sheets are given to the organizer of the competition and at not a few of these events the organizers have a vested interest in awarding at least some "prizes" to as many wineries as possible. That, after all, ensures that they will return and pay the entry fees the following year.

As to tasting sheets, you have me confused a bit. To the best of my knowledge, all tasting sheets culiminate in a score.

As to discussions, again I am befuddled. At many of the very best competitions judges taste anywhere between 60-80 wines daily. If each of those wines were to be reviewed by panels (even if agreement is unanimous throughout) that would take far longer than the tastings themselves. In a six day period, judges taste up to 480 wines. More than that, many of those wines are tasted and evalued by more than a single panel.

You say that scores are not the entire basis on which wines are awarded prices. Scores are, in fact, the only basis used at the most highly respected of competitions (e.g. IWSC, VinItaly, VinExpo). Many of these competitions, at which over 3,000 wines in all are tasted, have up to 12 panels, each with anywhere from 6 - 8 members. One might think that leads to statistical democracy. According to the laws of mathematics and statistics, it does not. It leads to a form of well-organized anarchy.

Methinks me rests me case.

Best
Rogov

Rogov and Hoke - i'll try to take a few of both your points; starting with Hoke. nothing down to the parsing issue seems to me to constitute a per se problem with competitions and many would surely look to balance and to make sure to invite one or two more competant judges than actually needed in case someone comes down sick or misses a flight - as Pooch did at last years FLIWC. as to the Riesling example: i simply don't understand how having say seven categories would yield seven times as many gold medals. a group of 50 rieslings no matter how divided might merit 10 gold medals or none. the chips fall where they may. is someone assuming that the highest scoring wine is automatically a gold medal? not so. then there is the subject of grade inflation. one needs actual evidence here; and it can't be hard to come by. is it in fact true that the SF or Riverside competittions have awarded more total and gold medals as a percentage of entries each year of their existence? that information is available. what does it show?

I speak from experience, David. Yes, one should have a couple of judges on hand. And yes, no matter how well you plan, shit happens; judges don't show up, get sick, etc., and stand-ins get asked. I have been in that situation myself, when I was asked to sub---at the very last minute---for a judge friend. I have also sat in on panels where the listed judge did not show and a "judge in training" was put in the chair at the last moment; this JIT, mind you, was a perfectly nice young lady who had worked in retail for all of two years in a small store. She was sitting down with me (no big shakes there, I'll grant) and also a Master of Wine and Winemaker of great repute, as well as an experienced grape grower and winemaker from another region. Like I said, shit happens.

The Riesling: you're not catching what I'm saying. If you provide one category (Riesling), then you're likely to get a certain number of medal awards. If you subdivide Riesling into sub-categories, human nature is to consider that each category MAY be deserving of medal or recognition And don't tell me that doesn't happen, David, please: I've been there and I've seen it happen.

Another point I didn't mention is the personal predilections and campaigns of judges: I have been in situations (more than once) when judges made a point of mentioning they were LOOKING to champion certain varieties or regions, and would be more than willing to cheer on a particular wine or style to a gold medal "to give it a little nudge in the market" or "to bring a little well-deserved attention" to it.

I have also been part of discussions where outright bargaining went on at the judge's table---if I give you this, you give me that. It happens. Judges are not impartial; sometimes they are distressingly partial. Most are not, or try not to be anyway. And as Rogov said in a post, some judges are considerably more forceful or commanding (and sometimes bullying) than others. (No derogation at all intended or implied, but have you ever worked with Darrell Corti on your panel? Harriett Lembeck? Very decided people with very decided attitudes, and potentially inhibiting to a younger, less experienced judge. Even Berger, god love him, can be unsettling to a younker, as he cruises at high speed through a flight, finishes before you do, and then sits there reading the paper or writing, obviously bored to death and ready to get on with it when you're just getting to wine three out of ten. )

As to "the chips fall where they may"? That is surprisingly naive, David. The chips may fall where they may, but if you don't award according to some very obvious direction by some competitions----you don't get invited back again. Does that happen with every competition? Of course not. Not even most of them. But it happens.

Same subject: I've been in more than one competition (and what were considered reputable competitions, and were, I think, deserving of it) where a panel suddenly says, "Oh, you know we haven't given any gold medals at all yet. We better pay attention to that." Also, "They're expecting at least two or three gold medals from this bunch, so let's see if we can find some."

Ive also been in a situation where we gave a high award and our table captain came out afterward and said, "Wow, the back room crew is really surprised you gave that to wine #5; that might be controversial." Gaffe? Not supposed to be done? Nope. But it happens. What would you do in a situation like that? (We could have revised our scores at any time we wished.)

Another situation: a younger, somewhat naif judge was at a panel of four (always a bad idea, even numbered panels) and it was obvious that she was feeling a nervous. Regardless of what score she wrote down, she always hedged when it came time to announce her score until others chimed in with their opinions. She was almost panicked when it was her turn to go first on a wine, thinking she might be outside the norm. And I know for a fact she would change her verbal score from her already written score to conform to the group consensus.

Finally,

is it in fact true that the SF or Riverside competittions have awarded more total and gold medals as a percentage of entries each year of their existence?

That's not what I said. I did not specify specific competitions in my original comments, David.

You seem to be particularly vehement about defending competitions from any and all potential problems. Your experiences appear to have been supportive of that stance. Mine haven't; not always. I think, overall, competitions are a good thing. I think they've also often been overly abused and sometimes shoddily run. And the public needs to be aware of both sides.

David
Thanks for your response. I'd also favour the 'Chairman' approach, whereby one is an arbiter in case of disagreement, or in position to apply casting votes when needed. I sometimes wonder whether winemakers ought to be precluded from chairing the judges, especially those who enjoy the spotlight (who seem to provoke the most ire at their instructions).
regards
Ian

Hoke - thanks for the detailed reply. re Riesling first. one category for riesling - that isn't a serious position. it is even dubious for chardonnay. unoaked and oaked judged together? traditional wines with .2%RS along with wines with over 1% RS(yes, they exist). the wines are divided into flites. this is true whether riesling is one category or several. if it is one category does each flite include a variety of RS numbers. why would they be divided that way. the single riesling category might be ranked by RS by the computer and the first 10 or 20 be the earlier flite and the next ones the next flite - going to another panel - and so on. what is the PRACTICAL difference between having various flites of one category going to different panels and having various categories going to different panels. is there really a competition where every riesling entered is judged by a single panel? every chardonnay? it may be human nature to think that every category MAY deserve a gold medal; and in fact that is the correct assumption. but it is up to each wine to prove its case to the panel. i have seen Scott Harveys panel give 5 golds to a flite of 14 wines. i have seen no golds in flites larger than that.

and hay, if you don't get invited back because you didn't take the hint - shame of them and not the kind of thing either of us wants to be involved in anyway. these things will only be as good as the judges force them to be. at least one very big competition is already having trouble getting judges to return. the FLIWC has made major changes due to pressure from the judges.

one point i did not mention last time - it is not true that all competitions are for profit. some - like our michigan one - lose money but do it for the benefit of the industry. whoever wins will of course use that in promotion - but the winners are certainly not predetermined.

as regards the grade inflation - no you did not mention any particular competition - nor did i really ask that. i simply want hard evidence that ANY competition has in fact given more medals or more golds over the years. the SF and Riverside were just 'for instance's. you claim grade inflation but if all that means is that 'double gold has been added - i mean sweepstakes is the same as best of class and best of class is not something new - then ok. but if you claim that the NUMBER of medals is increasing - which you did - then give examples.

apparently we agree on the idea of the value of competitions in general if only they are run correctly. when judges say in writing that they will not judge at your competition because it is unprofessional, the bad eggs may be shamed into reforming. but instead of painting them all with that brush, lets try to bring everyone up to snuff.

Yes, Rogov, apparently things are done much differently on that side of the atlantic. we do in fact discuss the wines - and this does take time; and yes, we taste about 60-80 wines per day. there is no need to submit tasting sheets to lawyers. the panel submits the final absolute judgement of the wine. if they say Gold - gold it is - no ifs, ands, buts.... no changes later - period. how they get there is up to them - thats why they are expert judges.

so, forget tasting sheets. when they exist here, they mostly exist for the use of the judge - if they wish - to help them make their decision. same with scores. if a judge wants to score the wines and use that to decide what medal to give that is up to the judge. there is simply no use telling experts how to do their job. then the panel dukes it out and what they say goes.

(philosophically i would say that tasting sheets that end up in a score provide a false sense of objectivity. furthermore, it seems simply absurd to assume that an experienced taster doesn't know how good a wine is until he or she has added up points. ((gosh, i thought the wine was better than that; i'd better go back and change some of my points; or gee, that wine isn't THAT good, i'd better go back and change some of my points; or worse - well, it must have been that good, that's what the points say)).)

it might be fun for you to experience the american form of the competition - or ask for further info from your contacts who have. upon further thought, i now believe that your point #9 is the only real objection you have to competitions. the greatest wines in the world - which you have a fairly precise list of in your mind - a list that would be surprisingly similar to mine btw - are the ones that if there were any justice at all would receive double gold medals. only other wines that offer a fair approximation of the finest burgundies, bordeaux, etc. could be given such an award by a panel of expert judges. so what is it with all these double golds - they CAN'T be DRC class wines or even close. and the silvers aren't even close to 1er cru burgundy class wines. a reasonable wine competition of international wines would give some bronze medals, a few silvers and one or two golds. that would be fair. that would approximate the qualitative relationship between the greatest wines and the general run of things. and there is some truth in all this. but this is not what american competitions are up to. we may be mistaken - probably are in some sense - but we are doing something different. i tried to say what that is in my last post; but i may need more time to refine it. anyway, Hoke thinks we are not even doing that well; so.......

apparently we agree on the idea of the value of competitions in general if only they are run correctly. when judges say in writing that they will not judge at your competition because it is unprofessional, the bad eggs may be shamed into reforming. but instead of painting them all with that brush, lets try to bring everyone up to snuff.

Yep, we do agree on that.

but we are doing something different. i tried to say what that is in my last post; but i may need more time to refine it. anyway, Hoke thinks we are not even doing that well; so.......

But I don't agree with that.

I think, speaking in general, that many of the top wine competitions are well run. And many of them I've been pleased to participate in. I've learned something from each and every one of them. (Sometimes, what I've learned has not been all that positive, but then, that's the nature of things.) I think, over all, competitions do serve a purpose (although perhaps not the purpose they initially did a few years ago.

If they served no other purpose than for me and some of my peers to send clear and declarative messages to winemakers, that would be enough; sometimes "you go!" and every now and then "You totally suck at this and you should consider either having someone else make the wines or cut your losses and get out of the game, because you have no taste."

There are inherently a lot of built in flaws in the competition game though---not least the fact that with the climbing fees being imposed now only the big corp players can enter a whole slew of competitions and garner impressive sheets of scores, when the little guys can't afford to do that. So it's not anywhere like a clear and level playing field. But then, it never was anyway, was it? *shrug*

So in summary, I think competitions are good in general, but often flawed in execution. There are too many, for too many spurious reasons, and probably would benefit from a little weeding on the ground. And they don't necessarily point you to what's good and what's not.