Urban Spoon maintains a blog leader board, whose formula is unknown and whose influences are unknown. In that respect, it’s a bit like a tissue slice in biochemistry. In enzymology, some enzymes cannot be isolated but are only known by their enzymatic effects. Likewise, to a food blogger, the formula for scoring blogs is only known by making a review, placing it on the Urban Spoon system, and observing the effects.
I learned most of what I know about Urban Spoon scoring in a short span of time in which John Bickford and I reviewed Fogo de Chao and Gourmandises respectively, and then later, John reviewed Flip Burger Boutique. As a consequence of that second review, John’s blog vaulted from 14th to 8th on Urban Spoon’s blog leader board. After looking at FLIP (#1 on the hotness chart at the time and about #8 or #9 on the popularity chart), Gourmandises (#2 on the popularity chart, with about 650 votes at the time), and Fogo (#3 on the popularity chart, with about 630 votes at the time), I formulated the static model of Urban Spoon scoring.
The static model would say that a blog score (they call them views, but understand, Urban Spoon isn’t measuring blog views in any real sense) is the sum of all the restaurant scores that the blog maps to Urban Spoon. The restaurant score consists of two parts: there is a popularity component, roughly equal to the number of positive votes for the restaurant, and there is a “hotness” component, which is hard to factor but often easy to measure. At the time that John Bickford posted his review of FLIP, he gained about 1,900 points, despite the fact that FLIP had only about 250 votes at the time. In other words, the hotness component was worth 1650 points, or about 7 times more than the voting component. Currently, the hottest restaurant in Atlanta is Livingston Restaurant, according to Urban Spoon. And since Aspirations of a Southern Housewife has only reviewed Livingston, we can know that Livingston’s combined score is, on July 11, 389 points, almost all of it “hotness”, since almost no one seems to like Livingston.
Hotness is affected by blog reviews. I’ve seen this by observing the effects of Kokai Thai’s hotness ranking. It was pretty soon after Jennifer Zyman’s review and the AJC review that I reviewed Kokai Thai. Subsequently, Kokai appeared on the top 10, so it takes about 3-4 media and blog sources combined in a short period to boost the hotness of a restaurant into the top 10.
Even before I puzzled all this out, I had wondered often about the ratio of Urban Spoon views, which varied wildly from one blog to the other. You could have Tongue Sausage, which had 4,000 views on 10 reviews at the time, or someone more like BuHi and Eat Buford Highway, who was averaging about 30 points a review. This disparity begins to make a lot more sense in the static model, and it allowed me to categorize blogs by their Hotness Factor, their ratio of views to reviews, or their Hotness Ratio, which is the Hotness Factor of any blog divided by the Hotness Factor of Blissful Glutton, who as a professional reviewer, is a useful standard in the Atlanta market.
At the time (this was before Urban Spoon divided all scores by 4/3 ), a Hotness Factor of 50 or less suggested either ignorance or indifference to the scoring system. Examples included Eat Buford Highway, and many other regional reviewers. Hotness Factors from 50 to 150 suggested that they either understood what was going on, or made it a habit to review both popular and unpopular restaurants. Chow Down Atlanta, Amy on Food, Atlanta Foodies, From My Table all fell in this category. At the time Jennifer Zyman and Blissful Glutton was averaging a ratio of about 100. Blogs with Hotness Factors over 200, with the exception of Foodie Buddha, I tended to just not pay any attention to. It was obvious these people were focused on popular restaurants, and would not be the kind of place to dig out a hole in the wall. Ratios like 400 were unsustainable. These were hot restaurant chasers of such fervor they would soon run out of restaurants to review that could score so highly.
This model explains a lot of behavior. For example, the model explains losing views over time. Food bloggers, especially those who reviewed FLIP, must have been seeing their points deflate as FLIP came down off that astounding high, and now has fewer hotness points than Livingston. Gaining perhaps 100 voters hasn’t eliminated the deflation effect, and I suspect that Urban Spoon divided “views” by 4/3 just minimize the obvious nature of the upcoming deflation.
There are other changes, however, that I can’t explain with the static model. The day after Amy of Amy on Food reviewed Shoya Izakawa she gained perhaps 500 points. That was not possible if the static model holds. Likewise, the day after Jimmy of Eat It Atlanta reviewed Hop City, he gained 1000 points (more than the entire total of Eat Buford Highway’s reviews). I’ve been puzzling these two gains for some time.
Let me note that Amy was not the first blogger to review Shoya Izakawa; Gene Lee was. But in both cases, within a day or two after their reviews, both restaurants suddenly appeared in the “hotness” top 10. I don’t think that appearance was coincidence. A week later than Amy I posted my review of Shoya and the score I received was a small fraction of these enormous scores. I suspect anyone else that reviews Hop City isn’t going to get 1,000 points either.
Conclusion: the scoring isn’t entirely static. It also has a dynamic, timing component. Get a review of the right up-and-coming restaurant, and get more points than BuHi will get in his lifetime. At one level, I’m very annoyed, as the static model has a nice mathematical feel to it. Your score would be a map of coverage of the restaurant universe, as defined by Urban Spoon, weighted by the value of the restaurants you reviewed. But when 1/6 of a blog’s total score is purely an act of verve, or luck, or for all we know, a bug in the scoring model, then any pretence to objectivity in the model is lost. And, likewise, the predictability of the static model is lost. Jimmy of Eat It Atlanta may be the kind of guy who focuses on openings and grand restaurant theater. Or he may be just savvy and lucky. Who knows? I surely don’t.
Update (7/12/2009 ca 10pm): I’ve made some corrections to point values in this article. I can’t validate 800 points for Amy’s as I really don’t have any data in that period. 500 seems more likely after some review. The original Hotness Factor where I would judge a reviewer as having a major focus on hot restaurants was around 200 back in the day. With Jennifer Zyman’s current hotness factor of 70, 150 would now be a fair guide for either (A) – a major focus on hot restaurants, or (B) some luck with dynamic point gifts. Converting to Hotness Ratios, which survive any multiplication or division of points, the interpretation of blogs would be:
Hotness Ratio — Interpretation of blogging style
0.5 or less — Indifferent to restaurants that Urban Spoon sees as popular. Perhaps a focus on niche or ethnic restaurants.
0.5-1.5 — Reviews a variety of restaurants, including popular or unpopular restaurants. May or may not pay attention to new restaurants.
2.0-3.0 — Focus on restaurant openings, new and popular restaurants, some luck with “dynamic hits” on ‘up and coming’ restaurants.
above 3.0 — Unsustainably high focus on new and popular restaurants. If continues, score ratios will deterioriate over time.
Food Near Snellville 
July 12, 2009 at 8:00 am
Wow – great analysis. Early on in my participation with Urban Spoon, I noticed that my “ranking” seemed erratic at best. I attributed it to the niche nature of my site and the fact that many of the restaurants I reviewed were not in their database – I had to request they add them.
Delicious Kabob is a good example – I had to ask 4 times to get them to add it. I believe my ranking actually dropped after adding the first review. With your “hotness” idea, if I had reviewed after Bliss (exactly one month later) I would have shot up rather than down…
I’ve actually quit paying much attention to Urban Spoon (I forget to add my reviews…). I do get regular traffic to the blog from them, but it’s rather low and never seems to grow. I’m thinking it’s really only relevant if you are covering a broad spectrum of cuisines. As a niche blog, I’ll remain a boundary condition.
I’m content with my low “Hotness Factor” – I’m indifferent, in spite of my ignorance. 😉
July 12, 2009 at 10:41 am
BuHi,
Thanks for the kudos on the analysis. I figured it had an audience of maybe 5 or 10, and was pointed enough to upset people. But in all honestly, guys like you and Gene Lee need to see this stuff.
I can understand the problems with shrinking ratings. That’s likely because when you reviewed them, you got a “hotness” component added to the rating, and then it shrank over time, as you ended up the only reviewer. I’ll note your ranking increased for a while too, as Quoc Huang became popular. I ate there, reviewed it, and soon after Chow Down Atlanta did. Your ratings rose for a while in that period.
It’s very clear the “system” neither encourages nor wants regional reviewers. Guys like you or Savannah Red (and me actually) are outliers in their system. They want people to descend upon the “hot restaurants” of this world and declare them exceptional; and in the process they snub the unpopular restaurants, the popular restaurants that aren’t hot, and every reviewer that wants to review a spectrum of restaurants. And of course their system doesn’t prevent you from doing anything you want, but it doesn’t encourage it either.
FnS
July 12, 2009 at 11:31 am
Exactly. I’m a natural born cynic, so I tend not to trust “aggregator” sites in general. Rating systems like theirs, as you point out, do have an agenda. I do like that Urban Spoon allows you to simply post your blog article link, rather than making you write a “new” review and couch a link in there (like Yelp).
I think simply clicking on a blogger at Urban Spoon and looking at their list of contributions allows a savvy reader to make their own evaluation of credibility – the arbitrary “scoring” is pointless. I don’t want to go to a restaurant because it’s popular – I want to go because it’s good. If I followed their recs, I would have had to eat a FLIP for 2 months straight. Clarification: I’m sure Blaise is a great guy and his food is good, but me being the boundary guy I am, I won’t go there BECAUSE it’s so popular. I know I wouldn’t be able to enjoy the experience because of the trend followers…
July 12, 2009 at 2:03 pm
My gut feeling is that a savvy reader would take into account more the number of reviews than the score of the reviews as a measure of the blog. It would also help if the blog author is reviewing a diverse collection of restaurants.
I think it would help Urban Spoon if they added a scoring component for adding restaurants to their system. It almost seems as if they punish people for adding unknown restaurants to their network.
I haven’t mentioned the mess they have made of Tucker restaurants. More than half of them are still “located” in Stone Mountain, too many for any one person to easily correct.
And the “because it’s popular” – I feel that too. Too many times I’d go downtown and spend an hour trying to get to the mall at Lenox. It’s a pain. It’s probably why I haven’t tried Fogo yet myself. And when I did review FLIP there was a reason I called it the Hajj of Atlanta Food bloggers. It was an allusive reference to its outsize effect on rankings at the time.
The flip side is, by being counter-popular, you’ll miss a few places like Gourmandises or Alon’s, both of which have some upscale appeal but really aren’t terribly expensive.
July 12, 2009 at 6:26 pm
Good point about the counter-popular – I try not to do that to a fault, but sometimes it’s unavoidable. Occasionally, I’ll take a hiatus from reading blogs with restaurant reviews, simply so I can discover on my own (with out someone else’s preordainment).
July 12, 2009 at 1:41 pm
Interesting analysis. I’ve always wondered about the ranking system. I will continue to review places on and off the beaten track no matter what, since I enjoy it all but I appreciate knowing more about hot “hotness” affects the ranking.
July 12, 2009 at 2:07 pm
If I’ve helped people understand a bit more, that’s a good thing. And I know you review almost everything at hand. Nice to see the reviews from Asheville, NC, or Charleston SC on your plate.
July 14, 2009 at 12:33 pm
Hi all. I’d like to say that we at Urbanspoon really appreciate this kind of thoughtful feedback and have been discussing it in the office this week.
Our blog leaderboard calculation is actually simpler than you’ve been guessing: It’s a sum of page views in the past 30 days for every restaurant you’ve reviewed. It was intended to be fun, so we didn’t put the time into constructing a more complicated algorithm. You’re right that this system has the unintended side effect of encouraging bloggers who care about their ranking to review more popular restaurants. We’ve talked about changing it for a while.
We’d be interested in the algorithm you’d construct (for the blogger leaderboard) that would be more fair. I certainly agree that we want to reward bloggers who review new and out-of-the way restaurants, because that content is valuable too.
Thanks also for pointing out how entangled Tucker and Stone Mountain restaurants were. I’ve fixed that, so take a look and you should find a much clearer delineation between the two areas. If you notice something like that again, send us a feedback (using the link at the bottom of any page) and we can fix that stuff in bulk.
July 14, 2009 at 7:52 pm
Kate,
There is no way you’re using the sum of page views for 30 days for every restaurant a blogger has on your system. For one, Urban Spoon doesn’t have the means to measure page views. The only tool for measurement you have is the Urban Spoon icon a browser will fetch the first time a browser hits a review. After that the icon is cached. At best you can underestimate my traffic. That’s item one.
Item two is the day-to-day changes I’m seeing cannot be explained by my hits. My hits are a matter of public record; see http://quantcast.com/foodnearsnellville.wordpress.com (it underestimates by 50% or so, but it’s good enough for this argument). And if today is day 1, yesterday is day 2, day before yesterday is day 3, then the difference in score between today and yesterday is equal to the hits on day 1 minus the hits on day 31. This difference has a very simple interpretation. I’ve had too many days where I’ve had daily changes in excess of 300 points, which is far beyond the largest number of hits I’ve had so far. That would be mathematically impossible if you were using views.
Item 3 is that the only times we see big changes is the day after reviewing a popular or hot restaurant (ignoring dynamic events, which also only happen the day after a review). That doesn’t make sense if you’re really counting views.
Item 4 is that those big changes tend to be, over a short period of time, predictable and repeatable. If John Bickford gets 1,900 points for FLIP back at its peak, I also got close to 1,900 points when I reviewed it. That shouldn’t happen if you’re really measuring views.
So in short, your company is clearly not measuring views and your ‘simple’ explanation is incorrect.
You don’t need a better algorithm to be fair. That misses the point. The fair approach would be to honestly explain how you rank bloggers. Every one understands that Urban Spoon is ad driven. If I were to build a ranking system, it would be designed to promote behavior that makes my company more revenue. But I’d tell people that.
In terms of the Tucker/Snellville, from the day I found Matthew’s Cafeteria in Stone Mountain I mentioned this was a problem, that there were a lot of restaurants that were miscategorized. I’d say I mentioned it as least 3 times. But until now, it wasn’t a priority.
Getting back to fairness: For people like Eat Buford Highway, getting the spotlight every now and then might help a lot. I recall in a survey you recently sent out where that idea was proposed.
When I taught school, I pointed out people who scored well on tests, and also I pointed out “most improved”, to get to the guy who might have gone from an F to a C on his own initiative. I had the most fun explaining to shocked parents that they couldn’t have cheated because their answers were too original to have been copied.
July 15, 2009 at 12:52 pm
I should have been more clear – here’s what we calculate: For every restaurant you review on your blog, we count the page views of the restaurant page ON URBANSPOON (not blog page views, which as you said, would be problematic).
As far as the motivation for the ranking system, the goal really was to engage bloggers and improve the site. You may not believe me when I say this, but the truth is we’re a small company that cares about our users and keeps costs low so that we don’t have to play such tricks to make a buck. Our priority is to make a high-quality site, first and foremost, with useful content. If we do that well, the money follows.
We’ll likely be revisiting the blogger ranking algorithm soon, because it’s not doing what we want. I’d still like to hear any input you have on that.
July 15, 2009 at 2:35 pm
Some quick thoughts:
1. Views from the Urban Spoon site actually makes some sense. I can see the correlation between votes and views, since to vote for a restaurant, you have to view the page. But one flaw in this concept is that the day after I post a review, I get credit for 30 days previous I really didn’t ‘earn’. If I got the views over the next 30 days, that minimizes the immediate reward but eliminates a lot of the “losing votes” effects that discourages bloggers.
One trick which would always yield positive votes and increases would be to start crediting people with views from the day the new review is registered and then give them points for that review for the next 30 days and no more. Their votes never go down, they never lose votes, but to gain more votes they need more reviews. Downside would be people who review restaurant X for a year or two, then X gets hot and someone posts the 5th review of X and gets huge #s of points. But perhaps “bonus points” for an old review could be awarded if something like that happens.
Simpler to implement would be something like: NewScore = Max( OldScore, Sum(Last 30 days)).
2. if you use an abstract “score” then you could provide scoring in a few or multiple parts.. one part could be the traditional views, as you do now. One part could be for “original content”.. giving someone credit for introducing restaurants and talking about them..I’d like to think number of reviews should also count, but with some compression for places like McDonald’s. If I went out and reviewed 10 McDonald’s, I shouldn’t receive the same credit as BuHi for taking on 10 original restaurants with different cuisines.
I’ll note you rank Urban Spoon site users in several ways, and doing much the same for bloggers would allow particular emphases to shine, even if another component of a leaderboard summary isn’t their forte. Examples: BuHi does very detailed reviews of ethnic restaurants, Amy on Food and Chow Down Atlanta have good photographs with their content. Gene Lee does fantastic food articles, etc. Foodie Buddha has a nose for restaurant openings. Your “Prime” users would be a good source of ideas, in this regard.
For example, where I’m at, on the edge of a city, I’ll usually not be the first to get to the restaurants that feature name chefs. I will probably be one of the first bloggers to touch on restaurants in Lawrenceville, or maybe find a dive in Loganville with really good barbecue. Categories chosen should allow all kinds of bloggers to look good.
Last, if I’m typical, most of the Atlanta bloggers are doing this because it’s entertaining. I find out new things, I get to show my wife a cool new place I never tried before. So, as long as it’s light hearted and shows bloggers at their best, you’ll probably have a lot of support for a multi-faceted leaderboard.
July 14, 2009 at 4:26 pm
UrbanSpoon accounts for roughly 3% of my traffic, which is a little less than the combined amount I get from all you lovely folk who are nice enough to include me on your blog roll and in your posts (thanks YA’LL!!!)
To be honest, I have little interest in their ranking system. I add in the links on the posts, and that’s about as far as my interest grows. It seems that I hit up most of the new restaurants in the ATL for just that reason: they are new. However, I’ve spent just as much time eating at long established joints and places that are far from “hot” (I can’t imagine Blue Ridge, Apres Diem, and Henri’s being considered hot places).
As a person who deals with data-aggregation on a daily basis, I find that the simplest systems seem to be the best. Meanwhile, the numbering system they have come up with is pretty useless.
To be honest, my biggest concern with Urban Spoon is those little “like” and “doesn’t like” comments they append from time to time. More often than note, I want to delete them because they don’t accurately reflect my opinion. I’ve seen my ratio of views/post plummet as my content repository has grown.
Though I’m not actually concerned about much of this at all (since there is no revenue stream), if I were to worry about it – I’d be much more concerned with my PageRank, inbound link results, and the like.
Great post, and nice to see someone in the “foodie blogging community” put some thought into something else besides the food we eat! Keep it up.
July 15, 2009 at 12:23 pm
Agreed. I get way more traffic from the link on Food Near Snellville than I ever get from Urban Spoon. There rankings are useless – I quit adding their widget to my posts a while ago – more effort that reward – especially when you have to repeatedly request that they add a restaurant, then wait a few days to go back and pick up the widget.
I still love that I have more posts on US than 13 of the blogs that are ranked higher than mine. If I had even less of a life, I might try to see if I could work out how to have the most posts in ATL and the lowest US ranking. Now that would be cool.
July 14, 2009 at 4:27 pm
post script – please excuse this poor grammar and repetitive tone of my last comment – i just wrote and submitted … with no editing 😉
July 15, 2009 at 3:56 pm
“Jimmy of Eat It Atlanta may be the kind of guy who focuses on openings and grand restaurant theater. Or he may be just savvy and lucky. Who knows? I surely don’t.”
LOL, I mostly started linking the urban spoon reviews when I did the pizza tour. None of those were new, and they are most certainly not grand!
I think lucky may be more appropriate than savvy.
I just realized I didn’t have you linked on the blogroll – that has been rectified.
November 21, 2009 at 9:08 am
Very interesting read. I’m going to have to do some experimentation.
It’s sad that UrbanSpoon rewards bloggers for writing primarily about restaurants everyone else is blogging about.
November 21, 2009 at 11:23 am
As Kate mentions above, they primarily reward people for writing about restaurants the people on their sites are reading about.
And yes, there is a powerful “me too” factor when a restaurant gets hot. Currently Grindhouse Burgers is hot in ATL, but it’s only open for lunch and it’s about an hour’s drive away from where I work. Fat chance I’ll get a shot at it.
The “veteran” bloggers with a lot of posts tend to go for new restaurants. They’ll mix in some old favorites and do some “me too” stuff, but as new restaurants bring in readers..
A high ranking is nice, and nicer when you’re starting, but readership is nicer.
FnS.
January 21, 2010 at 6:15 am
Very interesting post! I came here trying to figure out what the heck Urbanspoon meant by “views” because I knew my blog wasn’t getting 1,000 page views.
January 21, 2010 at 9:58 am
A lot of this confusion could be eliminated if the word “views” were a link with an explanation of how it all works.
January 28, 2011 at 10:51 am
[…] Hotness Factors and Urban Spoon: Is the blog leaderboard fair? […]
August 9, 2013 at 1:24 pm
Thanks so much for your analysis! I’ve been really confused with Urbanspoon rankings…so this was a good read…though it doesn’t help with the frustration to see my ranking actually go down as I post more (I post every other day!)…My ranking has gone up to 140 down to 170 in one day…and not encouraging to know that a blogger who keeps his rank hasn’t even posted anything for 3.5 months now. I’ll keep blogging though… 🙂
March 27, 2014 at 10:25 am
[…] Want a little more light reading? There’s an interesting (and kind of confusing to me at least) blog post about the “hotness factor” of blog posts and UrbanSpoon rankings from 2009 on Food near Snellville. […]