One of the most important drivers of an online internet community is efficient rating system. It suppose to promote the most appropriate content and slow down (or even block) any low quality content. The same applies to contributors or community members. Good guys should be promoted, bad guys - flagged (or even banned).
There are lots of social networking communities, which adapted one of the rating systems. It would help to review types of these systems and understand pros and cons each of them, as well as what these systems promote and what kind of behavior they impose on users.
In this article I’m going to review the most popular types of rating systems, highlight their positive and not so positive sides. What is outside the scope? Any rating system, where human votes are excluded. Automatically calculated scores and ranks (for example, alexa ranks) are not covered.
There are four the most popular rating system types.
Five Star Rating System
cons: limited granularity, context dependant, subjective
Usually this rating system applies to consumer goods, movies, services, or service providers. In the most cases one star means “very bad”, two - “bad”, three - “ok”, four - “good” and five stars - “excellent” (-2, -1, 0, +1, +2); in rare cases, stars mean only positive outcome (+1, +2, +3, +4, +5).
Highly depends on number of votes, ideally, should also display it. Not very efficient when applied to anonymous members of community.
Interpretation is tied up with meaning of number of stars. There is also a distance between users and rated items. This type of rating systems stimulate consumer point of view in users.
Shortcomings of this method are: a) if there is a big range in ratings (say, one guy gives the item one star and other guy - five), average value is not representing either of them; b) with small number of voters can easily be abused.
Popularity Rating System
cons: One dimensional, limited use
This type of rating system is very simple: one user has one vote which can be given to certain candidate. Usually, it’s the way to rate links, news, stories, TV shows etc. The item’s score is directly linked to number of users, expressed their interest in it. Indirectly it says about quality of the item, but only for this particular group of people.
This is also a good measurement for members of community, when is calculated based on objective data, such as number of visits of personal profile.
Interpretation of big number: lots of people seen it and marked as interesting;
Interpretation of small number: almost nobody seen it OR lots of people seen it, but it’s not interesting for them.
Big number tells us that the content is good. Small number doesn’t really tells anything.
Users of community with that kind of system are stimulated to be more active, to explore more and to form a circle of friends or people, who may have similar opinion.
Negative side of this type is support of “oldtimers”: “older” members of community with more history, friends and connections have significant advantage against new members, not matter what quality of the content they offer.
Dual Rating System
cons: open to interpretations, inaccurate near neutral value
It’s either “agree/disagree”, or “plus/minus”, or “interesting/boring” or something like that. Well, usually people either like something which they know or not.
Which means, numbers close to 0 can be interpreted as “people don’t care and don’t vote” OR “it’s very controversial, lots of votes got distributed equally”.
Not the best system for consumer goods, but very good for the cases when there is a need to know how people think “an mass”.
How it can be fixed? By supplying number of votes (as amazon.com currently does for the reviews).
Members of community with that kind of rating system usually more polite and intelligent than others (at leastm they display such qualities).
Value Rating System
cons: context dependant, limited use
When people tag other people (or items) with verbally expressed ratings (or ranks), it’s value rating system. It defines, what is the value of certain thing for group of users. It’s very relative and context dependant.
Excellent contributor within one community may be tagged as a “very bad guy” in another. It’s more or less objective, because it represents how this particular thing (or user) happen to correspond certain value system.
This type of rating can be used only in closed groups of communities with their own well defined value system. Members usually try to be objective.
Conclusion
What is the best rating system? There is no “one for all” system.
Depending on content, community goals, type of rated items and other factors, it may be either of the listed above or even a combination of a few of them.
3 comments ↓
You missed the two systems that actually work.
First, Bayesian filtering. Like imdb top lists. Technically it is averaging, but with a large number of “it is average” votes thrown in. So the only way something can get rated highly if it is popular and people like it. There is some very insightful mathematics behind this very simple method, but basically it avoids the effect where items with few votes can easily has insanely high (or insanely low) scores.
The second are all kind of clustering. Like StumbleUpon, last.fm, Pandora, and reddit’s recommended (but not the main reddit). There are many algoritms to do so, but basically only votes of users with similar voting pattern to yours are considered (or they get much higher weights).
Some time ago, I have written a bit about such systems.
taw, thanks for the addition!
Nice overview! I never really thought about this stuff in detail.
I noticed a lot of sites implementing voting systems though and I think it’s very useful b/c of what you said “Good guys should be promoted, bad guys - flagged (or even banned)”.
You must log in to post a comment.