This summer I had the opportunity to participate in a weekend hackathon where my team of 5 set out to build an entire web application, from scratch, within 24hrs. While the task itself was daunting and overwhelming, we felt confident we had a solid idea: to build a site that used crowdsourcing to verify the effectiveness of medical prescriptions.
The team met a week before the hackathon to review our idea and how to best present the information that was collected via crowdsourcing – there were many ideas brought to the table, but most of them focused on the most common method used to collect crowdsourced data: comments. We sketched some ideas for how to display our comments and how they would correlate to the associated prescriptions, but something always felt off and incomplete about this approach. Finally, the question that identified our hesitation to this approach was raised to the team: are comments really that useful?
We eventually went in a different direction with our comments approach, and ultimately we had a successful hackathon, but the question always lingered regarding the effectiveness and usefulness of comments.
I decided to tackle the question of the usefulness and efficacy of comments, and to that end I looked into three sites where comments are integral to the site visitor’s experience: The New York Times, Yelp, and Reddit. I looked at a few key attributes that I could use to help assess the usefulness of comments, whether the site was moderated, how the site was organized, and overall effectiveness of the integration of comments to the site’s stated purpose. Below is a summary of what I found.
New York Times
Last December the NYT changed a few features to their popular commenting system. A notable change allowed certain people to become ‘trusted or verified’ commenters so that they do not have to go through the standard process of waiting up to 24 hours to have their comments reviewed, approved, and posted. In return for being a verified commenter, these people must use their real names, which are verified through Facebook connect (there actually was a lot of backlash regarding using Facebook for verification purposes since many people are uncomfortable with Facebook being the internet’s ‘drivers license’). This new addition of a verified commenter was a slight change to their often criticized, overly-moderated approach.
Another change was how comments were posted: they added threaded comments, which allowed people to directly reply to a comment, and the addition of two tabs: ‘Readers Picks’ and ‘NYT Picks’.
So the question still remains, are the comments on the NYT effective? In a nutshell I believe the comments related to a NYT’s article does offer value to the article, and its moderated approach is one reason for this value. While I am not advocating censorship (mainly because I do believe the NYT is quite liberal with what comments they will approve for display on the website), I do believe that having comments reviewed for quality does ensure that the reader is thoughtful with his/her feedback. While this might not be the best approach for all sites, it does make sense that an industry leader in world news imposes some quality standards to its blog comments since these comments are now a significant part to the original news post. It also was a step forward to add threaded comments since the site visitor can now decide/control what to further dig into.
Yelp, the popular business review site that launched in 2004, was one of the original pioneers with crowdsourcing comments. It allowed anyone to conveniently review restaurants, businesses, etc., with the aim of using real reviews by real people as a valuable barometer of quality for their site members. Interesting enough, it is now 8 years later and Yelp’s original approach to include reviews by everyday people is still its primary purpose and drive of the site. Mostly the site is un-moderated (comments flagged as offensive are removed), and people are free to add as little or as much as they want. Unfortunately, with regards to the effectiveness of comments, undefined and unqualified comments are both an asset and a hindrance to the popular review site.
While I am still in the habit of checking Yelp for overall scores for newly opened restaurants, I must confess I am a bit skeptical of whether these reviews accurately reflect the quality of the establishment. For example, a recently opened restaurant around the corner from me receive 3 out of 5 stars, which according to Yelp means ‘A-OK’, but upon further reading of the reviews, it became clear that about half of the comments were irrelevant, trivial, and non-food related (e.g. one person was upset that the bathroom had gel soap and not foam soap).
Yes, overall Yelp is helpful and influential (this Harvard Business School Case Study Abstract provides some insights into the site’s efficacy), but I question if a little more handholding and guidance would elevate the quality of reviews on the site.
While you might not be a fan of the atheist, cat-loving, ‘omg I cannot believe you just said that’, influential website, you should be a fan of how well it tackles comments. Started in 2005, Reddit is a site that allows users to post submissions using a simple, stripped-down, bulletin board system (BBS) interface. Submissions run the gambit from cute puppies, to Ask Me Anything (AMA) sessions where famous people, including President Obama, answer questions posted by the Reddit community.
The true beauty of Reddit, lies with its community-controlled commenting system. Essentially, people can up-vote or down-vote a comment, which collectively ensures that the ‘best’ (best being subjective since it could be the best in regards to humor, or the best in regards to a explanation to a question) is the top comment associated with a post.
Additionally, not only is the order of the comments user-controlled, but the order of the submissions themselves are also controlled by the end-user. Like the comments, the posts are up-voted or down-voted, ensure that the ‘best’ (again subjective) are at the top of the page. You can even control which category (aka: sub-reddits) your submission is posted into (e.g. cute pictures are often posted in the ‘awww’ sub-reddit and science-specific submissions are posted to the ‘science’ sub-reddit).
The primary takeaway with this simple, user-centric commenting system is that quality and relevance is the primary indicator for how the site is ordered, as opposed to date or another attribute, which would be irrelevant for this site. This format also has the added benefit that each time you go back to the site, the page will be different (slightly or significantly) due to what was voted upon while you were away from the site.
In summary, there is a lot more to comments than just writing a few sentences and hitting ‘post’. The secret to a quality comments section is multi-faceted to include: threaded comments, guidance, user-controlled voting with regards to ‘best’, and when necessary, slight moderation. Without something that elevates the comments section past a venting, obnoxious, verbal stomping ground, the comments section will just remain as an unneeded, irrelevant, often-skipped section.
What are your thoughts about comments? Please post a comment