Right now impact is set by the author of a guide. Will this be sustainable, or will people keep upping their impact to get above the noise (since most people think that whatever they are doing has a bigger impact than other people, which is why they do it)?
Would a system that votes on the impact or in some way community corrects it be desirable?
Also, I think there should be more guidance on how to set impact, especially on things such as supporting a given cause. Since it is currently defined as the impact that one person following it has I would expect that these would each have very low impact (except for the one push that forces it over the edge) even when the end result if the cause moves forward is high. What are other people's thoughts on how to set impact in these situations?
I think there should be a rating system where impact is 1/2 the guide writer's input plus 1/2 the average of all ratings by other users. This would allow for the community to "correct" an impact if it has been set far too high, but it also allows the person who wrote the guide to have a say in the matter.
As far as supporting a cause, etc - while it's great that people support causes, I think that a big idea of this site should be to get people doing things, not just being happy that they're supportive. There are many things that people can do to help a cause if they really want to have a higher impact, both on SixLinks and in real life. Reply to comment
And just adding - 1) Sorry to Steven, that impact rating algorithm isn't very pretty, and 2) wanted to clarify that I love it when people support causes, I just don't think that it should be rated as high-impact unless action is being taken. Reply to comment
Just tossing in that this is an area we're actively debating, so please add your input!
One note as far as impact. We will (eventually) show both the impact of doing a guide once (in a fix), and the total impact of the guide (impact * number of completed fixes).
The issues Sam raises are one's we're currently thinking through, and exactly how much an author's view of the impact, difficulty, and quality of the guide affects its rating.
I would not use the authors idea of the quality of his/her guide beyond determining whether it is in a complete state. An author has a very good understanding of whether the guide is in progress, a rough draft, or in a presentable complete state. It might make sense to add a "draft" flag to show that it is not complete separate from the quality metric.
As to the quality of the guide beyond that, the author is probably the worst person to ask.
Is the impact scale supposed to be linear or log? ie does a 5 have five times as much impact as a 1 or 10^5 (100,000) times the impact (or pick a base if you don't like 10)? I would lean more towards a log scale, since in the current lingo five people surfing the web will not get to the top of mount Everest.
Is there any way of dealing with the fact that every fix to a given guide will have a different impact? If you live next door to your office biking to work will have less influence than if you live 10 miles away etc. Or is it better just to accept the fact that there will be some inaccuracy from this and move on.
Quality/completeness - those thoughts make sense, and some of that difference we'll gather with the 'user rating' field.
At the moment, my rough thoughts of the 'goodness' of a guide is a combination of its: user rating, times completed (excluding the author), and times favorited.
The impact scale is roughly log, which makes calculating the total impact very very nice.
As far as varying impacts, there's two ways we deal with that: one, the fix author picks their own impact. Two, we use a log scale, and for the most part, I think fixes will probably be within an order of magnitude of their guides, so we accept it and move on.
It might make sense to prevent user's from rating a guide until the author lists it as complete. This way you won't get low ratings early on that skew the results.
Also, is the user rating system implemented yet?
Will the rating system track (at least internally) which ratings are from who? If it does this it would be cool to also display to a user what they rated a guide as and allow them to change it if the guide improves/ devolves.
I assume there are/will be safeguards to prevent rating spam. If these aren't there yet I would suggest that rating of a given guide be limited to one vote per person, and that in order to vote, you must create at least one fix or guide. This will increase the incentive for people not to create junk accounts to up their ratings. Reply to comment
The prevent ratings makes sense if a guide's listed as a draft, etc.
The user rating system is done, but we don't yet have pages where you can enter them in besides our admin site. They're coming for the regular site, but we're looking to nail down how people want them to work first.
Ratings are tied to a user, and I think being able to edit your reviews makes sense. You also only get one.
As far as preventing spam, it's a balance between ease of use and quality. We already have a pretty high barrier in that you have to sign up and confirm an email address (may add captcha if we start to see junk accounts). A time limit (only after you've had an account for 24 hrs, etc) may also make sense, but it's also possible that we just let things roll, and count on the wisdom of crowds to sort things out. Reply to comment
So jeff and I just finished our call (all four hours' worth), using your feedback, and after discussion, decided that we needed more input from all of you smarter minds.
Guide quality will be revised to simply "draft" or "ready for use" - draft guides can't be rated, and are filtered out by default when searching.
Impact, however, is a trickier subject. We talked over everything from impact being entirely community rated to it being set for each guide by jeff and I (that idea was tossed pretty quickly). We talked about using flags to report inaccurate impact (to whom? and what do they do?), and in all, we never found a solution that was entirely satisfactory.
So, we bring the question back to all of you. Here's why impact is important, and what we hope to use it to do.
Some tasks (becoming vegetarian) have a lot more impact than others (changing a lightbulb). In many cases, these differences are orders of magnitude apart. We want to provide people a realistic idea of how much a particular fix will really impact the world, to combat the feel-good just-recycle-and-everything-will-be-fine environmentalism that ultimately makes people do less, not more.
So in that way, impact is critical to providing a good understanding that certain changes in lifestyle or action have much larger consequences than others (and that the sizes might not be what you'd expect).
The challenge is to do this in a way that is easy to use, and accurate. We also have to prevent malicious/misinformed-crusader-types from marking fixes much higher than they actually are.
Some ideas: * Leave the system as-is, but fix the names. * Allow them to be flagged, and manually adjusted by Jeff and I (prefer not) * Allow the community to vote (do they know more than the author?) * Have a combination of the author, community, and invisible hand of site operators(eek) * Author only * Community only * Something completely different.
I would definitely try to keep the admins out of the loop as much as possible. Once the site takes off, the maintenance would become intolerable.
In certain categories/ types of fixes you could use some objective measures (how many kWh does it save per year, how much CO2 equivalent does it keep from being emitted, but these may be hard to figure out for some fixes). Even so, having concrete values along with the impact "phrase values" might help people place impacts better. It would certainly help discussion about what impact something should be rated as. Perhaps an impact guide would be useful for this.
If you can have the website flag guide when being submitted that has a impact 2 or more than greater than the difficulty (This shouldn't happen very often, unless it's something like don't play with dynamite and uranium in your back yard) and have a confirmation box like the following pop up:
"You have stated that this guide will have a very large impact for very little effort. If this is true, we welcome it and look forward to implementing it. However, please consider whether a single person completing this guide will have this level of impact. If the guide is asking people to support a cause, remember that you are only counting the impact of a single person, not of the causes total impact"
I would have the author put their initial impact in. Perhaps a weighted average along the lines of what Rachel suggested (I would use the Author= the higher of people or 25% of the total rating rather than the 50%, but that is details). You could also require comments (either anonymous or named to accompany ratings). Reply to comment
Agree on keeping as much as possible of the admin's plate.
I really like that idea on impact >= 2 + difficulty - tying this in with scale could make an even better algorithm.
The other issue to handle is abuse (setting my guide to 6* impact because I can), but this may be handled ok with the report offensive/abusive flags we're putting in.
As far as getting better "phrase values", this was one thing that we agreed needs improvement.
Perhaps, like the difficulties, we should base them on common tasks: Completing this fix has about as much impact on the world as: * using an energy efficient lightbulb * driving a hybrid car * switching my city to clean energy * making a 200 square mile area of biodiverse land a protected area * switching my nation to 70% clean energy * creating an international carbon-trading system
The phrase values can provide a good first order of magnitude guess at where to put something. But trying to equate across the categories will be tricky. How many whales not hit by ships = 1 million kilowatt hours of energy saved? etc
I was trying to get at rather than replacing the phrase values, augment them with concrete values to give a reference if the exact savings/effect can be estimated. This could even be a pop up window next to the impact bar named something like "help valuing your impact" that pops up a page benchmarks for each of the major categories.
So something like:
Transportation * Reduce gasoline usage by 10 gallons a month ** Reduce gasoline usage by 100 gallons a month *** ... Making Energy * Reduce the CO2 per kilowatt by 100 lbs (making up numbers here) Using energy * Reduce energy usage by 100 kWh/ year (again making up numbers) ** Reduce energy usage by 1000 kWh/ year (again making up numbers) *** Reduce energy usage by 10000 kWh/ year (again making up numbers) ... etc
This would at least give objective metrics to judge against if there is a dispute over an impact rating.
(Also, minor point, if you want to go international, saying switching my nation to 70% clean energy is not a very objective measure) Reply to comment
Ahh, makes sense. The real challenge in doing that (other than keeping information overload down) is finding numbers that people understand. Contextualizing and giving concrete values I support strongly, and after visuals (big picture, where we actually show all the numbers, etc) come in, this may make a lot more sense.
(70% isn't too bad - very few nations are anywhere near that. hell, it should probably be above the carbon system. ) Reply to comment
As far as the flag-if-impact-seems too high, a formula that made a lot of sense to me was: if 2 =< ( (impact-difficulty) + (impact - scope) ): # flag it
Currently it looks like you favorite a guide automatically when you use it/ make it. Is this intentional? If it is, number of times favorited becomes less useful for evaluating goodness. Reply to comment
I guess my concern is I may want to make a fix off a guide which isn't well written (but has a good concept). In this case I wouldn't want to give a favorite vote to it.
If it is auto favorited, that also starts becoming essentially number of people who started the guide. Which will be similar numbers to who completes it. Reply to comment
But you probably would want to know if the original author updated their guide, thus the auto-favoriting.
Also, I'd say that # of times favorited is going to be a small portion of the guide's ranking. # of times completed is the number one criteria, by a huge margin. Reply to comment
I think what Sam meant by "switching my nation to 70% clean energy is not a very objective measure", is that some nations are much smaller than others, and some use much less energy than others. Changing the US to 70% clean energy would have a far different impact than changing Jersey or Macau or Jamaica (or even Germany) to 70% clean energy. Reply to comment
If you don't have an account, you'll need one. We recognize that this is a pain, but we do it to keep spambots from flooding the site. If this really bothers, you, please let us know. We're listening!
Get the Facts
Do Your Part
Me
My Part
Following
Community
Talk
Even More
Login or Get an Account!
Support SixLinks
Would a system that votes on the impact or in some way community corrects it be desirable?
Also, I think there should be more guidance on how to set impact, especially on things such as supporting a given cause. Since it is currently defined as the impact that one person following it has I would expect that these would each have very low impact (except for the one push that forces it over the edge) even when the end result if the cause moves forward is high. What are other people's thoughts on how to set impact in these situations?
Reply to comment
As far as supporting a cause, etc - while it's great that people support causes, I think that a big idea of this site should be to get people doing things, not just being happy that they're supportive. There are many things that people can do to help a cause if they really want to have a higher impact, both on SixLinks and in real life.
Reply to comment
One note as far as impact. We will (eventually) show both the impact of doing a guide once (in a fix), and the total impact of the guide (impact * number of completed fixes).
The issues Sam raises are one's we're currently thinking through, and exactly how much an author's view of the impact, difficulty, and quality of the guide affects its rating.
Thoughts? Reply to comment
As to the quality of the guide beyond that, the author is probably the worst person to ask.
Is the impact scale supposed to be linear or log? ie does a 5 have five times as much impact as a 1 or 10^5 (100,000) times the impact (or pick a base if you don't like 10)? I would lean more towards a log scale, since in the current lingo five people surfing the web will not get to the top of mount Everest.
Is there any way of dealing with the fact that every fix to a given guide will have a different impact? If you live next door to your office biking to work will have less influence than if you live 10 miles away etc. Or is it better just to accept the fact that there will be some inaccuracy from this and move on.
Reply to comment
At the moment, my rough thoughts of the 'goodness' of a guide is a combination of its: user rating, times completed (excluding the author), and times favorited.
The impact scale is roughly log, which makes calculating the total impact very very nice.
As far as varying impacts, there's two ways we deal with that: one, the fix author picks their own impact. Two, we use a log scale, and for the most part, I think fixes will probably be within an order of magnitude of their guides, so we accept it and move on.
Reply to comment
Also, is the user rating system implemented yet?
Will the rating system track (at least internally) which ratings are from who? If it does this it would be cool to also display to a user what they rated a guide as and allow them to change it if the guide improves/ devolves.
I assume there are/will be safeguards to prevent rating spam. If these aren't there yet I would suggest that rating of a given guide be limited to one vote per person, and that in order to vote, you must create at least one fix or guide. This will increase the incentive for people not to create junk accounts to up their ratings.
Reply to comment
The user rating system is done, but we don't yet have pages where you can enter them in besides our admin site. They're coming for the regular site, but we're looking to nail down how people want them to work first.
Ratings are tied to a user, and I think being able to edit your reviews makes sense. You also only get one.
As far as preventing spam, it's a balance between ease of use and quality. We already have a pretty high barrier in that you have to sign up and confirm an email address (may add captcha if we start to see junk accounts). A time limit (only after you've had an account for 24 hrs, etc) may also make sense, but it's also possible that we just let things roll, and count on the wisdom of crowds to sort things out. Reply to comment
Guide quality will be revised to simply "draft" or "ready for use" - draft guides can't be rated, and are filtered out by default when searching.
Impact, however, is a trickier subject. We talked over everything from impact being entirely community rated to it being set for each guide by jeff and I (that idea was tossed pretty quickly). We talked about using flags to report inaccurate impact (to whom? and what do they do?), and in all, we never found a solution that was entirely satisfactory.
So, we bring the question back to all of you. Here's why impact is important, and what we hope to use it to do.
Some tasks (becoming vegetarian) have a lot more impact than others (changing a lightbulb). In many cases, these differences are orders of magnitude apart. We want to provide people a realistic idea of how much a particular fix will really impact the world, to combat the feel-good just-recycle-and-everything-will-be-fine environmentalism that ultimately makes people do less, not more.
So in that way, impact is critical to providing a good understanding that certain changes in lifestyle or action have much larger consequences than others (and that the sizes might not be what you'd expect).
The challenge is to do this in a way that is easy to use, and accurate. We also have to prevent malicious/misinformed-crusader-types from marking fixes much higher than they actually are.
Some ideas:
* Leave the system as-is, but fix the names.
* Allow them to be flagged, and manually adjusted by Jeff and I (prefer not)
* Allow the community to vote (do they know more than the author?)
* Have a combination of the author, community, and invisible hand of site operators(eek)
* Author only
* Community only
* Something completely different.
If folks have suggestions, let's hear them! Reply to comment
In certain categories/ types of fixes you could use some objective measures (how many kWh does it save per year, how much CO2 equivalent does it keep from being emitted, but these may be hard to figure out for some fixes). Even so, having concrete values along with the impact "phrase values" might help people place impacts better. It would certainly help discussion about what impact something should be rated as. Perhaps an impact guide would be useful for this.
If you can have the website flag guide when being submitted that has a impact 2 or more than greater than the difficulty (This shouldn't happen very often, unless it's something like don't play with dynamite and uranium in your back yard) and have a confirmation box like the following pop up:
"You have stated that this guide will have a very large impact for very little effort. If this is true, we welcome it and look forward to implementing it. However, please consider whether a single person completing this guide will have this level of impact. If the guide is asking people to support a cause, remember that you are only counting the impact of a single person, not of the causes total impact"
I would have the author put their initial impact in. Perhaps a weighted average along the lines of what Rachel suggested (I would use the Author= the higher of people or 25% of the total rating rather than the 50%, but that is details). You could also require comments (either anonymous or named to accompany ratings). Reply to comment
I really like that idea on impact >= 2 + difficulty - tying this in with scale could make an even better algorithm.
The other issue to handle is abuse (setting my guide to 6* impact because I can), but this may be handled ok with the report offensive/abusive flags we're putting in.
As far as getting better "phrase values", this was one thing that we agreed needs improvement.
Perhaps, like the difficulties, we should base them on common tasks:
Completing this fix has about as much impact on the world as:
* using an energy efficient lightbulb
* driving a hybrid car
* switching my city to clean energy
* making a 200 square mile area of biodiverse land a protected area
* switching my nation to 70% clean energy
* creating an international carbon-trading system
other ideas? Reply to comment
I was trying to get at rather than replacing the phrase values, augment them with concrete values to give a reference if the exact savings/effect can be estimated. This could even be a pop up window next to the impact bar named something like "help valuing your impact" that pops up a page benchmarks for each of the major categories.
So something like:
Transportation
* Reduce gasoline usage by 10 gallons a month
** Reduce gasoline usage by 100 gallons a month
*** ...
Making Energy
* Reduce the CO2 per kilowatt by 100 lbs (making up numbers here)
Using energy
* Reduce energy usage by 100 kWh/ year (again making up numbers)
** Reduce energy usage by 1000 kWh/ year (again making up numbers)
*** Reduce energy usage by 10000 kWh/ year (again making up numbers)
...
etc
This would at least give objective metrics to judge against if there is a dispute over an impact rating.
(Also, minor point, if you want to go international, saying switching my nation to 70% clean energy is not a very objective measure) Reply to comment
(70% isn't too bad - very few nations are anywhere near that. hell, it should probably be above the carbon system. ) Reply to comment
if 2 =< ( (impact-difficulty) + (impact - scope) ):
# flag it
Reply to comment
Are there any good examples of high impact guides written yet? Reply to comment
If it is auto favorited, that also starts becoming essentially number of people who started the guide. Which will be similar numbers to who completes it. Reply to comment
Also, I'd say that # of times favorited is going to be a small portion of the guide's ranking. # of times completed is the number one criteria, by a huge margin. Reply to comment
If you don't have an account, you'll need one. We recognize that this is a pain, but we do it to keep spambots from flooding the site. If this really bothers, you, please let us know. We're listening!