Evaluating Paid dev, graphs are skewed.
First, I need to start by saying, I like the idea of paid developers.
I agree that 70 or so submitters is really not sufficient. Would probably have been better to collect for at least twice as long, hoping to at least have a full 100 opinion givers.
I like the graphs that are shown, but 1 issue is causing two problems in my opinion.
The significance of the negative response results at first glance is deceiving.
I'm not sure this is on purpose, or by some auto-scaling.
The first problem is that this unfairly decreases the voice of the opposition, it is important to know that some people are deeply against it. Otherwise what is the point of conducting any statistical review?
My second issue is that when people who have a keen sense for reading the results, notice that they appear to be scaled to reduce the power of negative votes, that they might associate those who are heavily involved in the software Joomla at any level to be possibly unethical or at minimum they may perceive that we are lacking attention to detail to disable the parameter that causes the de-significance.
In my opinion these are the charts that are affected. 1,6,9,10,11,12,13,14 and why is there not any data for Jan 2009 (other data)
http://community.joomla.org/blogs/community/1109-evaluation-results...
I agree that 70 or so submitters is really not sufficient. Would probably have been better to collect for at least twice as long, hoping to at least have a full 100 opinion givers.
I like the graphs that are shown, but 1 issue is causing two problems in my opinion.
The significance of the negative response results at first glance is deceiving.
I'm not sure this is on purpose, or by some auto-scaling.
The first problem is that this unfairly decreases the voice of the opposition, it is important to know that some people are deeply against it. Otherwise what is the point of conducting any statistical review?
My second issue is that when people who have a keen sense for reading the results, notice that they appear to be scaled to reduce the power of negative votes, that they might associate those who are heavily involved in the software Joomla at any level to be possibly unethical or at minimum they may perceive that we are lacking attention to detail to disable the parameter that causes the de-significance.
In my opinion these are the charts that are affected. 1,6,9,10,11,12,13,14 and why is there not any data for Jan 2009 (other data)
http://community.joomla.org/blogs/community/1109-evaluation-results...
-
Comment by Amy Stephen on February 1, 2010 at 11:56pm
-
KUDO's to OSM for collecting input - and double those KUDO's for sharing the text of what people had to say.
As far as I am concerned, you can toss the graphs -- there is no way to take 70 responses that include an anonymous mix of users and contributors -- and expect to get a scientific result as to whether or not this one issue impacted motivation. Is 70 a reasonable number? Did people not respond because they didn't expect to be taken seriously? Etc.
But, there's gold in those comments.
My feel is that single decision was simply the straw that broke the camel's back. And it had little to do with the specific decision and more to do with a collective frustration that decisions are made behind closed doors and announced, and "opposition" is characterized as whining.
I have no interest in revisiting this past few months, but it's worth remembering that even members of the Leadership Team were uninvolved and frustrated, there wasn't even a clear idea of who is responsible for what.
On the positive side, we've seen a lot of good change happening. I think even this survey is likely a response to the call for involvement. And, that's very good. It's probably not as important that the instrument produce accurate results this first time as it is that there was a sincere attempt to try and that results were shared. That *is* good!
I'm waiting to see what is done with the OSM board members.There is an opportunity to bust this project open and go for the bazaar or we can continue hanging on to the Cathedral.
I want to make this clear - Joomla! is evolving. Think back to 2005--very little was shared. FAR MORE is shared now. Those in OSM right now "caught the hot potato" of community awakening in this evolution but the truth is, they are sharing far more than has ever been shared before and they will be the ones who continue to make it possible for broader involvement.
-
Comment by Mike Hamanaka on February 2, 2010 at 12:01am -
Regardless of the sample size of only 70 people. What I am saying is that about half of the graphs that were displayed on Elin's blog page do not start at "zero" and they need to. Otherwise that are misrepresenting the information that was gathered.
-
Comment by Mike Hamanaka on February 2, 2010 at 12:02am
-
my last sentence there...
"Otherwise the graphs are deceiving and misrepresenting the information that was gathered.
-
Comment by Mitch Pirtle on February 2, 2010 at 12:13am -
I'm willing to wager that all the people that are in favor, and all that are against bothered to take the survey, and everyone else watched from the sidelines. Hrmph.
I agree with you Mike, in that maybe the graphs should have been percentages or all normalized to zeros for consistency. The first graph for example, there were 9 folks that had a very negative reaction to the question, which happens to be over 10% - significant enough for me to take notice. But if the graph starts at 5, visually it looks like less than 10%...
Most importantly, the validity of your data is directly relative to the sample size, and 70 folks just ain't enough to mean a thing. I can imagine more than half the people that responded were active members of the team, which IMHO indicates a huge bias in the sample set. However if there were say 700 folks that took the survey, I'd be comfortable with it.
I'm involved in a startup that is building an online-hosted focus group platform and as such have been waaaaaaaaay more into the structure and efficiency of focus groups and testing than I have ever been in my career. This looks more like a simple gallup style poll to me, and doesn't represent a big enough sample set to be meaningful or useful.
I'd love to see the survey repackaged with much simpler terms that normal people will understand, and advertised to a much larger audience.
-
-
Understood. I saw the scale, too. I would not read anything into that - except that the graph looks better to the graph maker that way. I think it's great OSM asked, they sure didn't *have* too - and they shared all of the data, including the raw comments, and that is admirable. I think people had some great things to say - not many negative, ugly comments - but lots of concern and a whole lot of support for Joomla!. In all, very encouraging.
-
Comment by Peter Russell on February 2, 2010 at 12:26am -
Agree with Mitch on this. Scewed data not representative of the community, doesn't prove anything — the opposite in fact it fuels concern because of "lies, damned lies and statistics".
-
Comment by Mike Hamanaka on February 2, 2010 at 12:41am
-
Yes Amy, this is what I concluded from the resulting information when I looked at it in the middle of the day at first glance. But since I have been at home this evening, I wanted to really read through the numbers and I think that they should be corrected or taken down, they are of low quality and in some graphs it is impossible to read the sub-heading. If this was just a standard report of small matters, I would not raise issue here, but the selectively skewed information does not show professionalism and good stewardship of this survey. I'm not even sure if the homepage of Joomla.org is a good place to display this information.
Peter, thank you for confirming that this could lead to greater concerns.
I want there to be publicly available information, but not if it is going to be reported with perceived bias. Even if there was none.
-
-
I want there to be publicly available information, but not if it is going to be reported with perceived bias. Even if there was none.
Good point.
-
Comment by Klas on February 2, 2010 at 4:50am -
A sociologist by profession* Elin should know that statistically this kind of online research is totally invalid (most important: non-random sample, also size of a sample). It can only be used for some kind of qualitative analysis (why, how..) or as questionnaire pretesting tool or similar. Presenting results of this survey in quantitative way (how much) and even those skewed obviously serves manipulative purposes (the only other logical option left here is lack of knowledge).
*at least here even sociology students get extensive knowledge of statistics and research methodology
-
-
I believe I read Elin state this is not a scientific survey. Yes, given her professional and Doctorate degree, she is obviously well trained at research. Advantage to Joomla! to have her involvement. We would be very silly to assume motive for how a few graphs were presented. If this were really intended to mislead, clearly we were able to punch out of the paper bag. The graphs weren't mislabeled, and the data was not inaccurate. Come on.
We pull together as a community, or we don't. If people get good information, then they start to understand the project and develop faith in the organization. If people are heard, they feel like it's worthwhile to share their perspectives. If their ideas are valued and they are welcome, they start to develop a sense of belonging. If their contributions are accepted and used, ownership results and they increase their involvement. When we have more and more people who who feel like like informed, involved, and valued members, then, we have a large workforce who are collaborating in friendly ways, sharing solutions with one another.
Graph that - and then, I'm going to be interested. That's the kind of thing Joomla! lives or dies on.
This was a good thing for OSM to do. They involved the community and asked for our perspective. They shared information, by publishing the results. That's good. More of the same on a regular basis will be very welcome. As we continue down this road, the project will learn to adapt to the voice of the community and the community will learn to get more involved.
IMO, this specific survey doesn't matter near as much as encouraging this type of involvement.
Brian - I agree. IMO, the ecosystem should support the project. I am concerned to have this continue and I said so when I took the survey.
© 2012 Created by Amy Stephen.
You need to be a member of All Together, As A Whole to add comments!
Join All Together, As A Whole