Vivid QA incentives report

[Elizabeth K. Joseph]

Hi everyone,

At our last meeting[0] efly and I had a good chat about the initiative
to give away stickers for people who contributed to QA[1]

The short version: It didn't help us gain new contributors, so we
won't be continuing the program into the Wily cycle.

The long version:

Stickers are fun, and I thought that a giveaway would be a fun way to
reward our QA contributors and encourage other people to get involved.

I've since learned a lot about motivation and realized that my
assumptions were way off. Essentially, studies have proven that
"carrot and stick" type motivation works for physical, mechanical work
(make x number of widgets,do more data entry, file a busywork report)
but in today's world where most work is problem solving and
creativity-based, it fails to motivate effectively and actually does
harm. The Candle Problem is one of the experiments often cited:
http://en.wikipedia.org/wiki/Candle_problem (see the section of the
article on Glucksberg).

Our work on QA is not mechanical, make-more-widgets-faster work, it's
problem-solving. You need to go through the test cases thoughtfully
and it takes creativity, knowledge about Xubuntu and effort to report
bugs effectively. Ones that are reported without thought just to get
your stats up on the QA tracker are not helpful, as they don't get to
the bottom of what the bug is reporting, making the report useless to
us.

As if the numbers didn't speak for themselves (we didn't really see a
rise in contributors), most of the people we contacted about their
winning of the stickers were surprised. One nearly declined to take
them at all until he decided he'd give the stickers to people he knew.
Stickers are not why people contribute to Xubuntu QA, they want to
help us and give of their time to support an operating system that's
important to them. This is reflected in other research that I learned
about that explained that monetary and "stuff" rewards for creative
work actually do harm to otherwise philanthropic deeds because they
devalue them. Gosh, I got this all wrong!

Now, how to move forward:

We still need to increase participation in QA, but we need to think of
better ways to encourage participation. Acknowledgement and
recognition given by members of the Xubuntu team should be part of
this. Perhaps an email from the project lead thanking them for their
work when we notice someone is doing a lot of testing, or mention that
they've done good work on social media or a profile of them on our
site. These kinds of surprise, non-mechanical tokens of gratitude go
much further than some stickers they won for competing for most tests
completed. Speaking for myself, these kinds of non-stuff rewards
certainly have motivated me in the past. Thoughts?

As a final note, if you want to learn more like I did, I recommend
reading Drive[2]. The book itself is a quick read (under 300 pages
with TONS of references, footnotes, etc) but led me down the rabbit
hole of other motivation research. The recommendation of this book
came to me from Robert Collins, who I know from Ubuntu stuff and now
work with on OpenStack.

[0] https://wiki.ubuntu.com/Xubuntu/Meetings/Archive/Minutes/2015-05-12
[1] http://xubuntu.org/news/help-the-community-with-testing-and-win-xubuntu-stickers/
[2] Drive: The Surprising Truth About What Motivates Us
http://www.danpink.com/drive/

[Paul White]

Post by Elizabeth K. Joseph
We still need to increase participation in QA, but we need to think of
better ways to encourage participation. Acknowledgement and
recognition given by members of the Xubuntu team should be part of
this. Perhaps an email from the project lead thanking them for their
work when we notice someone is doing a lot of testing, or mention that
they've done good work on social media or a profile of them on our
site. These kinds of surprise, non-mechanical tokens of gratitude go
much further than some stickers they won for competing for most tests
completed. Speaking for myself, these kinds of non-stuff rewards
certainly have motivated me in the past. Thoughts?

As one of your "sticker" winners I was very surprised that my rather
small contribution qualified as a the winning contribution but looking
at the QA Tracker I could see that I was one of just a few non-Xubuntu
team members recording their tests.

Although I did record a couple of installations onto actual hardware the
bulk of my tests were for live sessions, both 32 and 64-bit, running not
only the documented test cases but also making sure that certain other
applications and functionality was working correctly very much as if I
was using Xubuntu for the first time. As you know the QA Tracker gets
littered with "bug icons" which simply alert users to the existence of a
bug but hide the bug description. Holding the mouse pointer over each
icon to find out which bug is being represented is tedious and I'm
really not interested in knowing that a bug has been reported for a KDE
application if I'm testing Xubuntu, like wise a Xubuntu/Xfce specific
bug doesn't interest me if I'm testing for Kubuntu. Obviously the QA
Tracker is not going to change in the near future to make things easier
for testers of specific flavours but I would be very interested to hear
how things might be made easier for testers. Might this also help
attract more testers or increase the testing of existing testers?

The Kubuntu Team have their Trello board where certain tasks can be
allocated and commented on. How about something similar for Xubuntu such
as an etherpad where known bugs can be recorded and comments added? Any
potential bug fixes could be posted and moved around depending on their
importance. Once bugs have been fixed or problems resolved, items could
then be removed. The etherpad would therefore be a summary of
outstanding issues.

I'm suggesting this as I'm often at a loss what to test for. I'm
inundated with emails from various lists which I read at various times
of the day on various devices and by the time I get around to do some
testing at the weekend I've forgotten about the most important items.

Anything that can highlight current testing requirements would be of
great help to me and I'm sure to others.

[Pasi Lallinaho]

Post by Paul White

Post by Elizabeth K. Joseph
We still need to increase participation in QA, but we need to think of
better ways to encourage participation. Acknowledgement and
recognition given by members of the Xubuntu team should be part of
this. Perhaps an email from the project lead thanking them for their
work when we notice someone is doing a lot of testing, or mention that
they've done good work on social media or a profile of them on our
site. These kinds of surprise, non-mechanical tokens of gratitude go
much further than some stickers they won for competing for most tests
completed. Speaking for myself, these kinds of non-stuff rewards
certainly have motivated me in the past. Thoughts?

As one of your "sticker" winners I was very surprised that my rather
small contribution qualified as a the winning contribution but looking
at the QA Tracker I could see that I was one of just a few non-Xubuntu
team members recording their tests.
Although I did record a couple of installations onto actual hardware the
bulk of my tests were for live sessions, both 32 and 64-bit, running not
only the documented test cases but also making sure that certain other
applications and functionality was working correctly very much as if I
was using Xubuntu for the first time. As you know the QA Tracker gets
littered with "bug icons" which simply alert users to the existence of a
bug but hide the bug description. Holding the mouse pointer over each
icon to find out which bug is being represented is tedious and I'm
really not interested in knowing that a bug has been reported for a KDE
application if I'm testing Xubuntu, like wise a Xubuntu/Xfce specific
bug doesn't interest me if I'm testing for Kubuntu. Obviously the QA
Tracker is not going to change in the near future to make things easier
for testers of specific flavours but I would be very interested to hear
how things might be made easier for testers. Might this also help
attract more testers or increase the testing of existing testers?

There is some development ongoing on the tracker(s) and the tooltip
issue is one of the high priority ones (on at least my list). However,
this is another community project with virtually no Canonical
(programming) manhours assigned, which has been slowing the improvements
down.

If you have other issues/bugs with the tracker that aren't already
reported [1], I would love to see you file a bug [2] and/or tell about
your issue on #ubuntu-quality.

Post by Paul White
The Kubuntu Team have their Trello board where certain tasks can be
allocated and commented on. How about something similar for Xubuntu such
as an etherpad where known bugs can be recorded and comments added? Any
potential bug fixes could be posted and moved around depending on their
importance. Once bugs have been fixed or problems resolved, items could
then be removed. The etherpad would therefore be a summary of
outstanding issues.

The Xubuntu team used Trello for a cycle or two, but ultimately we
decided against continuing it for the broad coordination.

Instead, we're using the Launchpad blueprints (for the Vivid cycle, see
[3] and all of it's children under the dependency tree) of which are
automatically gathered under status.ubuntu.com [4] (for Vivid: [5]).
Does this help with the organizational aspect - or the apparent lack of it?

Post by Paul White
I'm suggesting this as I'm often at a loss what to test for. I'm
inundated with emails from various lists which I read at various times
of the day on various devices and by the time I get around to do some
testing at the weekend I've forgotten about the most important items.
Anything that can highlight current testing requirements would be of
great help to me and I'm sure to others.

Cheers,
Pasi

[1] https://bugs.launchpad.net/ubuntu-qa-website
[2] https://bugs.launchpad.net/ubuntu-qa-website/+filebug
[3] https://blueprints.launchpad.net/ubuntu/+spec/topic-v-flavor-xubuntu
[4] http://status.ubuntu.com/
[5] http://status.ubuntu.com/ubuntu-v/
--
Pasi Lallinaho (knome) » http://open.knome.fi/
Leader of the Shimmer Project » http://shimmerproject.org/
Ubuntu member, Xubuntu Website lead » http://xubuntu.org/

[Paul White]

Post by Pasi Lallinaho
Instead, we're using the Launchpad blueprints (for the Vivid cycle, see
[3] and all of it's children under the dependency tree) of which are
automatically gathered under status.ubuntu.com [4] (for Vivid: [5]).
Does this help with the organizational aspect - or the apparent lack of it?

Thanks for your reply Pasi.

I wasn't really aware of what status.ubuntu.com was or that the Xubuntu
team even used it. When I have a little more time I need to study that
site to get a better understanding of what I am being shown.

I was really suggesting the introduction of something much simpler that
could perhaps be brought into use quickly and updated easily, especially
after a call for testing for the various milestones.

Simple tools need to be used in order to attract testers and keeping
them testing. Referring users that have only just started to contribute
to testing Xubuntu to status.ubuntu.com will probably just frighten them
away.

What I am suggesting is an easy to understand one-screen approach to
tell testers what needs special attention during their testing session.

[Simon Steinbeiß]

Hey Paul,

On Sat, 16 May 2015 16:46:38 +0100

Post by Paul White
I wasn't really aware of what status.ubuntu.com was or that the Xubuntu
team even used it. When I have a little more time I need to study that
site to get a better understanding of what I am being shown.
I was really suggesting the introduction of something much simpler that
could perhaps be brought into use quickly and updated easily, especially
after a call for testing for the various milestones.
Simple tools need to be used in order to attract testers and keeping
them testing. Referring users that have only just started to contribute
to testing Xubuntu to status.ubuntu.com will probably just frighten them
away.
What I am suggesting is an easy to understand one-screen approach to
tell testers what needs special attention during their testing session.

Yes, those are the exact reasons we started to experiment with Trello. However, being able to integrate bug reports etc into blueprints turned out too big a benefit and in the end, either is work. It's not like keeping Trello up-to-date isn't time-consuming or at times boring. Launchpad might feel (and be) slower, we can only hope for improvements there, and it might have a steeper learning curve, but in the end it seems like it better satisfies our needs.

Within sub-teams (e.g. development or QA), Trello can (and is) still be used of course, so maybe that can help with what you're pointing at.

Cheers
Simon

[elfy]

[snip]
I'm suggesting this as I'm often at a loss what to test for. I'm
inundated with emails from various lists which I read at various times
of the day on various devices and by the time I get around to do some
testing at the weekend I've forgotten about the most important items.
Anything that can highlight current testing requirements would be of
great help to me and I'm sure to others.

Sorry about being a tad afk ...

For something this specific I don't actually believe that Launchpad is
agile enough.

Picking up on this point there's no reason why we couldn't look at using
something to do this.

Maybe Trello ... maybe an etherpad ... maybe wiki ...

The issue with etherpad (at least using Ubuntu one) is people need to be
in a LP group to access that - not everyone is.

Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.

Wiki - I started last cycle with a 'skeleton' wiki for us to use to
build release notes against, no reason why not having a section there.

Mailing lists, I already mail the dev list and the LP testers when we
want something tested - no reason why not be more specific when doing so.

Finally, if there is anything that I and the QA team can do to make life
easier - we're all ears.

thanks

Kev

[Paul White]

Post by elfy
Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.

I've looked at the Xubuntu Trello board and like what I see.

Post by elfy
Mailing lists, I already mail the dev list and the LP testers when we
want something tested - no reason why not be more specific when doing so.

May be a weekly update of issues that require the attention of testers?

Either way, I hope next week's meeting will pick up on this.

Thanks to all those that replied to my initial email. :)

[elfy]

Post by Paul White

Post by elfy
Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.

I've looked at the Xubuntu Trello board and like what I see.

Post by elfy
Mailing lists, I already mail the dev list and the LP testers when we
want something tested - no reason why not be more specific when doing so.

May be a weekly update of issues that require the attention of testers?

Frankly - anything that I can do to make things easier for testers is
something I'd do, given time and the tools ;)

Post by Paul White
Either way, I hope next week's meeting will pick up on this.

If you're going to be around - that would be awesome - you can even
table it to the agenda ;)

Post by Paul White
Thanks to all those that replied to my initial email. :)

[Istimsak Abdulbasir]

Post by Paul White

Post by elfy
Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.

I've looked at the Xubuntu Trello board and like what I see.

What is this Xubuntu Trello?

Post by Paul White
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

[Paul White]

Post by Istimsak Abdulbasir
What is this Xubuntu Trello?

https://trello.com/xubuntuteam

--
Paul White ***@ubuntu.com

[Istimsak Abdulbasir]

Post by Paul White
https://trello.com/xubuntuteam
--
Paul White

Thanks. I am inferring this will give me an idea of what to expect in the
meeting?

Post by Paul White
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

[elfy]

Post by Paul White
https://trello.com/xubuntuteam
--
Paul White

Thanks Paul.

Post by Paul White
Thanks. I am inferring this will give me an idea of what to expect in
the meeting?

Did you look at it - not sure why you think it'll give you an idea of
what to expect in a meeting.

It's something that 'we' already use - Paul and I are discussing methods
to give testers more information.

Post by Paul White
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel
--
"Collaboration is the new innovation" (Istimsak Abdulbasir, 2013)

[Istimsak Abdulbasir]

I am looking at. I had a crazy thought it was going to be part of the
meeting. Please excuse my short commings.

Post by elfy

Post by Paul White
https://trello.com/xubuntuteam
--
Paul White

Thanks Paul.
Thanks. I am inferring this will give me an idea of what to expect in the
meeting?
Did you look at it - not sure why you think it'll give you an idea of what
to expect in a meeting.
It's something that 'we' already use - Paul and I are discussing methods
to give testers more information.

Post by Paul White
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

--
"Collaboration is the new innovation" (Istimsak Abdulbasir, 2013)
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

[Istimsak Abdulbasir]

Post by elfy

Post by Paul White
https://trello.com/xubuntuteam
--
Paul White

Thanks Paul.
Thanks. I am inferring this will give me an idea of what to expect in the
meeting?
Did you look at it - not sure why you think it'll give you an idea of what
to expect in a meeting.
It's something that 'we' already use - Paul and I are discussing methods
to give testers more information.

Post by Paul White
This is good stuff. Looks more like a blue-print of Xubuntu activities. I

won't say everything, but what relates to the Xubuntu project is tracked in
this tracker. The bugs are imported from launchpad. I like the detailed
descriptions of them. I learned a few things when it comes to filing a bug
report. For instance, installed the gdb program to help with bug
backtracing.

Now I can subscribe to "cards" if I understood that right to track a
certain activity. However, I did not see the option to add a "card". It
appears I have to be an administrator.

Wish launchpad could look this neat.

Post by elfy
--

Post by Paul White
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

--
"Collaboration is the new innovation" (Istimsak Abdulbasir, 2013)
--
xubuntu-devel mailing list
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel

[Pasi Lallinaho]

Post by Istimsak Abdulbasir
Thanks. I am inferring this will give me an idea of what to expect in
the meeting?

The meeting agenda is located on the meetings page of the Xubuntu wiki at
https://wiki.ubuntu.com/Xubuntu/Meetings

Cheers,
Pasi
--
Pasi Lallinaho (knome) » http://open.knome.fi/
Leader of Shimmer Project » http://shimmerproject.org/
Ubuntu member, Xubuntu Website Lead » http://xubuntu.org/

[elfy]

Post by Paul White

Post by elfy
Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.

I've looked at the Xubuntu Trello board and like what I see.

Post by elfy
Mailing lists, I already mail the dev list and the LP testers when we
want something tested - no reason why not be more specific when doing so.

May be a weekly update of issues that require the attention of testers?
Either way, I hope next week's meeting will pick up on this.
Thanks to all those that replied to my initial email. :)

If you swing by it again - I've got a draft board set up.

Assuming you're able to open the cards (- testing that would be great ;)
) I could put notes as to anything specific we need looking at during
milestones there for anyone.

I'm still not sure if this would be any better than specifying when we
call at milestones.

I'm also not sure about weekly updates.

Now - as far as the meeting goes - I'm happy for it to get discussed,
but unless we get people from the community along who are happy to test
and have opinions - it's just going to end up with me saying "some
people have suggested foo"

What would be a better idea in my opinion would be a testing/qa specific
meeting. If there are enough people interested in that type of thing
then maybe start a m/l thread and we can see who's up for it - then set
up a meeting in #xubuntu-devel when all (or the majority) can attend.

[elfy]

[snip]
What would be a better idea in my opinion would be a testing/qa
specific meeting. If there are enough people interested in that type
of thing then maybe start a m/l thread and we can see who's up for it
- then set up a meeting in #xubuntu-devel when all (or the majority)
can attend.

(always wait an hour before sending for the idea you've not thought of
quite)

or I could set up an etherpad for us and then people can 'attend' with
their idea's when others are asleep

With Xubuntu being quite communal - I'm happy to be guided by 'you' here

[Paul White]

Post by elfy
Assuming you're able to open the cards (- testing that would be great ;)
) I could put notes as to anything specific we need looking at during
milestones there for anyone.

Yes, I can open the cards. I've registered as a Trello user as well.

Post by elfy
I'm still not sure if this would be any better than specifying when we
call at milestones.

May be, may be not but at least cards can be updated and the information
is still in one place.

Post by elfy
I'm also not sure about weekly updates.

That was only something I was suggesting for a totally e-mail solution,
otherwise not required.

Post by elfy
Now - as far as the meeting goes - I'm happy for it to get discussed,
but unless we get people from the community along who are happy to test
and have opinions - it's just going to end up with me saying "some
people have suggested foo"

I'll be there to at least add a 'voice'.

Post by elfy
What would be a better idea in my opinion would be a testing/qa specific
meeting. If there are enough people interested in that type of thing
then maybe start a m/l thread and we can see who's up for it - then set
up a meeting in #xubuntu-devel when all (or the majority) can attend.

Another more specific meeting? Fine. The etherpad suggestion that you're
going to make in your *next* email will be a good one especially for
those far west of us and for anyone down under. :)

[f***@gmx.co.uk]

Post by elfy

[snip]
I'm suggesting this as I'm often at a loss what to test for. I'm
inundated with emails from various lists which I read at various times
of the day on various devices and by the time I get around to do some
testing at the weekend I've forgotten about the most important items.
Anything that can highlight current testing requirements would be of
great help to me and I'm sure to others.

Sorry about being a tad afk ...
For something this specific I don't actually believe that Launchpad is
agile enough.
Picking up on this point there's no reason why we couldn't look at
using something to do this.
Maybe Trello ... maybe an etherpad ... maybe wiki ...
The issue with etherpad (at least using Ubuntu one) is people need to
be in a LP group to access that - not everyone is.
Trello - there is a QA trello instance - we could have a 'we want to
test this' board on it - that trello is publically readable.
Wiki - I started last cycle with a 'skeleton' wiki for us to use to
build release notes against, no reason why not having a section there.
Mailing lists, I already mail the dev list and the LP testers when we
want something tested - no reason why not be more specific when doing so.
Finally, if there is anything that I and the QA team can do to make
life easier - we're all ears.
thanks
Kev

So - restarting this discussion.

What do you the testers think is the best way forward?

Frankly - I'm happy to do what works best for you.

At the moment the concensus from a few people is use trello to give
testers the information they need.

Currently we've got a few weeks to get this sorted out - this cycle has
been a bit odd for QA - but what we do need to do, is be ready for the
next cycle that being testing for the next LTS.

thanks all

[Dave Pearson]

[Pasi Lallinaho]

I'm all up for using something that can give us testers something to
look at and work from.
it would help a great deal, and allow specific scenarios that need
testing.

Apart from specific scenarios that the team devised, what would you like
to see listed in such a "notification board" on a daily basis?

Just throwing a wild on in.. how about googledocs for what needs testing?

Since the aim is accessilibity for anybody, anytime, anywhere, I'm not
sure if a Google doc is the best option. We do have other places to dump
"simple text data" too.

That said, please do not feel restricted to tools that exist now with
ideas – if an idea is good and people think it would be useful to have
that, we can create things that haven't existed before. This can also
include pulling data automatically out of existing tools etc.

Cheers,
Pasi

[Simon Steinbeiß]

Hey Lyz,

first of all: sorry it took me so long to reply and thank you for this great and comprehensive write-up!

The short version: Not all points in the argument convince me and I'm not sure we should jump straight to the conclusion of discontinuing the programme. (Maybe we should still do it, though. Thanks Lyz and Elfy for your hard work on this!)

The long version:

It's true, the handing out of stickers is sort of a "carrot and stick" situation. But then again it isn't. First of all, that is one tiny carrot. In my opinion, it's about as much as the "email from the project lead". So I'd say it depends on your point of view, whether this is truly "stuff" or just a sign of recognition like an email. (I agree that an email is less work than sending stickers by snail mail, so if the workload is the main argument for discontinuing, I won't argue.) Sending stickers automatically to contributors might be a bad idea indeed, so if someone doesn't want them, well, I see no reason to convince them to take them. But yeah, I agree that the size of the carrot is not the problem here in the sense that we would have needed a bigger one or that that would have made things better. I think it's the smallest possible carrot and in that I'm wondering whether it's truly doing the damage of devaluing the work. I for one am happy that you once sent me a sticker! :)

Another counter-argument is that we obviously didn't advertise this enough (the winners being surprised and not knowing about it), so it couldn't even work as a carrot in these cases. So us not gaining contributors could also be due to a lack of advertising the stickers programme or lack of an audience.

Looking at similar incentives, the bug bounty programme a la bountysource is an obvious candidate for evaluation under these psychological/behavioristic theories. It's true that bountysource doesn't work for everyone, at least with Xfce it didn't seem to bring in new contributors (at least so far). However, it also seems to work just fine for others like elementary [1]. So yeah, might really depend on the community/audience one has.

From where I stand it's really hard to motivate people who are otherwise not into QA for testing, or maybe FOSS in general. I wouldn't say that the stickers programme has worked, especially after your compelling argument (or the fact that we didn't gain new contributors) that it didn't. However, I guess the stickers programme could only work if people really took note of it and that's my general take-away from this: we probably need to do a better job at communicating with users or the community (as far as we have one, or if not, try harder to build one).
In the end that might be a more time-consuming task than sending out stickers and yeah, this is obviously nothing new.

As indicated in the short version, it might still be for the best to discontinue the stickers programme. It didn't help us get new contributors, it actually meant more work for the people in the team. Maybe we can also alter the programme so it doesn't cause that much work (like sending the stickers of one cycle out at once at release time) or try to make the stickers more of an offer than an automatic reward (a la: "Thanks for your contributions, we'd like to thank you with a sticker. If you want one, send us your physical address and we'll mail it to you at the end of the cycle, if not live long and prosper!"). Step forward if you have creative ideas or if any of my suggestions make sense to you.

Either way, I felt like chiming in here since I was one of the folks pushing this initiative at the outset, but obviously others did the heavy lifting. So first of all, a big thank you to Lyz and Elfy for taking care of this – even if we find that it hasn't resulted in what we had hoped for, I think it was worth a shot and if anything, this email is intended as a sign of appreciation and recognition of your work!

Cheers
Simon


[1] http://blog.elementary.io/post/118806943356/thank-you-everyone