Volunteers needed -- Automated Image Testing for flavors

[Nicholas Skaggs]

Greetings everyone! I wanted to share some news about a renewed effort
to restore automated installer testing of ubiquity using the daily
images generated by cdimage. Up until last cycle, the images were being
test automatically via a series of autopilot tests, written originally
by the community (kudos to you Dan!). It was noticed the tests didn't
run this cycle, and wxl accordingly filed an RT;
https://rt.ubuntu.com/Ticket/Display.html?id=26570. In short, as part of
some datacenter shuffling, we learned it's no longer possible for CI to
run or maintain the tests. They recommended we host and run them
ourselves as a community.

With the directions from CI in hand, I initially asked DanChapman and
dkessel to investigate setting up a jenkins to run these tests. But they
need your help! The autopilot tests for ubiquity have a few bugs that
need solving. You can see them here:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bugs?field.tag=autopilot.

To help solve these bugs or to learn more, check out this document:
http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/view/head:/autopilot/README.md.
It will guide you through running the tests yourself locally. You should
be able to replicate the tests following those instructions. From there,
patches and comments on the bugs would be most welcome (as would
additional bug reports should you find them).

Even with the tests working, however, we still need to setup a jenkins
to run them. We'll also need to maintain this server. Anyone with
experience or desire in this area? Ideas for reporting results (on the
isotracker for instance) also need to be explored. This is likely to
involve some python, and potentially some web work. Is anyone interested?

If you have some technical skills and want to help out, please do get in
touch with myself, DanChapman or dkessel. The goal behind this effort is
to see these tests be useful again this cycle for image testing and
lowering the burden for manual testers.

Thanks!

Nicholas

[Nicholas Skaggs]

Post by Nicholas Skaggs
Greetings everyone! I wanted to share some news about a renewed effort
to restore automated installer testing of ubiquity using the daily
images generated by cdimage. Up until last cycle, the images were
being test automatically via a series of autopilot tests, written
originally by the community (kudos to you Dan!). It was noticed the
tests didn't run this cycle, and wxl accordingly filed an RT;
https://rt.ubuntu.com/Ticket/Display.html?id=26570. In short, as part
of some datacenter shuffling, we learned it's no longer possible for
CI to run or maintain the tests. They recommended we host and run them
ourselves as a community.
With the directions from CI in hand, I initially asked DanChapman and
dkessel to investigate setting up a jenkins to run these tests. But
they need your help! The autopilot tests for ubiquity have a few bugs
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bugs?field.tag=autopilot.
http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/view/head:/autopilot/README.md.
It will guide you through running the tests yourself locally. You
should be able to replicate the tests following those instructions.
From there, patches and comments on the bugs would be most welcome (as
would additional bug reports should you find them).
Even with the tests working, however, we still need to setup a jenkins
to run them. We'll also need to maintain this server. Anyone with
experience or desire in this area? Ideas for reporting results (on the
isotracker for instance) also need to be explored. This is likely to
involve some python, and potentially some web work. Is anyone interested?
If you have some technical skills and want to help out, please do get
in touch with myself, DanChapman or dkessel. The goal behind this
effort is to see these tests be useful again this cycle for image
testing and lowering the burden for manual testers.
Thanks!
Nicholas

Thanks to everyone who expressed an interest! I've created a document
that will help answer your questions about what we are trying to do, and
perhaps generate some more. http://bit.ly/1IO103i
<http://t.co/Rj0w6yQefI>. Please leave your feedback and ideas inside
the document.

In addition, for all those who are able and want to help out, we'll be
having a meeting tomorrow, July 31st, at 1900 UTC in #ubuntu-quality on
freenode. We'll go over the document, and finalize the plan of work.
We'll also start to divvy up the needed tasks and pick a good day and
time for future meetings as needed. If you miss the first one, don't
worry, stay involved.

You can join the channel easily via webchat;
http://webchat.freenode.net/?channels=ubuntu-quality

I hope to see you there!

Nicholas

[Nicholas Skaggs]

Post by Nicholas Skaggs

Post by Nicholas Skaggs
Greetings everyone! I wanted to share some news about a renewed
effort to restore automated installer testing of ubiquity using the
daily images generated by cdimage. Up until last cycle, the images
were being test automatically via a series of autopilot tests,
written originally by the community (kudos to you Dan!). It was
noticed the tests didn't run this cycle, and wxl accordingly filed an
RT; https://rt.ubuntu.com/Ticket/Display.html?id=26570. In short, as
part of some datacenter shuffling, we learned it's no longer possible
for CI to run or maintain the tests. They recommended we host and run
them ourselves as a community.
With the directions from CI in hand, I initially asked DanChapman and
dkessel to investigate setting up a jenkins to run these tests. But
they need your help! The autopilot tests for ubiquity have a few bugs
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bugs?field.tag=autopilot.
http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/view/head:/autopilot/README.md.
It will guide you through running the tests yourself locally. You
should be able to replicate the tests following those instructions.
From there, patches and comments on the bugs would be most welcome
(as would additional bug reports should you find them).
Even with the tests working, however, we still need to setup a
jenkins to run them. We'll also need to maintain this server. Anyone
with experience or desire in this area? Ideas for reporting results
(on the isotracker for instance) also need to be explored. This is
likely to involve some python, and potentially some web work. Is
anyone interested?
If you have some technical skills and want to help out, please do get
in touch with myself, DanChapman or dkessel. The goal behind this
effort is to see these tests be useful again this cycle for image
testing and lowering the burden for manual testers.
Thanks!
Nicholas

Thanks to everyone who expressed an interest! I've created a document
that will help answer your questions about what we are trying to do,
and perhaps generate some more. http://bit.ly/1IO103i
<http://t.co/Rj0w6yQefI>. Please leave your feedback and ideas inside
the document.
In addition, for all those who are able and want to help out, we'll be
having a meeting tomorrow, July 31st, at 1900 UTC in #ubuntu-quality
on freenode. We'll go over the document, and finalize the plan of
work. We'll also start to divvy up the needed tasks and pick a good
day and time for future meetings as needed. If you miss the first one,
don't worry, stay involved.
You can join the channel easily via webchat;
http://webchat.freenode.net/?channels=ubuntu-quality
I hope to see you there!
Nicholas

A big thank you to everyone who came out today to talk through the next
steps. Our next meeting will be 1900 UTC on Thursday August 6th in
#ubuntu-quality on freenode. Please join us if you are able!

We took some actions for the next meeting.

-svij and shrini agreed to setup a test jenkins instance to help answer
our lingering questions on what we need. Specifically they'll be looking at
where should we host this?
can we test in the cloud?
what type of setup should we have (how many slaves, how many instances)?
and trying to get us all setup with a jenkins instance we can add jobs
to and iterate on moving forward.

-flocculant took an action to investigate when the images came available
and gave feedback on avoiding failing images and those being built only
for LTS releases as part of the initial release.

-DanChapman agreed to create a readme for debugging autopilot and
ubiquity in order to help fix and maintain the current autopilot test
suite.

-shrini also volunteered to help out by looking into the failing tests


Finally, we discussed reporting and determined simple would be best at
first, coming up with a couple easy ideas to choose between later.

Again, thanks to everyone who volunteered! That said, the are still some
outstanding needs you can help us with!

We need folks willing to help fix and maintain the ubiqiuty autopilot
tests. There is some learning curve, but you won't be alone. Dan has
offered to help guide anyone who wants to undertake this work. The first
step would be fixing the known bugs within the tests so they can run
successfully.

We also needsomeone who knows python to have a look at the isotracker
API, and create a script to update the notice board with the latest
results.This was the favored idea by those in attendance at the meeting
for reporting the build results to everyone in the community. The API is
here: http://iso.qa.ubuntu.com/qatracker/api. Note, you need to login to
see it!


Nicholas