Status report: 2013-04-15 - 2013-04-21

A really long time without activity. AsiaBSDCon 2013, vacations, coming back to work, and a toddler with self-mobility kill all chances of having some spare time. Anyway:

• Implemented support to filter particular result types out of HTML reports. This is a requisite for generating reports for the HTML test suite, as including details for all passed results increases the size of the reports (and the number of files to sync across mirrors) significantly.
• Fixed some build failures in FreeBSD 9.1 powerpc64 with Clang.

Status report: 2013-03-04 - 2013-03-10

• Modified Anita to transparently use Kyua when the NetBSD system being tested was built with MKKYUA=yes. Sent the patch for review and I'm now awaiting integration upstream.

Other than this, there hasn't been much activity last week. I've been incredibly busy closing things out in preparation for AsiaBSDCon 2013. As a matter of fact, I'm almost on my way to the airport! Unfortunately, I didn't expect to have almost-finished the import of Kyua into NetBSD by now, and that's the reason why I didn't apply for a slot during the Call for Papers :-(

Status report: 2013-02-18 - 2013-02-24

There is only one big —well, actually two— news items this last week:

• Released kyua-cli-0.6.
• Imported kyua-testers, kyua-cli and kyua-atf-compat into the NetBSD source tree!

The goal set two and a half years ago has finally materialized, although it's nowhere near completion.

As a quick note, you can start playing with this by setting MKKYUA=yes in your build configuration. I still have to ready some minor changes and send a formal announcement with all the details. Stay tuned!

Status report: 2013-02-11 - 2013-02-17

• Released ATF 0.17 and imported it into NetBSD.
• Added caching support to the configure tests of kyua-cli so that cross-building is possible.
• Started import of Kyua into NetBSD by adding support for MKKYUA to the build system and importing Lutok 0.2.
• Build-tested the preliminary release of kyua-testers under NetBSD i386 and amd64, and Fedora 18 and rawhide.
• Wrote the pkgsrc and RPM packages for kyua-testers.
• Prepared changes to import kyua-testers into NetBSD.

I'm working on the kyua-testers import into NetBSD as we speak, but this could not happen in time for last week's report (aka this report) ;-P

Status report: 2013-02-04 - 2013-02-10

I spent 10 hours in a bus this week and had some incredibly productive time:

• Fixed kyua help to not fail when the configuration file is bogus. Getting help is something that should work all the time. (Issue 36.)
• Simplified the syntax() call in configuration and Kyuafile files to not include the format name. The format name is implied and was unnecessary. (Issue 37.)
• Added the metadata of test cases to HTML reports. This is handy for postmortem debugging.
• Stopped wrapping text on screen boundaries for most messages printed by kyua. Doing wrapping with newline characters broke copy/pasting and was incredibly annoying. This now only happens in the output of kyua help, which is the only thing that should look "nice".
• Rewrote the Kyuafile parser in C++ instead of relying on an external Lua file. The resulting code is lightyears clearer and helped to removal of a bunch of poorly-understood code. The kyua binary is now self-contained (modulo some non-critical distribution documents).
• Fixed the configure checks that look for GNU Getopt to work in cross-compilation environments. (In ATF only; still getting this to work with the Kyua tests.)
• Fixed a bunch of run-time issues found when running under Debian testing on my shiny-"new" PowerMac G5.
• And, lastly, updated the import plan into NetBSD and notified tech-userlevel@ about the impeding changes.

So, stay tuned: I shall start importing a new release of ATF and Lutok into the NetBSD source tree Real Soon (TM) now. Once those are done, kyua-testers and kyua-cli will follow after they pass release tests in a variety of systems. core@ has given permission to do these imports as incremental steps, provided that they respect the new MKKYUA variable, which will simplify this process significantly.

Status report: 2012-12-31 - 2013-01-06

• Fixed the performance regression in kyua report introduced in the testers branch and, actually, made it run faster than ever.
• The generation of a report for the Kyua test suite itself used to take about 12 seconds. However, the schema changes that went in with the generalization of the test interfaces pushed the run time to over 2 minutes. Now, with the addition of a missing primary key (duh!) and a couple of indexes, the same report generation takes less than 2 seconds.
• Wrote the migration code to upgrade an old-schema database to the new generalized schema. I have not submitted this yet as I'm busy writing tests to ensure the code works fine, and making them general enough so that introducing a new schema change in the future is easier to test than today.

Status report: 2012-12-24 - 2012-12-30

Good news everyone!

• Initial kyua-testers code published. It took a long time, but it has finally happened.
• After this announcement, continued the work to clean up the new testers branch in the main kyua-cli repository. At this point, the latter has lost 5K lines of code and got another 5K lines replaced with more generic and simpler abstractions. There are more changes to come that will make things leaner, but these are starting to become tricky to implement.

Have a happy beginning of 2013 :-)

Status report: 2012-12-17 - 2012-12-23

Long time since the last report. Have been busy sorting things out post-Sandy and with a work trip... But, finally, I have been able to put some hours of work into Kyua these last few days:

• Integrated the independent testers into Kyua, and got all tests to pass. The work of the last two months if finally seeing its results!
• Started the removal of all code in the kyua binary that implemented the execution of tests in-process. There is still quite a few code that needs yanking though.
• Many cleanups to the testers code in preparation for publication: fixing pending to-dos and adding tests.

Stay tuned for the shortcoming announcement about the publication of the testers and a branch with the code to use them. Once this happens, I'll still have to address a serious performance regression in kyua report before this can hit mainline.

Status report: 2012-11-19 - 2012-11-25

• Got kyua list to work with the independent testers and pass all tests!
• Started cleaning up the testers code so that it can be published.
• Initial work to get kyua test to use the testers too. Most tests pass, but there are still some rough corners (obviously, as the implementation is still full of to-do entries).

Status report: 2012-11-05 - 2012-11-11

• Completed preliminary implementation of the ATF tester.
• Wrote (very) basic integration tests for the plain and ATF testers.
• Changed Kyua to use the external testers for test case listings. Working for the most part, but still some rough edges left that will probably require a bunch of work.

The code for the testers is starting to be pretty decent, so I should be publishing these as a new module in the Git repository soon. Keeping so much unsubmitted code in my machine is not a smart move, as Hurricane Sandy showed recently... Stay tuned!

Status report: 2012-10-29 - 2012-11-04

This last week has been eclipsed by hurricane Sandy. Even though I could not go to work, I haven't been able to put much time in Kyua either. That said:

• Implemented the parsing of ATF test case lists in the ATF tester.
• Implemented the parsing of ATF results in the ATF tester.
• Tweaked my Emacs configuration files to highlight test errors in the compilation buffer. Jumping to test failures right from there is cool. (My idea of providing a "kyua-mode" may not be so difficult after all...)

The independent testers are now to a point where integration with the main Kyua command line can start — yes, they are still woefully incompletely, but seeing the results of this last month of work would be cool. The problem is... I just found out that my laptop does not have a copy of that work: all the experimental code is sitting at my still-powerless home in my desktop computer... gotta grab it soon to recover the files!

Status report: 2012-10-15 - 2012-10-21

• Finished porting the work directory cleanup logic to C so that it can be used by the independent testers.
• Moved the independent tester's code to a separate package. This is still a proof of concept, and it's easier for it to be developed this way. I'm not yet sure whether these will end up being part of the main Kyua package or not.
• Spent quite a bit of time setting up my Mac Mini G4 as a development machine. Yes, that machine is very slow, but that's one of the "good things": I'm trying to cut down bloat from the existing code, and I'll be "done" when the build times on such machine are reasonable!

Status report: 2012-10-08 - 2012-10-14

Wow, been so busy this week that I forgot to write this earlier...

• Ongoing work on the separate testers implementation. In particular, started migrating some of the "fs-management" code previously written in C++ to C so that it can be used from the separate binaries.

Status report: 2012-10-01 - 2012-10-07

• Generalized test helper functions into new "utils" modules in ATF. These provide common functionality needed when having to deal with test data files from within C and C++ code, for example. The result is that the Makefiles of Kyua are significantly simplified.
• More work towards independent, self-contained testers: cleaned up some existing code and added tests for some core functionality. Still not published though.

Status report: 2012-09-24 - 2012-09-30

Significant amount of progress towards the TestersDesign idea:

• Got approval from NetBSD's core to proceed with my integration plan.
• Introduced a new metadata object to hold the properties of a test case or a test program. The idea is that both will be able to carry the same properties at a later stage, and thus plain test programs will be able to share most of the features exposed to ATF-based test cases.
• Got rid of the test case and test program class hierarchies. Instead, just rely on an "interface type" field and generalize all the other concepts. This is a prerequisite for separate, pluggable testers. The result is a much simpler codebase and database schema. (Still mostly a "prototype" though, as there are many rough edges left behind.)
• Continued work on my preliminary C-based testers. So far focusing on extending ATF with a variety of helper functions that I frequently need to test command-line interfaces. (Should have done this ages ago...)

Update (20:46 EST): Forgot to include the most exciting news, which are now shown in the first item of the list!

Status report: 2012-09-17 - 2012-09-23

Belated report for last week. I've been so busy this week so far that I even forgot to look at my scheduled to-dos...

• More work in getting rid of the test case and test program class hierarchies for simplicity reasons. In particular, many different attempts at generalizing the representation of a test's metadata. Most of these attempts lead to dead ends, but finally chose one that might work.

Meh, we are already mid-week and I have exciting news to share! But I'll keep them for the next report. I promise: the above one-liner does not include things that happened since this Monday.

Status report: 2012-09-10 - 2012-09-16

• Wrote a plan on how to modularize the code that implements the different test interfaces. This would result in a much simpler code base and in a bunch of code being rewritten in C (thus reducing build times).
• Started work on the previous plan by:
• Simplifying the test case and test program abstractions to allow dynamic loading. In particular, this gets rid of class hierarchies in favor of just generalizing their implementations. The side-effect of this will be that pretty much all the requirement checks provided by the ATF interface will also be available in the plain interface.
• Writing some preliminary code for the testers. Have got the basic command line processing implemented and some basic form of plain test case execution.
• Sent the plan to import Kyua into NetBSD to the NetBSD core@ team for approval.

Status report: 2012-09-03 - 2012-09-09

• Converted the documentation from texinfo to manpages. This was a suggestion raised during the review of the plan to integrate Kyua into NetBSD. I had been thinking about this for a while and the fact that somebody else reminded me of it made me give the change a chance. And after finishing with this, I am quite happy with the results actually.
• Changed the exit status codes of Kyua to provide different numbers on different types of errors.

Status report: 2012-08-27 - 2012-09-02

• Wrote a plan on how to integrate Kyua into NetBSD so that it's ready by 7.0.
• Sent the plan for review to tech-userlevel@.

Status report: 2012-08-20 - 2012-08-26

You may have noticed that there was no status report last week. It was not an overlook: I find it frustrating to write reports when there has been no project activity, so I'm not going to write those any more :-P

But, finally, some things happened this week!

• Integrated a couple of ATF and bsd.test.mk fixes from Garrett Cooper, which come from his Testing FreeBSD project.
• Created initial reachover Makefiles to integrate Kyua into NetBSD. These only cover the kyua binary for now (i.e. none of the bundled tests). Yes, it's extremely boring to write these files...
• The previous point triggered a couple of contributions to NetBSD: the addition of the -P option to mkdep(1), which is needed to support building Kyua's libutils from a single Makefile; and the addition of placeholder unit tests for share/mk, which are needed to validate the tweaks that will be necessary to bsd.dep.mk at a later stage.