Status report: 2012-12-24 - 2012-12-30

Good news everyone!

• Initial kyua-testers code published. It took a long time, but it has finally happened.
• After this announcement, continued the work to clean up the new testers branch in the main kyua-cli repository. At this point, the latter has lost 5K lines of code and got another 5K lines replaced with more generic and simpler abstractions. There are more changes to come that will make things leaner, but these are starting to become tricky to implement.

Have a happy beginning of 2013 :-)

Initial kyua-testers code published

Three months, a hurricane and an apartment move after the proposal to modularize testers as independent binaries, I am pleased to announce the publication of the initial code implementing such proposal! It has been a long way full of interrupts, but this marks a major milestone in the development of Kyua.

The code for the independent testers can be found in the new kyua-testers module, which weights at ~10K lines of C code (yep, no C++) and currently provides testers for ATF-based and old-style (framework-less) test programs. As the documentation describes, the goal of these programs is to provide a scriptable and generic interface to run diverse test programs. This provides the ability to: first, plug new testers dynamically into kyua(1)(e.g. imagine a new tester for googletest-based test programs); and, second, to simplify the code of the frontend by removing tricky low-level code so that, potentially, most of kyua(1) could be latter rewritten in Lua.

The changes to make kyua-cli use kyua-testers are now in a separate branch of kyua-cli called testers, which drops ~4K lines of C++ code (much more slaying to come). While these changes already work and do not appear to introduce functional regressions, there are still two major issues to fix:

1. The first pending item is to address a serious performance regression in kyua report. This regression is understandable considering how horrendous the database querying code is and the generalization of the database schema that went in with this refactoring. Conceptually easy to fix, but the database-backing code needs some serious cleanup.
2. The second pending item is to improve the code in kyua-cli to execute the testers, which in turn will obsolete —and thus enable the removal of— lots of nasty and unnecessary abstractions. The current implementation to do this has been a hack of mine in an attempt to get things to work as quickly and as unobtrusively as possible. Now that the code is in place and that all the tests pass, it will be easier to reorganize the code.

If you inspect the code, you will realize that the engine/atf_iface/ and engine/plain_iface/ directories of kyua-cli are now gone. This is a big win because the abstractions in kyua-cli (those representing test programs, test cases and test results) are now really unaware of the specific interface implemented by the test programs being executed.

As soon as the two above items are addressed, I will start evaluating the merge of the testers branch into master, which should unblock the integration of Kyua into NetBSD! Stay tuned.

Status report: 2012-12-17 - 2012-12-23

Long time since the last report. Have been busy sorting things out post-Sandy and with a work trip... But, finally, I have been able to put some hours of work into Kyua these last few days:

• Integrated the independent testers into Kyua, and got all tests to pass. The work of the last two months if finally seeing its results!
• Started the removal of all code in the kyua binary that implemented the execution of tests in-process. There is still quite a few code that needs yanking though.
• Many cleanups to the testers code in preparation for publication: fixing pending to-dos and adding tests.

Stay tuned for the shortcoming announcement about the publication of the testers and a branch with the code to use them. Once this happens, I'll still have to address a serious performance regression in kyua report before this can hit mainline.

Status report: 2012-11-19 - 2012-11-25

• Got kyua list to work with the independent testers and pass all tests!
• Started cleaning up the testers code so that it can be published.
• Initial work to get kyua test to use the testers too. Most tests pass, but there are still some rough corners (obviously, as the implementation is still full of to-do entries).

Status report: 2012-11-05 - 2012-11-11

• Completed preliminary implementation of the ATF tester.
• Wrote (very) basic integration tests for the plain and ATF testers.
• Changed Kyua to use the external testers for test case listings. Working for the most part, but still some rough edges left that will probably require a bunch of work.

The code for the testers is starting to be pretty decent, so I should be publishing these as a new module in the Git repository soon. Keeping so much unsubmitted code in my machine is not a smart move, as Hurricane Sandy showed recently... Stay tuned!

Status report: 2012-10-29 - 2012-11-04

This last week has been eclipsed by hurricane Sandy. Even though I could not go to work, I haven't been able to put much time in Kyua either. That said:

• Implemented the parsing of ATF test case lists in the ATF tester.
• Implemented the parsing of ATF results in the ATF tester.
• Tweaked my Emacs configuration files to highlight test errors in the compilation buffer. Jumping to test failures right from there is cool. (My idea of providing a "kyua-mode" may not be so difficult after all...)

The independent testers are now to a point where integration with the main Kyua command line can start — yes, they are still woefully incompletely, but seeing the results of this last month of work would be cool. The problem is... I just found out that my laptop does not have a copy of that work: all the experimental code is sitting at my still-powerless home in my desktop computer... gotta grab it soon to recover the files!

Status report: 2012-10-15 - 2012-10-21

• Finished porting the work directory cleanup logic to C so that it can be used by the independent testers.
• Moved the independent tester's code to a separate package. This is still a proof of concept, and it's easier for it to be developed this way. I'm not yet sure whether these will end up being part of the main Kyua package or not.
• Spent quite a bit of time setting up my Mac Mini G4 as a development machine. Yes, that machine is very slow, but that's one of the "good things": I'm trying to cut down bloat from the existing code, and I'll be "done" when the build times on such machine are reasonable!

Status report: 2012-10-08 - 2012-10-14

Wow, been so busy this week that I forgot to write this earlier...

• Ongoing work on the separate testers implementation. In particular, started migrating some of the "fs-management" code previously written in C++ to C so that it can be used from the separate binaries.

Status report: 2012-10-01 - 2012-10-07

• Generalized test helper functions into new "utils" modules in ATF. These provide common functionality needed when having to deal with test data files from within C and C++ code, for example. The result is that the Makefiles of Kyua are significantly simplified.
• More work towards independent, self-contained testers: cleaned up some existing code and added tests for some core functionality. Still not published though.

Status report: 2012-09-24 - 2012-09-30

Significant amount of progress towards the TestersDesign idea:

• Got approval from NetBSD's core to proceed with my integration plan.
• Introduced a new metadata object to hold the properties of a test case or a test program. The idea is that both will be able to carry the same properties at a later stage, and thus plain test programs will be able to share most of the features exposed to ATF-based test cases.
• Got rid of the test case and test program class hierarchies. Instead, just rely on an "interface type" field and generalize all the other concepts. This is a prerequisite for separate, pluggable testers. The result is a much simpler codebase and database schema. (Still mostly a "prototype" though, as there are many rough edges left behind.)
• Continued work on my preliminary C-based testers. So far focusing on extending ATF with a variety of helper functions that I frequently need to test command-line interfaces. (Should have done this ages ago...)

Update (20:46 EST): Forgot to include the most exciting news, which are now shown in the first item of the list!

Status report: 2012-09-17 - 2012-09-23

Belated report for last week. I've been so busy this week so far that I even forgot to look at my scheduled to-dos...

• More work in getting rid of the test case and test program class hierarchies for simplicity reasons. In particular, many different attempts at generalizing the representation of a test's metadata. Most of these attempts lead to dead ends, but finally chose one that might work.

Meh, we are already mid-week and I have exciting news to share! But I'll keep them for the next report. I promise: the above one-liner does not include things that happened since this Monday.

Status report: 2012-09-10 - 2012-09-16

• Wrote a plan on how to modularize the code that implements the different test interfaces. This would result in a much simpler code base and in a bunch of code being rewritten in C (thus reducing build times).
• Started work on the previous plan by:
• Simplifying the test case and test program abstractions to allow dynamic loading. In particular, this gets rid of class hierarchies in favor of just generalizing their implementations. The side-effect of this will be that pretty much all the requirement checks provided by the ATF interface will also be available in the plain interface.
• Writing some preliminary code for the testers. Have got the basic command line processing implemented and some basic form of plain test case execution.
• Sent the plan to import Kyua into NetBSD to the NetBSD core@ team for approval.

Modularizing testers as independent binaries

While waiting for an approval to get Kyua into NetBSD (just sent the plan to the core team), I have started to think about how to implement a highly desired feature: that is, the parallel execution of test cases.

Theoretically, this should be relatively simple in most cases because test cases are independent entities and because they are run in an isolated manner precisely to allow this. However, the current implementation of the test case isolation makes this very hard — or rather, extremely hard to get right — as all the logic would be in a single process and would require threads.

I have written a more detailed exposition of the problems to solve and the idea that is floating my mind in the new TestersDesign wiki page. Please take a look!

(Alright, alright: sincerely speaking, the major reason I am considering this restructuration right now is because the new code could be easily written in C and a lot of C++ code could be dropped from Kyua... which could be a major selling point for NetBSD users. But it's true that such a design could make the whole implementation clearer and the addition of parallel execution of tests much simpler!)

Status report: 2012-09-03 - 2012-09-09

• Converted the documentation from texinfo to manpages. This was a suggestion raised during the review of the plan to integrate Kyua into NetBSD. I had been thinking about this for a while and the fact that somebody else reminded me of it made me give the change a chance. And after finishing with this, I am quite happy with the results actually.
• Changed the exit status codes of Kyua to provide different numbers on different types of errors.

Status report: 2012-08-27 - 2012-09-02

• Wrote a plan on how to integrate Kyua into NetBSD so that it's ready by 7.0.
• Sent the plan for review to tech-userlevel@.

Importing Kyua into NetBSD: the plan

After years of development (and long pauses due to "real life"), the time to propose the import of Kyua into NetBSD has come. This needs to happen for two reasons: first, to force me to finish the little details necessary for this transition; and, second, to finally get rid of the old ATF code and allow us to have shiny new features in Kyua. And today is just as good as any other time to start the process.

The detailed plan can be found in the "Kyua: The way into NetBSD" wiki page and the initial discussion will take place in the tech-userlevel@ "Importing Kyua into NetBSD" thread.

Let the commenting start!

Status report: 2012-08-20 - 2012-08-26

You may have noticed that there was no status report last week. It was not an overlook: I find it frustrating to write reports when there has been no project activity, so I'm not going to write those any more :-P

But, finally, some things happened this week!

• Integrated a couple of ATF and bsd.test.mk fixes from Garrett Cooper, which come from his Testing FreeBSD project.
• Created initial reachover Makefiles to integrate Kyua into NetBSD. These only cover the kyua binary for now (i.e. none of the bundled tests). Yes, it's extremely boring to write these files...
• The previous point triggered a couple of contributions to NetBSD: the addition of the -P option to mkdep(1), which is needed to support building Kyua's libutils from a single Makefile; and the addition of placeholder unit tests for share/mk, which are needed to validate the tweaks that will be necessary to bsd.dep.mk at a later stage.

Status report: 2012-08-06 - 2012-08-12

• Unfortunately, no activity either. Still working on scripts for NetBSD to simplify the maintenance of my machines, which in the end will come very handy and give me more time for Kyua :-P

Status report: 2012-07-30 - 2012-08-05

• No progress either. Been busy with sysupgrade, whose need also appeared with the necessity to rebuild by NetBSD development machine. (Long term, less manual maintenance of my development environment means more time for Kyua!)

Status report: 2012-07-23 - 2012-07-29

• No progress. Been busy with other NetBSD stuff (aka sysbuild) as the need for this appeared when trying to rebuild my development machine...

Status report: 2012-07-16 - 2012-07-22

• Started the integration of Kyua into the NetBSD build system and so far only got Lutok in place. Then got distracted trying to use Git to manage NetBSD sources so that keeping these local patches becomes sane...

Status report: 2012-07-09 - 2012-07-15

• Published kyua-0.5, atf-0.16 and the new kyua-atf-compat-0.1.
• Wrote a more detailed introduction to Kyua for NetBSD users, and got two bug reports as a result so far. (The page has not been widely announced within NetBSD yet.)

An introduction to Kyua for NetBSD users

The Kyua wiki site used to contain a pretty brief document detailing how to run the NetBSD test suite using Kyua. Such document was quite incomplete and out of place: a document for NetBSD users should really be placed on the NetBSD site, and give all the relevant information for such users.

To start paving the way for the integration of Kyua into NetBSD (which is close, but not that close), I have spent the day writing a more detailed introduction to Kyua for NetBSD users. The main purpose of such document, at this point, is to explain what benefits Kyua brings to NetBSD and how to use the current Kyua from pkgsrc to run the NetBSD test suite. (Eventually, I hope the document to include a detailed transition proposal, to be able to discuss it with other developers.)

The new document lives in the NetBSD Wiki under the page "Kyua: an introduction for NetBSD users". Please give it a look and send feedback!

Kyua 0.5 released, and more!

Today is a release day:

• kyua-cli-0.5: The main highlight of this new version is that it achieves (what I think is) full feature parity with the atf-run and atf-report utilities, meaning that the latter can be finally replaced.
• kyua-atf-compat-0.1: The release of the Kyua-based ATF compatibility tools was blocked on the release of kyua-cli 0.5. Finally, this new package is now available for download. It should provide drop-in replacements for the atf-run and atf-report utilities, but using the Kyua binary.
• atf-0.16: Miscellaneous improvements. I would say that the major changes are the disabling of the build of the tools by default, and the decision to start versioning the shared libraries at all, but these hardly have an impact on users at the moment. This new version also includes some backported fixes from NetBSD.

The links above will take you to the specific release pages which include the list of major changes in every release.

Now that Kyua is officially two components, I'm wondering how I should be naming the releases... On the one hand, saying "Kyua 0.5" may now be misleading, but on the other hand, keeping all the components at the same release number means that users will have to go through upgrades for no reason (as not all components will include changes on every upgrade). Anyway, this is something to think about later. Now, enjoy the new features!

Status report: 2012-07-02 - 2012-07-08

• Added automatic stacktrace collection of crashed test cases to Kyua. This brings us one step closer to Kyua 0.5, which according to the plan is almost ready.
• Decided to start versioning the shared libraries in ATF and got rid of the --enable-unstable-shared hack.

Gathering stacktraces of crashed test cases

One major feature that will be part of Kyua 0.5 — just landed onto HEAD — is the automatic collection of stacktraces for any crashing test case. This feature was originally implemented in ATF 0.13 and it had to make its way to Kyua as part of the ATF tools deprecation. As the linked commit message well mentions, the implementation of this functionality in ATF was a non-portable hack. So, while rewriting this feature for Kyua, I have attempted to come up with a more portable and, specially, testable solution.

Collecting stacktraces is a tricky business. Attempting to do it "by hand" — i.e. by reading a core file and walking the stack manually while resolving symbols possibly in shared libraries and dealing with all sorts of platforms/architecture differences — is just crazy talk. Fortunately, GDB has the ability to do this and, even better, comes with a batch mode that allows the unattended collection of a stacktrace.

Here is how, given a binary and a core dump, you generate a stacktrace:

$ gdb -batch -q -ex bt binary-name core-dump

And this is how the new stacktrace module goes by collecting stacktraces: it just calls out to GDB (if present) to gather the stacktrace and append it to the test case's stderr output.

Easy enough, huh? Well, yes, but you still need to determine what the name of the core file is! And that is, of course, platform specific and (even worse) user-customizable. Here are some examples:

• NetBSD places the core files in the current directory and names them after the binary that generated them, with a .core suffix. The names have a maximum length of MAXCOMLEN. To make things trickier, this is all customizable through the kern.defcorename sysctl setting.
• Mac OS X places core files in /cores/ using a naming scheme of the form core.PID. This is also customizable through the kern.corefile sysctl setting.
• Fedora places core files in the current directory with a naming scheme of core.PID. I guess this is also customizable, but I haven't investigated how.

In the current implementation, I have special-cased all these default values in the find_core() function, which has been tested to work on all the operating systems above mentioned. I could go one step further and attempt to cope with the user customizations, but haven't done so at this point.  (Why? Because I wasn't aware that this was all tunable until I wrote this post!)

Lastly, and as mentioned earlier, the new code comes with unit and integration tests to ensure that the gathering of stacktraces works. This is something that the ATF codebase did not have, and thus is why nobody spotted that this did not work in any other system than NetBSD.

Status report: 2012-06-25 - 2012-07-01

To be honest, I have not had any time to work on Kyua during the last week. The reason: my wife and I welcomed our first child on Wednesday 27th and have been busy since! :-) I'm not expecting to have a lot of time either during the next few weeks...

Anyway, the project saw these minor changes:

• Split the atf libraries in pkgsrc to their own atf-libs package, and applied a couple of fixes. This is a necessary step for the addition of the upcoming kyua-atf-compat package.
• Pushed the kyua-cli (and atf and lutok) package updates in Fedora to the stable track. Reminder: the updates provide a new tests subpackage that installs the run-time test suites, ready to be run by end users. See the README.Fedora file in the documentation directory for further details.

Status report: 2012-06-18 - 2012-06-24

• Issue 34: Added the contents of the stdout and stderr of test cases to the HTML reports.
• Ongoing work to add automatic stacktrace gathering of crashed test cases.
• Split the libraries out of the atf package and placed them in a new atf-libs package in pkgsrc, and created a package for kyua-atf-compat. Because pkgsrc is currently undergoing a freeze in preparation for a new stable branch, these changes have not been submitted yet.

Tests packaged for Fedora

The initial versions of the atf, lutok and kyua-cli packages that landed in Fedora did not install the bundled test suites. This was done to simplify the initial addition of the packages to the distribution, as using /usr/tests right away for the installation of the tests would have probably been a package review blocker.

With the original packages already in place, I have now changed them to supply an additional -tests package that installs the test suite. Because the use of /usr/tests has not yet been discussed in detail (and obviously not approved yet), I have opted to go for a compromise in the meantime. These test packages install the test suites under /usr/libexec/<package>/tests and they include a README.Fedora file explaining why the tests are in such location and how to run them.

No matter what, running the test suite of any of these packages is actually pretty simple. For example:

# yum install kyua-cli kyua-tests
$ kyua test -k /usr/libexec/kyua-cli/tests/Kyuafile

Note that, at the moment, these new -tests packages are only available in the updates-testing repository of Fedora 17 and will not move onto updates for over a week.

Status report: 2012-06-11 - 2012-06-17

• Changed ATF's build system to not build the end-user tools (atf-run and atf-report) by default. The build of these can still be explicitly requested by passing --enable-tools to the configure script.
• Wrote preliminary compatibility wrappers for atf-run and atf-report, accompanying tests and manual pages. These have all gone live in a new kyua-atf-compat module.
• Pushed the kyua-cli package to the stable updates repository of Fedora 17. Just type yum install kyua-cli on your Fedora 17 (or rawhide) system and start enjoying Kyua!

New Kyua-based versions of atf-run and atf-report

The plan for the upcoming Kyua 0.5 is to have full feature parity with the obsolete atf-report so that we can finally say goodbye to both atf-run and atf-report. The fact that these tools still exist and are in widespread use prevents us from adding much-needed features to Kyua (for compatibility reasons). It is therefore important to get rid of them as early as possible. We are almost there, except for a few details that need to be addressed in the report generation.

As part of this plan, ATF 0.16 will not build the end-user tools by default: they will still be there, but only built whenever the user asks configure to do so. Users will instead be directed to use Kyua itself.

But, of course, making binaries unreachable by the user is never nice. And this is why, in order to simplify the transition from the ATF tools to Kyua, I have written a bunch of simple scripts that reimplement the major features of atf-run and atf-report on top of Kyua. They don't behave in exactly the same manner, but they are close enough to support the most common command line invocations: in particular, a bunch of our users have atf-run | atf-report ingrained in their minds, so it is important to keep this basic use case working to mitigate their frustration (if any).

With all that said, it is time to say hello to the new kyua-atf-compat module! This new package includes the following tools:

• atf2kyua(1): A tool to convert existing Atffiles to Kyuafiles. This was previously part of the kyua-cli package, had the name kyuaify.sh and was shipped as a miscellaneous script. It has now been promoted to a first-class tool, with a manual page and functionality tests.
• atf-run(1): Reimplementation of ATF's runtime engine on top of kyua test.
• atf-report(1): Reimplementation of ATF's reporting engine on top of kyua report.

Testers welcome, of course. A formal release with its corresponding binary packages will follow soon (before Kyua 0.5 and ATF 0.16) so that we can get some mileage on this new code.

Status report: 2012-06-04 - 2012-06-10

Now that Kyua has its own blog, the weekly status reports are back as promised! I'm glad to see that this last week has been quite fruitful:

• Released Kyua 0.4.
• Implemented support for "build directories".
• Got a kyua-cli package into Fedora 17's updates-testing repository.
• Thoughts on what the upcoming Kyua 0.5 release should focus on. The goal is for this release to include all the functionality needed to replace both atf-run and atf-report, and to provide drop-in replacements for these obsolete tools.
• Wrote some tentative code to add the contents of a test case's stdout and stderr to the HTML reports. The implementation of this feature is pretty simple, but I still need to write the corresponding tests before submission (not that easy).

Fedora 17 package available for testing

This last week, I have been working on getting a new kyua-cli package into the Fedora repositories. The obvious goal is to reach more users by providing an easy mechanism for Fedora consumers to install Kyua. Just now, the package has been pushed to the updates-testing repository, which means it is available for testing before it gets moved to the regular updates repository.

So, go ahead, issue yum install kyua-cli (or use any graphical frontend) and start playing with Kyua! Do not hesitate to report any problems that you encounter, or to give karma if the package works fine for you. You may need to enable the updates-testing repository first though.

One thing to note is that none of the packages that are Kyua-related (atf, kyua-cli and lutok) install their test suite as they should be doing (yet). This is because the ideal place for such tests would be under /usr/tests/ as we do in NetBSD, but this directory is not allowed by the Linux Standard Base guidelines. While I think what the best way to resolve this is, I'm going to modify the packages to place the tests elsewhere in the meantime (most likely somewhere under /usr/libexec/)... so expect some -tests packages soon!

Support for build directories

Build directories are a mechanism for keeping the source tree and its build products separate: during a build, the source tree remains read only and all the results of the build are placed in a different directory. It is often considered good practice to use build directories, as this ensures that the build does not have any effects on the sources and permits creating different binaries (think about "debug" vs. "production" binaries) from the same source tree without an intermediate make clean.

Most, if not all, modern build systems support the concept of build directories. The GNU Automake/Autoconf combination is not a exception, although I believe that not many people are aware of this. However, those that are probably use this feature routinely.

Up until now, integrating Kyua into a project that used build directories had always been cumbersome. In the case of Lutok and Kyua, for example, I came up with a populate-builddir.sh script that did a recursive copy of the Kyuafiles from the source directory into the build directory. This script was run as part of make check, and allowed the subsequent kyua test to run within the build directory before the final installation. Needless to say, this hack was awful.

Just yesterday, there was a discussion in the atf-devel mailing list that about this exact topic: how to get Kyua to work with a project that uses build directories. Along the thread, I realized that the obvious solution to this problem was to add support for build directories to Kyua. And this is what just happened!

The code that just landed in HEAD adds support for a new --build-root option.  When specified, the value of this option points to the directory containing the build products if it is different from the source tree.  A common usage example for your Makefile.am files would be:

check-local:
        kyua test --kyuafile='$(top_srcdir)/Kyuafile' \
            --build-root='$(top_builddir)'

And that's it. It had never been easier.

Kyua 0.4 released

Dear readers,

I am very pleased to announce that Kyua 0.4 is available for download!

This new release brings in a lot of new features. The major highlights are:

• Preliminary HTML reports for test suite runs. These are still very rudimentary and lack a lot of detail, but you can start playing with them.
• Better CLI presentation. Long lines now wrap on word boundaries and continuation lines are padded as needed to make reading easier.
• New configuration engine. The parsing of configuration files has been completely reworked internally. While this has no user-visible effects now, it paves the way for further customization in upcoming releases.
• Ability to disable the build of the tests that ship with Kyua. This is only to simplify the bootstrapping of Kyua on arbitrary systems, and to allow an easier creation of packages for a variety of Linux distributions (including Fedora and Debian).

The devel/yua-cli package in pkgsrc has been updated to this new release just now. A package for Fedora 17 is in the review stage, as the possibility of having such a package was blocked on the 0.4 release; expect this to be widely available within the next two/three weeks. A package for Debian should be coming as soon as I clear the prerequisites (ATF and Lutok).

Enjoy this new release!

Welcome to the Kyua blog

Since the creation of the Kyua project, I had been posting Kyua-related news and articles to my personal blog under the kyua label. This has been working OK until now, but the time has come to give such posts a stronger identity by creating a Kyua-specific blog.

So, ladies and gentleman, this post opens a new blog that will cover all-things Kyua! I will be posting release announcements, weekly status reports, details about large refactorings, and maybe even arbitrary articles for user-specific stuff in here, and stop doing so in my personal blog.

I am unsure about what to do with old posts though. I have played a little bit with exporting the contents of The Julipedia and importing the relevant posts here, but the process of cleaning up unwanted posts and comments is tricky. Also, it's hard to know where to draw the line between what feels too personal and what does not, and whether to decide if ATF-specific posts should be imported at all or not. So, for now, I will keep this clean and only post new content. We can always revisit this decision and import older posts at a later stage.