Status report: 2012-06-18 - 2012-06-24

• Issue 34: Added the contents of the stdout and stderr of test cases to the HTML reports.
• Ongoing work to add automatic stacktrace gathering of crashed test cases.
• Split the libraries out of the atf package and placed them in a new atf-libs package in pkgsrc, and created a package for kyua-atf-compat. Because pkgsrc is currently undergoing a freeze in preparation for a new stable branch, these changes have not been submitted yet.

Tests packaged for Fedora

The initial versions of the atf, lutok and kyua-cli packages that landed in Fedora did not install the bundled test suites. This was done to simplify the initial addition of the packages to the distribution, as using /usr/tests right away for the installation of the tests would have probably been a package review blocker.

With the original packages already in place, I have now changed them to supply an additional -tests package that installs the test suite. Because the use of /usr/tests has not yet been discussed in detail (and obviously not approved yet), I have opted to go for a compromise in the meantime. These test packages install the test suites under /usr/libexec/<package>/tests and they include a README.Fedora file explaining why the tests are in such location and how to run them.

No matter what, running the test suite of any of these packages is actually pretty simple. For example:

# yum install kyua-cli kyua-tests
$ kyua test -k /usr/libexec/kyua-cli/tests/Kyuafile

Note that, at the moment, these new -tests packages are only available in the updates-testing repository of Fedora 17 and will not move onto updates for over a week.

Status report: 2012-06-11 - 2012-06-17

• Changed ATF's build system to not build the end-user tools (atf-run and atf-report) by default. The build of these can still be explicitly requested by passing --enable-tools to the configure script.
• Wrote preliminary compatibility wrappers for atf-run and atf-report, accompanying tests and manual pages. These have all gone live in a new kyua-atf-compat module.
• Pushed the kyua-cli package to the stable updates repository of Fedora 17. Just type yum install kyua-cli on your Fedora 17 (or rawhide) system and start enjoying Kyua!

New Kyua-based versions of atf-run and atf-report

The plan for the upcoming Kyua 0.5 is to have full feature parity with the obsolete atf-report so that we can finally say goodbye to both atf-run and atf-report. The fact that these tools still exist and are in widespread use prevents us from adding much-needed features to Kyua (for compatibility reasons). It is therefore important to get rid of them as early as possible. We are almost there, except for a few details that need to be addressed in the report generation.

As part of this plan, ATF 0.16 will not build the end-user tools by default: they will still be there, but only built whenever the user asks configure to do so. Users will instead be directed to use Kyua itself.

But, of course, making binaries unreachable by the user is never nice. And this is why, in order to simplify the transition from the ATF tools to Kyua, I have written a bunch of simple scripts that reimplement the major features of atf-run and atf-report on top of Kyua. They don't behave in exactly the same manner, but they are close enough to support the most common command line invocations: in particular, a bunch of our users have atf-run | atf-report ingrained in their minds, so it is important to keep this basic use case working to mitigate their frustration (if any).

With all that said, it is time to say hello to the new kyua-atf-compat module! This new package includes the following tools:

• atf2kyua(1): A tool to convert existing Atffiles to Kyuafiles. This was previously part of the kyua-cli package, had the name kyuaify.sh and was shipped as a miscellaneous script. It has now been promoted to a first-class tool, with a manual page and functionality tests.
• atf-run(1): Reimplementation of ATF's runtime engine on top of kyua test.
• atf-report(1): Reimplementation of ATF's reporting engine on top of kyua report.

Testers welcome, of course. A formal release with its corresponding binary packages will follow soon (before Kyua 0.5 and ATF 0.16) so that we can get some mileage on this new code.

Status report: 2012-06-04 - 2012-06-10

Now that Kyua has its own blog, the weekly status reports are back as promised! I'm glad to see that this last week has been quite fruitful:

• Released Kyua 0.4.
• Implemented support for "build directories".
• Got a kyua-cli package into Fedora 17's updates-testing repository.
• Thoughts on what the upcoming Kyua 0.5 release should focus on. The goal is for this release to include all the functionality needed to replace both atf-run and atf-report, and to provide drop-in replacements for these obsolete tools.
• Wrote some tentative code to add the contents of a test case's stdout and stderr to the HTML reports. The implementation of this feature is pretty simple, but I still need to write the corresponding tests before submission (not that easy).

Fedora 17 package available for testing

This last week, I have been working on getting a new kyua-cli package into the Fedora repositories. The obvious goal is to reach more users by providing an easy mechanism for Fedora consumers to install Kyua. Just now, the package has been pushed to the updates-testing repository, which means it is available for testing before it gets moved to the regular updates repository.

So, go ahead, issue yum install kyua-cli (or use any graphical frontend) and start playing with Kyua! Do not hesitate to report any problems that you encounter, or to give karma if the package works fine for you. You may need to enable the updates-testing repository first though.

One thing to note is that none of the packages that are Kyua-related (atf, kyua-cli and lutok) install their test suite as they should be doing (yet). This is because the ideal place for such tests would be under /usr/tests/ as we do in NetBSD, but this directory is not allowed by the Linux Standard Base guidelines. While I think what the best way to resolve this is, I'm going to modify the packages to place the tests elsewhere in the meantime (most likely somewhere under /usr/libexec/)... so expect some -tests packages soon!

Support for build directories

Build directories are a mechanism for keeping the source tree and its build products separate: during a build, the source tree remains read only and all the results of the build are placed in a different directory. It is often considered good practice to use build directories, as this ensures that the build does not have any effects on the sources and permits creating different binaries (think about "debug" vs. "production" binaries) from the same source tree without an intermediate make clean.

Most, if not all, modern build systems support the concept of build directories. The GNU Automake/Autoconf combination is not a exception, although I believe that not many people are aware of this. However, those that are probably use this feature routinely.

Up until now, integrating Kyua into a project that used build directories had always been cumbersome. In the case of Lutok and Kyua, for example, I came up with a populate-builddir.sh script that did a recursive copy of the Kyuafiles from the source directory into the build directory. This script was run as part of make check, and allowed the subsequent kyua test to run within the build directory before the final installation. Needless to say, this hack was awful.

Just yesterday, there was a discussion in the atf-devel mailing list that about this exact topic: how to get Kyua to work with a project that uses build directories. Along the thread, I realized that the obvious solution to this problem was to add support for build directories to Kyua. And this is what just happened!

The code that just landed in HEAD adds support for a new --build-root option.  When specified, the value of this option points to the directory containing the build products if it is different from the source tree.  A common usage example for your Makefile.am files would be:

check-local:
        kyua test --kyuafile='$(top_srcdir)/Kyuafile' \
            --build-root='$(top_builddir)'

And that's it. It had never been easier.

Kyua 0.4 released

Dear readers,

I am very pleased to announce that Kyua 0.4 is available for download!

This new release brings in a lot of new features. The major highlights are:

• Preliminary HTML reports for test suite runs. These are still very rudimentary and lack a lot of detail, but you can start playing with them.
• Better CLI presentation. Long lines now wrap on word boundaries and continuation lines are padded as needed to make reading easier.
• New configuration engine. The parsing of configuration files has been completely reworked internally. While this has no user-visible effects now, it paves the way for further customization in upcoming releases.
• Ability to disable the build of the tests that ship with Kyua. This is only to simplify the bootstrapping of Kyua on arbitrary systems, and to allow an easier creation of packages for a variety of Linux distributions (including Fedora and Debian).

The devel/yua-cli package in pkgsrc has been updated to this new release just now. A package for Fedora 17 is in the review stage, as the possibility of having such a package was blocked on the 0.4 release; expect this to be widely available within the next two/three weeks. A package for Debian should be coming as soon as I clear the prerequisites (ATF and Lutok).

Enjoy this new release!

Welcome to the Kyua blog

Since the creation of the Kyua project, I had been posting Kyua-related news and articles to my personal blog under the kyua label. This has been working OK until now, but the time has come to give such posts a stronger identity by creating a Kyua-specific blog.

So, ladies and gentleman, this post opens a new blog that will cover all-things Kyua! I will be posting release announcements, weekly status reports, details about large refactorings, and maybe even arbitrary articles for user-specific stuff in here, and stop doing so in my personal blog.

I am unsure about what to do with old posts though. I have played a little bit with exporting the contents of The Julipedia and importing the relevant posts here, but the process of cleaning up unwanted posts and comments is tricky. Also, it's hard to know where to draw the line between what feels too personal and what does not, and whether to decide if ATF-specific posts should be imported at all or not. So, for now, I will keep this clean and only post new content. We can always revisit this decision and import older posts at a later stage.