Status report: 2012-12-31 - 2013-01-06

• Fixed the performance regression in kyua report introduced in the testers branch and, actually, made it run faster than ever.
• The generation of a report for the Kyua test suite itself used to take about 12 seconds. However, the schema changes that went in with the generalization of the test interfaces pushed the run time to over 2 minutes. Now, with the addition of a missing primary key (duh!) and a couple of indexes, the same report generation takes less than 2 seconds.
• Wrote the migration code to upgrade an old-schema database to the new generalized schema. I have not submitted this yet as I'm busy writing tests to ensure the code works fine, and making them general enough so that introducing a new schema change in the future is easier to test than today.

Status report: 2012-12-24 - 2012-12-30

Good news everyone!

• Initial kyua-testers code published. It took a long time, but it has finally happened.
• After this announcement, continued the work to clean up the new testers branch in the main kyua-cli repository. At this point, the latter has lost 5K lines of code and got another 5K lines replaced with more generic and simpler abstractions. There are more changes to come that will make things leaner, but these are starting to become tricky to implement.

Have a happy beginning of 2013 :-)

Initial kyua-testers code published

Three months, a hurricane and an apartment move after the proposal to modularize testers as independent binaries, I am pleased to announce the publication of the initial code implementing such proposal! It has been a long way full of interrupts, but this marks a major milestone in the development of Kyua.

The code for the independent testers can be found in the new kyua-testers module, which weights at ~10K lines of C code (yep, no C++) and currently provides testers for ATF-based and old-style (framework-less) test programs. As the documentation describes, the goal of these programs is to provide a scriptable and generic interface to run diverse test programs. This provides the ability to: first, plug new testers dynamically into kyua(1)(e.g. imagine a new tester for googletest-based test programs); and, second, to simplify the code of the frontend by removing tricky low-level code so that, potentially, most of kyua(1) could be latter rewritten in Lua.

The changes to make kyua-cli use kyua-testers are now in a separate branch of kyua-cli called testers, which drops ~4K lines of C++ code (much more slaying to come). While these changes already work and do not appear to introduce functional regressions, there are still two major issues to fix:

1. The first pending item is to address a serious performance regression in kyua report. This regression is understandable considering how horrendous the database querying code is and the generalization of the database schema that went in with this refactoring. Conceptually easy to fix, but the database-backing code needs some serious cleanup.
2. The second pending item is to improve the code in kyua-cli to execute the testers, which in turn will obsolete —and thus enable the removal of— lots of nasty and unnecessary abstractions. The current implementation to do this has been a hack of mine in an attempt to get things to work as quickly and as unobtrusively as possible. Now that the code is in place and that all the tests pass, it will be easier to reorganize the code.

If you inspect the code, you will realize that the engine/atf_iface/ and engine/plain_iface/ directories of kyua-cli are now gone. This is a big win because the abstractions in kyua-cli (those representing test programs, test cases and test results) are now really unaware of the specific interface implemented by the test programs being executed.

As soon as the two above items are addressed, I will start evaluating the merge of the testers branch into master, which should unblock the integration of Kyua into NetBSD! Stay tuned.

Status report: 2012-12-17 - 2012-12-23

Long time since the last report. Have been busy sorting things out post-Sandy and with a work trip... But, finally, I have been able to put some hours of work into Kyua these last few days:

• Integrated the independent testers into Kyua, and got all tests to pass. The work of the last two months if finally seeing its results!
• Started the removal of all code in the kyua binary that implemented the execution of tests in-process. There is still quite a few code that needs yanking though.
• Many cleanups to the testers code in preparation for publication: fixing pending to-dos and adding tests.

Stay tuned for the shortcoming announcement about the publication of the testers and a branch with the code to use them. Once this happens, I'll still have to address a serious performance regression in kyua report before this can hit mainline.

Status report: 2012-11-19 - 2012-11-25

• Got kyua list to work with the independent testers and pass all tests!
• Started cleaning up the testers code so that it can be published.
• Initial work to get kyua test to use the testers too. Most tests pass, but there are still some rough corners (obviously, as the implementation is still full of to-do entries).

Status report: 2012-11-05 - 2012-11-11

• Completed preliminary implementation of the ATF tester.
• Wrote (very) basic integration tests for the plain and ATF testers.
• Changed Kyua to use the external testers for test case listings. Working for the most part, but still some rough edges left that will probably require a bunch of work.

The code for the testers is starting to be pretty decent, so I should be publishing these as a new module in the Git repository soon. Keeping so much unsubmitted code in my machine is not a smart move, as Hurricane Sandy showed recently... Stay tuned!

Status report: 2012-10-29 - 2012-11-04

This last week has been eclipsed by hurricane Sandy. Even though I could not go to work, I haven't been able to put much time in Kyua either. That said:

• Implemented the parsing of ATF test case lists in the ATF tester.
• Implemented the parsing of ATF results in the ATF tester.
• Tweaked my Emacs configuration files to highlight test errors in the compilation buffer. Jumping to test failures right from there is cool. (My idea of providing a "kyua-mode" may not be so difficult after all...)

The independent testers are now to a point where integration with the main Kyua command line can start — yes, they are still woefully incompletely, but seeing the results of this last month of work would be cool. The problem is... I just found out that my laptop does not have a copy of that work: all the experimental code is sitting at my still-powerless home in my desktop computer... gotta grab it soon to recover the files!