August 24, 2005
Nonfunctional Functional Tests, and iGet 2.0a691
Somewhat recently, Apple added unit testing to Xcode. That's really cool, and it is especially cool (for me) that they used OCUnit to do it (for Objective-C), because that's the unit testing framework we already used here at the Five, so adapting was hella easy, as they don't say out on the East coast.
A little less easy, maybe, was converting our functional tests. In the old days, Five Speed used OCUnit for unit tests, and an internally developed, hacked-together framework for functional tests. (I know, I know, it's not hip to call them "functional tests" anymore.)
Unlike unit tests, which exercise a small and discrete area of the software, functional tests are more complex and involve multiple parts of the application--and in iGet's case are very often both multithreaded and dependent on a lot of network traffic, and sometimes take a long time to finish running. Hence the separate framework.
(I promise, this is gonna lead up to how we ended up releasing iGet 2.0a691 last night...)
It always bothered me that there was so much overlap between the unit testing system and the functional testing system, and yet they were totally different. Both frameworks needed to provide a system for tests to be compiled, linked against the product to be tested, run, and have their results reported to the developer (or tester). So I thought, "What the Frankenstein, now that OCUnit has official Apple sanction, can't we combine the two?"
But the two types of test are significantly different.
An iGet unit test might go something like this:
- save a shortcut to a file on disk
- make new shortcut by reading that file back from disk
- compare shortcut objects 1 and 2 by sending -isEqual: and assert that the result is YES
...whereas an iGet functional test might go something like this:
- connect to a remote testing server over the Internet via a real-world connection
- upload a known set of files and folders, including valid and invalid symlinks, aliases, and files with weird permissions
- wait for the upload to complete (which might take a long time)
- modify three of the local files which were uploaded by changing their contents
- initiate an iGet download of the files just uploaded back into the local folder
- assert that the expected download conflicts occur with those three changed files
Let's ignore the objections of purists who might point out that this test is very convoluted and tests many different things at the same time (that's obviously true). But the unit test above executes in something like 0.02 seconds, and the functional test takes ten minutes. Not only that, but if we want to use iGet without changing it, which is probably a good idea if we're trying to test it, then the functional test also requires a new thread be detached to do network I/O, and then a bunch of interthread communication to happen between the network and GUI threads.Well, it turns out that is really pretty easy, from within a test:
// theTask is the network task we want to wait for, e.g., an upload
NSDate *deadline = [NSDate dateWithTimeIntervalSinceNow:120.0];
do {
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:1.0]];
} while ([deadline timeIntervalSinceNow] > 0 && [theTask isRunning]);
That's the basic idea, and this code let's me watch iGet pop up it's windows and progress bars and dialogs while it runs these lengthy tests (or, more likely, go grab some dinner). All the normal ways of working with unit tests apply, since the functional tests inherit from SenTestCase, so they are unit tests. (At least, they are unit tests as much as those disgusting bloated GMO farm-raised salmon are salmon.)
So far, so good. Over the course of a day an a half I field-stripped and disassembled my functional test framework, removed it, and started smooshing my round functional pegs into Xcode's square unit holes. Uhm, converting the functional test cases to work under Xcode/OCUnit, I mean. Anyway, it worked great! That part only took a few hours, and there are a lot of tests.
But wait, can't I go even better? I asked myself. I'm sick of waiting bleems for these functional tests. The near-instant unit tests are so much nicer. Why don't I make all these test files a lot smaller? It will be pretty much the same, right? I mean, what's a few megs between pals?
I did that, and shaved forty minutes of the functional tests run. Great!
The only thing was, in so doing, I missed a freaky-deaky bug in the iGet test release we shipped last week. (Ironically, it was a bug that illuminates why I strongly believe in doing complicated functional tests as well as lots of unit tests.) The executive summary is that on real-world Internet connection (not fast LAN) transferring a file over 50 MB or so would excercise an unknown bug in our new throughput-calculation code. This would lead to the calculation taking progressively longer and longer until it started to consume enough CPU to slow the file transfer down to a crawl.
The upshot is that we shipped iGet 2.0a691 to the test group last night, which fixes the bug, and I have restored the functional test suite to using huge files, so that we won't miss something like this again.
Posted by mason at 12:34 AM | Comments (0)
August 22, 2005
The path of least resistance
Yesterday saw the private test release of iGet 2.0a687. This is more or less analogous to "beta 2" and I think we won't have so many more test versions before we have a stable release.
The big change from the previous test release is Automator support, and some (pretty basic) AppleScript support. These features were not phased into our previous test releases because we needed time to test the various changes we had to make to support adding these features. The last major feature to be added will be the (overhauled) Dash Board, oops I mean Dashboard widget, which is largely separate from the app itself.
As always, as we come towards the end of a release cycle, I can't help but start thinking about the next release. One thing that these cool new features make me want to do is overhaul iGet's notion of a path.
Normally, a path is something like this:
/Applications/iGet.app
/Volumes/MyOtherHardDisk/Cool Programs/iGet.app
In the old Mac days, it was (and still in some legacy apps) like this:
Macintosh HD:Applications:iGet.app
MyOtherHardDisk:Cool Programs:iGet.app
But, in iGet, we made the choice to do them like this:
/Macintosh HD/Applications/iGet.app
/MyOtherHardDisk/Cool Programs/iGet.app
At the time, it made sense. Or so we (OK, I) thought. In iGet 1.x, the user never dealt with paths directly; they used the GUI. Showing the name of the hard disk at the start of the path seemed more intuitive (indeed, that's why the Mac did it that way originally). But with iGet 2's new features, you might want to manually specify paths in an AppleScript or Automator action for transferring files. Or, you might want to use the new URL support, and this:
iget://mason.fivespeed.com/Users/mason/whatever
...is a lot more sensible than this:
iget://mason.fivespeed.com/Macintosh%20HD/Users/mason/whatever
...especially for users already accustomed to working with paths and URLs. (As it stands, in iGet 2.0 you need to use the latter.)
I'd hoped to address this for 2.0, but we're out of time. It's not a huge change, but too big for the 2.0 time frame. But for 2.x I'd like to make iGet's notion of a path mesh with the POSIX standard. iGet already uses some "smart" path interpretation in some places; if you "Go to Folder" and enter "/etc/" it will figure out that you meant a POSIX path, not an iGet path.
But iGet paths were a design mistake that I look forward to fixing.
[Living In Love by I Wayne]
Posted by mason at 07:09 AM | Comments (3) | TrackBack
August 17, 2005
Welcome to my blog. Now go home.
Hi! Lucky for you and the rest of the world, I am here with JUST WHAT THE WORLD NEEDS: another weblog that catalogs the rambling thoughts and pet peeves of some dude you've never heard of. All f*&#%g right!
So, welcome. Word came down from on high that all cool small software companies (not to mention lame and big ones) now have developer blogs, and it would be untoward if Five Speed didn't also.
Well, now we do. Isn't that so great.
Posted by mason at 01:21 AM | Comments (0)