One of the questions I’ve been asked after yesterday’s blog post about Phix’s ContractLib is why not just use PHP’s built-in assert() function? I think it’s a great question, and the best way to answer it is to take a look at the key differences between two solutions.

Side By Side Comparison

Feature assert() ContractLib
Implementation PHP extension written in C (ships as standard part of PHP) PHP library written in PHP
Enable / disable execution Partial (there is an overhead when disabled, but it’s low) Partial (there is an overhead when disabled, but it’s higher)
Issues PHP4-style warning when tests fail Yes (configurable) No (throws a ContractFailedException instead)
Terminate PHP script when tests fail Yes (configurable) Only if the ContractFailedException is never caught
Quiet eval of test expression Yes (configurable) No (not required; test expressions are pure PHP code, not eval() strings)
Callback on failed test Yes (configurable) No (unwinds the stack instead by throwing ContractFailedException)
Throws Exception when tests fail No (but can emulate if you write your own assert() callback method) Yes (standard behaviour)
Tests are pure PHP code No – recommended way is to pass strings into assert() to be eval()’d Yes
Error report includes original value that failed the test No Yes
Support for per-test custom failure messages No Yes – are required to provide one
Support for Old Value storage and recall No (but can emulate by hand) Yes

The Differences Explained

The key difference is one of philosophy. assert() sits well with the PHP4 style of error reporting and handling, whereas ContractLib is firmly in favour of the OO world’s approach to reporting errors.

It’s a personal preference, but I think that PHP4-style errors have no place in code that has any desire to be robust. Exceptions aren’t perfect, don’t get me wrong, but their core property of unwinding the call stack in an orderly fashion makes writing robust code much easier. And they also carry a payload – information about what went wrong and why – which PHP’s assert() cannot provide to the same extent.

It’s much quicker to debug something when there’s a record of the value that failed the test. For that reason alone, I’d always prefer something like ContractLib over the built-in assert() approach.

But we can’t ignore the fact that these are tests that get shipped to, and executed in, the production environment. Unlike unit tests, adopting programming by contract will slow down your PHP code in production. The question is: by how much?

What About The Performance?

I’ve done some benchmarking between the two, using the five tests listed in the final example in yesterday’s blog post. It’s a real-world example of the kind of tests that I would look to add to code to improve robustness.

Here are the results I gathered, calling the tests 10,000 times in a tight loop. The tests were run from the command line, and the times do include PHP start-up / shutdown time and the time taken to parse each test file. I assumed a best-case scenario, where the tests would always pass.

Test Approach Time w/ Tests Disabled Time w/ Tests Enabled
Tests written using assert() 1.103s (100%) 5.989s (543%)
Tests written using ContractLib 3.055s (277%) 3.096s (281%)

When tests are disabled, using assert() is much cheaper than using ContractLib today. That’s to be expected, as assert() is written in C. I imagine that we could get close to the same performance if ContractLib was rewritten in C as a PHP extension.

But, when tests are enabled, assert() is much slower than ContractLib. Why? Because the recommended way to use assert() is to pass the test in as a string. PHP has to convert that string into bytecode to execute, and that conversion appears to be quite expensive.

Given the choice, I’d rather trade things running a little slower in production for having much faster tests when I’m writing code, and that’s why I created ContractLib. Plus I get much better information to understand why the test failed, and if I wanted to run the tests in production, I can handle their failures in a much saner way.

Final Words

In my experience, the time it takes to develop and ship code is normally more critical than how fast the code runs in production. Developer time has become a scarcer resource than CPU time.

Used intelligently, these kinds of tests in your code can help your team deliver quicker, because the code they are using and reusing is more robust first time around. Programming by contract is different to, and complements, unit testing because contract tests catch errors in using the code.

Whether you use ContractLib, assert(), or you create your own solution, you should really consider how much it is costing you when you don’t use these kinds of tests.

Be the first to leave a comment »

This Month

January 2012
« Oct   Feb »