[How to] Run PHPUnit tests using database 10x faster
Overview
PHPUnit tests that are heavily using the database can run like 10x faster when you run the database from the RAMDisk.
It's the easiest way I know to improve the speed of the tests execution.
Problem - PHPUnit tests can run long time when they are using the database
Project I'm contributing at my work - License Statistics - uses quite a lot of unit tests.
One of the reasons is that we try to do the Test Driven Development (TDD). But this method of creating the software has some drawbacks.
Right now we have something around 1 000 tests that check around 3 000 assertions. Much of them involve database operations. That's why running all tests on Windows machine can take up to 25 minutes.
Tests on Linux machine are much faster - around 7-10 minutes but we still need to run tests under Windows environment due to fact we have to support both platforms and we have to be sure that both of them work perfectly fine.
Solution - put the Database on a very fast hard disk - RamDisk is the fastest
I was "angry" (to say it nicely) that I have to wait almost half hour to see if my changes did not broke anything before I'll do a commit. Although we use continues integration it's nice to know that everything works fine on your machine. That's why I tried to figure out something that will speed up the tests execution.
It was not hard to see that the CPU has to wait for the HDD due to heavy database operations - database is cleaned after every test and initialized with small amount of needed, basic fixtures. Well maybe not every test, but every test that uses Doctrine somehow.
So the simplest solution would be speeding up the HDD. Especially amount of the the I/O operations per second. There's no way to "overlock" the HDD like CPU so I tried to run the database from the pendrive. This gave me very small speed improvement (maybe because my pendrive is not so fast). But still I had to wait more than 20 minutes for the tests to finish.
Next idea was to "sacrifice" some part of the RAM for the purpose of creating a virtual HDD in that memory - so called RAM DISK.
This solutions is known for years but I didn't see a really good reason till then to use it every day.
Finally I used the Dataram RAMDisk. I created a 0,5 GB RAMDisk and put the database on it. I still have like 3,5 GB of RAM so it's enough for PHP development right now.
Thanks to that the tests are done in around 2 min 30 seconds! That's like 10 times faster comparing to the time that was around 25 minutes when using regular HDD.
I can say that it is worth for sure to sacrifice some part of the RAM to use the RAMDisk.
PS Dataram RAMDisk is available as a freeware. It allows to create a disk up to 4 GB. For $10 you can buy a license that will allow you to create larger disk - up to 32 GB.
Facebook comments:
January 4th, 2011 - 16:46
Good idea. Need to try it.
January 5th, 2011 - 09:28
“it is not a unit test if it touches database” period
what you do is integration testing, even if you use tool with “unit” in name (PHPunit)
apart from that, nice idea
January 6th, 2011 - 17:26
Could you tell me what’s the name of the test I’m performing that are using the database?
According to the wikipedia: “In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application.”
So maybe I got it wrong but the “database” tests I’m running test for example if the query does the things it is supposed to be doing.
There’s no way to test something “smaller” in that case because “deeper” in the code, I have only the Doctrine – which I assume to be working fine.
I did also a small research about your comment and I see it quite often that unit tests can use database.
PS thanks for your comments. I’m glad you liked my idea.
January 7th, 2011 - 08:21
I guess you should look up the phrase “integration test” as well
And yes, Unit Tests may use the Database, but should take care of that sentence: “A unit is the smallest testable part of an application.”
HF dude, Dörte.
January 7th, 2011 - 19:25
It’s not quite integration testing. At least I would not call it like that.
I think doctrine query is the smallest unit we can test in many cases. And that’s what is taking most of the test execution.
We could improve the tests speed by not cleaning the database after every test but that would require other problems to be solved, like how to be sure that tests are still independent.
January 7th, 2011 - 13:21
Hi,
It depends of the software tested, but usually the unit tests requiering the database are only a limited part. Unit test suits should be executed in (micro)seconds, if not, you have an issue.
In your case, I suspect (no offense) that the architecture of the code is not following certain principles that will make your code “testable” (Demeter law, avoiding singletons, avoiding global variables, etc.).
For further information and good talks, I can only advice you to watch this “changing life video” http://www.youtube.com/watch?v=RlfLCWKxHJ0
Misko Hevery has done many talks, you can find other recorded talks on youtube. You can also look for his blog, there is a pdf which is a Google reference document for writting “testable” code.
Enjoy!
January 7th, 2011 - 19:31
Thanks for the link. I didn’t see that one. Although I’ve seen couple of his lectures on the youtube.
I must admin that some parts of the “old” code could be written better way and that’s why we have to test it involving database. But it works so it does not make sense right now to rewrite it just to be able to write “better” tests.
Right now we are creating everything having the tests in our minds so we can test the code without using the database.
January 7th, 2011 - 15:39
Unit Tests can test database interaction those who claim otherwise are wrong. Simple as that. Its not an integration test to test if a function returns correct output given certain inputs.
However you can go overboard and having unittests that takes 30 min to run is a big big nono. UnitTests should go so fast that you can press a button and take a zip of tea and they are done.
You should probably take a look at what exactly your tests are testing and given what fred wrote you should probably give some thought to how you can refactor your code.
I recommend two books:
Test-driven development by example by Kent Beck
Continues Integration Duvall, Matuas and Glover
January 7th, 2011 - 19:39
I’ve already read quite a lot about how to write good and testable code. Right now I’m trying to make use of what I’ve learned and I feel that it’s paying off.
I have loosely coupled objects that are using dependency injection so I can test them much easier without a need of using the database.
But as I said before I still have to run “old” tests from time to time. That’s where this RamDISK really helped me and my colleagues from my job. It saved us a lot of time.
I’ll certainly take a look at the books you proposed. Thanks