It's a memoryleak bug that hasn't been fixed yet. Here's a few workaround solutions: (thanks to Allister for telling us on the official dev board)
1. Save game and reload.
(Fixes the problem only for about an hour and then you are back where you started)
2. Go to www.tweakfiles.com. From there download a little program called
Cacheman, set the memory to "Power User", and it will have the game
a) speed up load times, and
b) put off memory leak problems by nearly doubling the amount of time you can play before unloading data from the cache.
3. Do a full install.
(For some reason, doing a full install does everything solution 2 does but permanently.)
Also, set your desktop to 16 bits color. Some people reported that this worked for them.