I've never seen more than five people on at a time on the test server. Most of the players who I see logged in, don't actively play very much or at all. If the loss of less than five players inactive players from the main server is driving down the population, then it's more of a problem with the game than the test server itself.
Besides, the new client is coming out sometime soon. This means that there's not a huge amount of motivation to play right now since they've already hinted at the fact that they're going to put up an alternative fresh server when the new client is completed. In most games you see a decline in players before new content is released. However, things like this are just more noticeable when you have an average server population way below fifty players.
On another note, I'm of the opinion that if people were as friendly towards each other on the main server as they are on the test server, people might just stick around a little longer.
Pennywise - 7 Seconds - Fugazi - Husker Du