Hello Crozade,
First off may I ask what your average connection speed is? (Speedtest.net is a good way to find out.) Generally ping varies depending on your Upload / Download speed as well as your distance from the server, thats if you know where the server is located.
Another thing would be other players, do they expirence these ping changes? If so I'd guess the server itself was the cause of the problem. I do doubt your firewall would be affecting the connection but I'd also recommend checking that also as there is a slight chance it could be affecting it slightly.
From what you described it seems like a client problem unless you say everyone else expirences the same as this, if so I'd recommend getting the server owner to contact his provider. But my advice would be try a few things such as stopping downloads while playing or making sure your firewall isn't blocking the game. Generally I'd expect it to be your distance from the server and connection or the server itself.
Hope the information I provided was of use, feel free to PM me or reply here again if you still have no improvement.
Regards,
Felix