Do you reset your router a lot? If so that's an issue.
Do you use a wired or Wireless connection? If it's wireless try investing in Powerline adapters
Is there anyone else using your internet when you play? If so kick them off.
Have you contacted your ISP? Generally they should be your first port of call with this stuff, as often they can fix this stuff quickly, or at the very least inform you of the issue if there is one.
Finally, it could simply be that your unlucky and it's a server side issue.