From what I've seen, the only thing that really, really works is a perm ban with a semi-generous appeal system. The people who really didn't mean to teamkill are more likely to be the ones appealing to be unbanned.
Auto-ban works for this, too. If someone keeps teamkilling, regardless of if they meant to or not, should probably not be playing the game at that time anyway, since it begins to impair the fun for others.
The most important part is to just make sure people are aware of the ban appeals. If someone is TKing to troll they probably don't care that much about being banned, they'll just brush it off and quit playing. The ones who do appeal you can just keep track of and stop unbanning.