Some feedback on that, coming from first hand experience.
Bayonets do not impact reloading speed. The video you linked provides a wrong picture, since the guy in that is trying to do a speedrun to see how fast he can reload. Regular reloading is not that fast, so bayonets do not affect speed as you did indeed say earlier.
If you still want to make bayonets a penalty, you should, for both realistic and gameplay purposes, have bayonets severely impact accuracy. A musket is generally a very well balanced gun. The bayonet shifts the weight from the middle of the barrel to the front, causing the musket to have the tendency to dip downwards.
There, I have done my duty of annoying Marks with historical details!