I'll explain how to have the gameplay you're hoping for, as well as realism.
First: You must consider the real reasons why people used line formations. These were manifold, but summarized by few main reasons.
Line formations are easier to manage large units of men with. Instead of them wondering what to do or where to go, they simply retain their position relative to everyone else, and fire on orders.
Skirmish lines require more initiative, and thus more training and discipline to maintain effectiveness, as men must actively seek cover, and fire or reload at will, and retreat or advance at will. A skirmish unit is easy to lose track of from the commander's viewpoint, so it is limited in its size by what can be micromanaged, or how well the skirmishers understand what they must do.
Lines offer strength in numbers, and are a force equaled in sheer strength, only by a line of the same size.
The line can be easily managed by officers, and offers safety in a melee against anyone who would come for you while you're reloading.
The fact that having as many friends as possible within weapons reach of you in a melee, is a good thing, is a very easy concept to understand.
The purpose of a line is essentially the ability to deliver an effective mass melee charge, to force your enemy to retreat. If battles were decided entirely by who kills the most of the other guy from a distance, you'd get something resembling modern combat, which involves cover, suppression, and firepower as key elements on the tactical level. This is what light infantry did even in the Napoleonic wars.
Secondly: The reason for the above being the case is a question of the effectiveness of ranged weapons.
Why in the hell were massed line formations still in use after 1850 if the rifle was as accurate then as it is now?
It is because the rate of fire was substantially different. It takes 20-30 seconds for a soldier to go from shot to shot. No, I'm not kidding, it's a slow and tedious process.
Open cartridge pouch, grab cartridge, open cartridge, place powder and bullet in barrel, remove ramrod, ram cartridge, replace ramrod, half cock lock, grab percussion cap, place cap on nipple, cock lock, aim/fire.
Because this took so damn long, it made bayonet charges possible. Why? Because any man can run a hundred meters/yards in the time it takes to reload.
For about every 100 meters of ground your enemy crosses, you can fire once, or twice if he's not moving fast and you're reloading fast.
This means that depending on the openness of the terrain, an entire line could move into a melee against an enemy as they reload.
In addition, shot does not mean dead. Simply because you've been shot, you do not die. A person dies near instantly if hit in the brain, bleeds out in seconds to a minute if a heart/lung/major blood vessel is hit, and it takes several minutes or even hours to bleed out if hit in a nonvital area.
I suggest that the game reflect these things, where reload speed is incredibly important tactically, and where wound damage is heavily locationally based. In this way, you can ensure that the game is both realistic, and that melee combat is feasible.
The purpose of a line formation is to keep your men organized until or if possible during a melee fight, not to absorb or be effective in preventing damage from ranged fire. That's all it's ever been for. Formations are stronger in a melee than infantry out of formation.
