depends on what you want to portray.. and where you live there may be restrictions on what you can and cant do.
a full infantryman's kit (without musket) would be ranging from about 400 to 850 roughly, then you will want to upgrade your kit where possible, eg new trousers, you can buy really good ACW trousers for about 180-250.
with a musket, it depends, a pedersoli 3 band rifled enfield that can fire blanks is going to cost up to 2 grand.. but there are much cheaper options, they just arent as good.
You're wrong about the rifle. Pedersoli AND Armisport are good, its the paki-india stuff that is crap however none of sutlers carry it anymore. Its all about pedersoli, armisport or Dixie Arms. My armisport '61 Springfield is excellent, majority of my unit use them, they go for 600 to 1k. Mostly 800. I got mine for 600 with original bayonet as well. and upgrading cheap gear that you got in one week is a huge mistake, a fianical mistake. You will save alot of money by going straight to quality items. Alot.
ChaBoy, where are you based at?
If you are in US, Check these sutlers out and get the general idea of pricing. http://www.wwandcompany.com/
(Best Uniform imo) https://www.njsekela.com/
(little bit of everything) https://www.regtqm.com/
(Good stuff, check out campaigner section. Stay away from uniform though, you can waste your money on better uniform) http://www.authentic-campaigner.com/
Great forum for starters, especially who want to do it right. It has great section for used quality items, check from time to time, strike yourself some great deal. My best deal was getting Wambaugh State Jacket for price of 175 instead of 225. Their articles and discussion are the tits!
Also, they have a wonderful sutler list to help you out as well.
Now if you're in Europe http://www.lederarsenal.com/
Probably the best you can get in Europe beside America.
I wish you luck with your interest and hope you will be on the right path. My first advice to you is "Buy yourself a nice canteen, you will not regret it". Sound lame but it is definately the first thing you wanna buy.
Feel free to PM me with questions.