If you take out the Federal Reserve, there is much less pressure for the citizens to invest in the market just to maintain the wealth they have earned. The Federal Reserve also drives the market at this point.

If you get rid of IRS/Taxation then alternative currencies can fairly compete with the Dollar in the US economy and this will have the effect of neutralizing the Federal Reserve (and by extension Wall Street)