I was told a while back: if you ride hard pack/fast and with people in back (leave it) everything else (take it out)
That being said. I unhooked mine, it still there I just zip tied it outta the way. Rides great, lets the rear articulate better. And I also ride with the wife n kids 99% of the times.
Lots of post on here, I removed mine and no issues, I have even taken it to Glamis with no sway bar. There should not be any negative affect on your ball joints as the sway bar is only in the rear, and there are no ball joints back there. I drive on the hard pack a lot with no issues, just remember you are not driving a race car so don't try to corner like one.
I've had mine off for 1,200 miles, not one regret or drawback at all. The machine has way more body roll, but I've taken some pretty hard turns at 55+ and she stays on all 4. Sway bars are for sports cars. Not off road vehicles.