So, the feed to the barn is 3 wires. 2 hots and a ground. (A) hot going to one phase, (B) to another, and the ground to the ground bar. 120v neutrals and grounds return to the same bar.
View attachment 441264
You don’t say the age of the barn in your comments , but this looks like an old school 3 wire 220 setup. In my area it would be commonly seen, and correct for the date installed, but not considered proper for new installations these days. Having a main breaker on the sub and another breaker in the main panel is only bringing redundant breakers, and not required in my area on small subs like this one.
In my view this isn’t likely where your problem is at. I also wouldn’t replace the wiring either. But…I’m not familiar with code in your area either.
My money is on that you have a box or something in this circuit past the panel (outlet, switch, etc ) that has burned, corroded, wet or otherwise poor connections causing the issue you are seeing and bringing the stay voltage to the ground bar ( via the small gauge neutrals ) in this box and then back to the house panel via the ground wire.
I’d start by unhooking 1 by 1 each breaker circuit in this box (hot, neutral and ground) and checking back at your house. Leave them unhooked until you find the change. Maybe you already did this and I miss read the posts in this thread.
If you unhook all the circuits in this sub panel and the issue remains, then it’s on the supply to the sub panel or the sub panel itself. The final last check would be to unhook the 2 hots and the ground for the final last check…
A ground rod is needed but it’s not solving the problem…or the source in my view.