I can appreciate your points, however, I wouldn't blame the dealer on this one, especially so if it's a smaller operation.
He just got it in, he knows it's a high demand item and will sell right quick like. He gets your call and offers to hold it for you for $500 non-refundable. That locks you in, yes, but what If someone steps through the door an hour later and says, "I'll take it!', he's got to turn down that sale. If you had a refundable deposit, he could lose days of sales opportunity, while waiting for you to pick it up, only to have you bail at the last moment. Now "we" know you'd never do that, but he might not.
When I asked the the Reno dealer (Big Valley Honda) to put my name on an SE they had "supposedly" incoming, they refused to take a deposit - and I asked them several times. They did the same with "TexasJackKin" on his LE, before me. If I'da waffled, they could have sold it same day, in fact he told me they had other buyers in line.