I completely agree. Removeing some material where it’s pened would have helped. I likely would have not bent my punch or would have had to use heat. I can say forsure the pin is a interference fit and the pene is so it won’t just fall out if it somehow works loose.If you didn’t remove the peening, that completely explains why it was a bear to remove.
The peening can be removed with a stone or carbide in a dremel to remove the pin. Eay enough to repeen after assembly with a hammer or center punch.
I like chooglig idea to drill and tap opposite side. Then fill it with set screws. That could work from a consumers point.