Try the other way around and you'll need to put stress on the spindle end. Something is not aligned, I think it's the ears on the trailing arm itself.

I did not need to grind the spacers.