OK, so it's been a couple of days (and Christmas of course) but I've ended up taking the whole thing off again. Indeed the coil was interfering with the arm:
At first I tried the prybar method, but that did not work.
I changed the top ball joint nuts one at a time with long studs:
Also, I put a jack under the arm to "catch" the arm. I started decompressing the coil very slowly so I could safely take it off.
Another angle:
I recompressed the springs at my local shop and wrapped them in wire. Removed the compressor and put the spring (in compressed state) back in the arms, but this time I've put the upper arm in the upmost position first. After that, I've put my jack under the arm again and started jacking the lower arm up. Slowly but surely, I could reconnect the upper arm again. With the regular bolts this time, not the long studs.
Long story short, my car is now finally on it's wheels again. Properly this time!
Oh and I didn't forget the rubber by the way, I've put it on before reassembly.
![Very Happy :D](./images/smilies/icon_biggrin.gif)