I had no issue removing them...pull straight out with needle-nose plyers. My issue is how to push those front plastic screws back in?
They are expanding rivets - to remove, you pull only the "cap" of them out. When you do that, the "body" of the rivet contracts and lets you pull it out.
To replace, you put the "body" in (should be easy to squeeze in) and then push the the cap&stem back into the body and it expands, holding it in place.
Sounds like you may have pulled out the whole thing without removing the cap/stem first?