There is a little expansion pin in the hole that needs to be either pushed through or pulled out. If I remember correctly, once its out, you can turn the latch and pull it off. You may need to apply some pressure to the outside piece (think of it like a nut and its bolt). When I got my car, I replaced mine as they were old and nasty. The hole was so filled with grime I had to WD-40 out the grime to see in there, then push the pin one way, then when some was exposed, pull it out with pliers. They're almost as much fun to put back, as you need to compress the pin to get it back in the hole.