How about updating your profile, or better yet posting what year car we're talking about?
If it's a MKIII, feel along the back side and you'll feel two holes on each side of the mirror's arm. Stick a finger in each one and pull forward on the housing while also pulling down... usually it's a royal bitch and you think you're going to break it. It's a bit easier on a hot day or you could try warming it with a hair dryer first.