Blower Motor Trouble, (not brushes or FSU)

Well, at least I don't think the FSU is the main problem here.

The Car: 96 E36 328is

So I've been having the standard problem of the fan (usually) only running at full speed, I say usually because sometimes I do have control over the fan speed. A long with the fan going away or coming back with a bump at times. Everything I had read pointed towards the fsu, but here is the thing. While investigating, I noticed that a small tap, barely a tap, on the air box would make the motor go on. So assuming it was a poor connection, I ripped everything a part. The connections on the motor seem fine, at first I barely touched the wires where they go into the rubber grommet behind the motor, and it kicked to life. Currently, however, it seems that doesn't do it anymore.

I've tried hammering the motor, to no avail. I accidently made contact between the body and the motor housing...turns out that the motor housing is hot. When that short to ground is held, the motor spins, even with the HVAC system off. Currently the motor seems to run even with the hvac system set to off, though it might just be running slightly longer than usual. ( sometimes it would run a min or two after I shut the hvac controls off ).

Does anyone have any ideas? Is the housing supposed to be hot? This couldn't ALL be from the FSU could it? Doesn't seem like it, especially with the light tap to the air box making the motor spin.

Edit: I've checked and replaced all the fuses, they were all good but I did it anyway.