When taking off the rotor on a car do you have to take the hub lug nut off?

I have a 2003 Expedition 4wd and i have just taken off the brake pads. Usually I thought that the rotors come right off after that? Now these havent been changed in a long time and there is some rust. So do I just have to beat the rotor off or does the hub lug nut have to come off too?