Passenger Window problem

OK, 2008 Saturn XR, Passenger window (power windows) stops working intermittently; either up or down, and will not respond to driver's switch or passenger switch. I can hear the switch clicking, but no movement. Then a day or two later it starts working again just fine. The other day I pulled the inner panel off, pulled the switch box apart, and the contacts look fine. I'm thinking window motor, but I really don't want to go there. How hard is it to replace the window motor? Thanks.