[meder24]

This is a commonly discussed problem over on the Lexus forums, so likely the same for Toyota.
There is one remedy that seems to have worked for some, albeit with an older Lexus. See post #8 of this thread Side mirror tilt problem - ClubLexus - Lexus Forum Discussion
It is at least worth a try.
Others seem to be convinced it is a stripped gear somewhere in the mirror assembly. There is no way to verify this though without taking the mirror apart.