My 2002 Highlander has the exact same issue!! and our local dealer said the same thing, "it's out of warranty so it will cost $667".
With this many people having the same issue, and a safety issue at that there is definetly a design defect in the visor design.
I followed Olive's link (2 or 3 posts before thisone) and reported it to NHTSA.
I also reported to Toyota at
http://toyota.custhelp.com/cgi-bin/t..._intercept.php
In your language to both NHTSA and Toyota refer to the problem as a "safety issue" and a "design defect"