griffill,
I owned a 2004 Sienna XLE (vin 266) until just a few days ago. With such any early production unit (vin 266 / Jan 2003) meant that my Sienna ran into just about every problem known to the current generation Sienna, including the steering intermediate shaft problem you're experiencing.
There is a TSB on this very issue (issued 2/6/2004) ST003-04. The TSB outlines the replacment of the steering intermediate shaft to eliminate a squeak, pop or click noise is heard or felt in the steering while making turns. The TSB states that this issue is only covered under for 36 months or 36,000 miles, whichever occurs first.
The best I can say to you is I hope you complained about the noise/problem before the 3 year/36k mile point and have it documented on a service repair order (with no problem found). You might be able to get them to cover the issue (as they should have then) under this TSB.
When I have this steering intermediate shaft replace, it may a WORLD of difference in the steering of my Sienna.
I hope that helps.