Hi, I did one of those 12 pack detailing jobs this weekend in lieu of the std. 6 pack wash and wax only. Engine, interior, etc detailing

I removed the wheel flares on my SR5, you know the inexpensive rubber with chrome strip kind and discovered they will hold a tremedous amount of dirt and mud inside them. The truck is a 2004 and sees regular off road use here in Colorado. Luckily there was no rust what so ever.
I left them off and was wondering if others had encountered the same accumulation issue under them and removed them too and what negative affects their removal will have on the side body of the truck? I do not have tires wider than stock so I don't think too many rocks will be thrown up against the paint. Stock flaps are still in place.
Thanks