I've found that one reason I don't mind subtitles, and indeed seek subtitles, even in English-language entertainment, is because I have a problem with being easily distracted. Subtitles give me something to focus on, to pay attention to while all this sound and imagery is going on. I tend to really concentrate when I'm reading, which makes all the other distractions kind of fade out into irrelevancy.
I'm not entirely sure why this is so.
This does make the whole sub vs dub debate kind of tangential for me: assuming equal quality, I don't have much preference over whether the voices are in Japanese or English, but I must have subs, preferably in a language I can understand. The clutter on the screen that I believe is dropped as an anti-sub argument, like some sort of depth charge, is to me the entire point.
On the upside, subtitles have actually allowed me to enjoy a lot more anime than I would otherwise. I tend to react badly when what was supposed to be a cute girl is drawn in a distinctly off-model style, and reading the subtitles really closely is one way I can just get the scene over with.