Interesting discussion.
Here are my further thoughts (layman of course).
I'll use reading and listening as two examples of the point I'm trying to make.
Reading
When someone reads in their native language, it is not necessary to read the entire word or phrase. The brain naturally "fills in the blanks", due to the countless years of practice the brain has had in processing and decoding the language and various letter/word combinations. Pretty wondrous actually.
Listening
I believe the same to be true of the brain's ability to decipher the spoken language. The brain's experience with a listener's native language allows for the same "leap" in processing what it hears. Phrases get completed and words are inserted "automatically" based on the knowledge and experience acquired through years of exposure with the language.
When I listen to AmE being spoken, my processing is almost effortless. I expend very little energy, regardless of the speaker's accent. The same could not said when I listen to BrE. I treat this as a different language (hahaha).
Word usage, sentence construction, common phrases can be radically different. It is not just about accent. I will obviously "comprehend" what a British interlocuter is saying (the better educated, the easier it becomes), but the effort expended is significantly increased.
Aside: The amount of "active" listening that I had to perform during my trip to PL last summer was exhausting. I would fall into bed at the end of the day drained from the effort of processing what I heard during the day. LOL.
Thoughts welcome.
:)