I had this issue as well.
I found that if I wore a t shirt, it was an issue.
If I wore a shirt with a proper collar, the issue was less pronounced.
So dress up a little, and the problem goes away...that is what BMW was going for - prim and proper drivers.