Going by entirely personal experience here, but in 5 years of living through Newfoundland winters (with winds like hell) I only wore scarves about half the time. My face was freezing, of course, but that's to be expected.

I second the long underwear, though.