My golfing buddies, 2 Bama fans, 1 Tennesse fan, and 1 Kentucky fan, ALWAYS let smaller or faster groups play through. It's the rules of golf.
If ever in the situation you were in, the asshole of our group (the UK fan) will drop a few hints to let them know we are waiting on them. After two holes of waiting, he ALWAYS hits into them. Out of the groups we've hit, only once did I think we were about to fight. The group of 4 didn't have a collared shirt on them. Two didn't have sleeves.
Of course, on a scale of 10, the course was a 4. Drunk and fighting is probably more common than birdies and pars.