Staff at Chamberlayne Leisure Centre on Weston Lane "have faced abuse" after a group of around 20 caravans turned up in Southampton.