I think you are right about the rugby, I would think that the majority of their support is from the county rather than the town,
Its from the area not the county. I know a few Saints fans, and they have told me, that they have buses coming in from all over East Anglia. It makes sense, given there is no other option for them to follow their sport.
One other point. I cant remember where we going, but I was standing waiting for our away coach to turn up at Jimmys End. This was 10am. As we stood there, a coach came past with away rugby fans. I think that it was Cardiff. We aimed to hit the pub at noon, they hit the ground at 10am. What are they doing down there to get people there so early? Five hours before kick off. I think that we have a lot to learn.