Yarrangobilly is not a village as such... it's just by the Snowy Mountains Highway where it crosses the Yarrangobillly River and is a good place to camp. The road out to Rules Point also has some good spots (but not as good for hot weather swimming opportunities).
Three Mile Dam and Ravine are great places to camp, as is Paddys River Falls. Eucumbene River near Kiandra is good (again, where the SMH crosses the river - if heading for Adaminaby you turn right just before the bridge over the river).
Elliot Way along the Tumut River has HEAPS of spots to camp. There's also places along the road from Cabramurra to Khancoban, but again you need a bit of local knowledge along there to find the places to swim.
Providence Portal and Tantangra Dam both offer camping areas.
I grew up in Cabramurra in the seventies, that part of the world is very special to me