The amount of canvas we take away with us is based on where (how far away), how long we are going (and staying), and will it be a travelling holiday (setting up everynight).
With most of our trips away being travelling ones, and with the need to keep the weight down, we only take the annex roof (which is folded up with the camper anyway), and one end wall.
We have in the past taken all the annex walls (and used them) on a trip to a caravan park for a week long, set up and stay there holiday. On a week long holiday in one spot, especially in a caravan park, the enclosed annex can help with privacy as much as dry living space in wet weather.
Everyone is prepared to cart extra canvas around, and then bang on about everything having two uses to be worth taking, and it's not like it's light or compact stuff when you've got enough of it.
It's probably more about reducing setup and pack up times, as much as reducing weight.
Almost complete annex setup (minus long side of annex canvas walls)
Simple annex roof and endwall setup.