We have had our handbrake ripped off, but to be fair it was on the Old Coach Road on Cape York, and we'd literally dragged the camper over terrain a tad rougher than the usual off road stuff we tow it over.
Our camper trailer is on the heavy side though, so even after having had that happen, there is no way I'd ever remove it. It's peace of mind on sloping sites plus the safety factor when unhitching and hitching up to the car. We use chocks as well, but the handbrake has saved us on one or two occasions.
If it's not a requirement, then it comes down to personal preference.