I grew up on dirt roads, so I generally drive a bit faster than others on dirt, and when people don't slow down or worse, deliberately sit in the middle of the road to stop me from overtaking, this is very frustrating, and generally results in a stone shower when I eventually get past. If they want to go 70kph everywhere, and I am comfortable doing 90kph, I don't see why they want to hold me up.
When other vehicles are approaching, I will generally back off to reduce dust and stones, and I appreciate it when others do the same.
I am a firm believer that you should drive to your abilities / comfort level on dirt, and I am very comfortable with dirt so will travel a bit faster, however my wife is hopeless on dirt and will travel quite slowly, if she will even drive dirt at all.
If someone wants to go past, let them, there can be very little benefit to you if you deliberately hold them back.