Nah mate

I had the fence battle from hell last year..

From my learnings and court orders from the judge dude ...
this is in QLD.
They have taken the assumed risk on building the additional 25mtrs based solely on your word, this is what I did and it took over a year and court to get my money.. so both parties should have entered into a fencing agreement..SEE BELOW LINK
http://www.justice.qld.gov.au/__data/assets/pdf_file/0006/180483/form-2-ndr.pdfNow I am sure that you are man of your word so they will get the payment for the 25mtrs

In lieu of that document i had to go to great lengths and be very detailed to prove that my neighbours actually agreed to the fence. (15 page court brief) Now I was in similar ..part of the fence was there before they built and the judge clearly stated that he was glad I was only claiming for the portion built since they arrived as i would have had no claim against them for a fence built before they arrived.
So the advice of your solicitor is spot on , how could you have given any authority to construct a fence on land you did not rightfully own.
I believe the court would take a dim view of their actions, my advice ..
1, Right a letter of response.. RE-payment of costs for adjoining fence between xxxxx and xxxx (Property lot numbers) outline your verbal agreement and offer to pay for the costs as you agreed .. be as detailed as possible. Also add
" in respect of previous dividing fence construction between lot xxx and xxx no payment will be made in respect of these works as I/we were not consulted in relation to these works and furthermore held no legal authority to approve such works at the time of construction.
- would love to see how they prove you agreed to it... I had 15 pages and photos and letters .. I think they might struggle considering you did not live there LOL

Pretty low on their part IMHO
Always hard to balance good neighbourly relationships with these sorts of issues..
good luck