The prince was in Lesotho to see the charity's work a few months ago but now he says he has no choice but to part ways.