I did some research and it indeed looks like you're forced to accept him, so I went ahead and did it.
Guess I'll just ignore him and hope he'll leave...
On the bright side, I can now sell plots of land and visit mystery islands with possible villagers on them.
So now it's time to waste some...