Dont show them your fear. They dont know that they are scary. Put few cheap militias and most likely they will think you ar strong and wont dare to atack.

I am pretty sure they are less likely to atack if you have garisoned units. It does not evaluate what units it only counts how many. So you can have a lots of rubish and it will think  that you are risky to atack (it still can atack, but chances are quite low).

Othervise you can challenge him you might need few (3 -4) units equiped with high intiative weapons (if you have those first knifes researched with some intiative they will do) and stack dodge traits. It depends what race are you (if wright then its cool to fit with dodge). I made 1 such golem with army of 3 early units, but my hero was assasin (he died, but he dodged like 7 turns before dieng) and i won battle with cassualities... but i won... and my ppl were safe and happy again.

P.S. Your scout tactic by the way is also decent: its cheap, safe and nobody has to die (unless scout >) It buys you a time to gather a force to deal with issue.

The scout tactic is something I've employed often, especially when a random dragon has decided to wander into my territory and threaten my cities when I still haven't even discovered iron weapons yet. I must admit, it's kind of annoying having to spend the next 200 turns moving a scout around distracting and baiting the dragon, but it beats the alternative of all my cities getting razed because there is no possible way to defend them.