Well, in a fantasy world, it's hard to establish what is silly and what isn't. You can argue that fireball can be as silly as unarmed combat.
Fantasy genre is a hodge-podge of various mythologies - the taste usually varies greatly.
But when you look at those mythologies, you will find a lot of "silly" elements. In Mahabharata, the five brothers often used uprooted trees as weapons - how silly is that? Herakles often fought unarmed. Ilia Muromec from Russian "Byliny" was so strong that "each strike of his mace opened a wide street in enemy army", with soldiers flying everywhere. I have even read Mongol myth when a hero grabs his adversary by his waist, rises him above head, and smashes him against the ground so strongly that he ends up buried, with only head sticking out of the ground. A Korean hero, upon seeing enemy navy, jumped to the sea from a cliff, swam eight miles under water to approach undetected, and then emerged so forcefully that he smashed the enemy admiral's flagship to pieces, sinking it, and forcing the enemy ships to retreat. A Chinese hero Cao-cao allegedly fought bandits and killed ten thousands of them singlehandedly, armed only with a short iron pole.
Mythology is full of these stories - people all around the world like to exaggerate. Compared to this, warrior monks fighting barehanded do not seem so bad to me.
Of course, in real world medieval armies, bare-handed monks would be chopped to pieces - that's why no nation ever fielded bare-handed warriors. However, unarmed combat played a role in countries where the ruling class forbade weapons to control the populace (Japan).
We must decide, whether our game treads on the grounds of myths or some kind of sober parallel history. With games like WOM and FE, I think the case is clear.