This happens to me too, reliably, and particularly for quest huts I create with the Altar quest scroll. I unfortunately have no time now, but I have a save with 3 huts where traveling to them does absolutely nothing, as though they were an empty tile.
