The BBC are showing and excellent documentary on the nativity. What emerges is the sheer strength and character of Mary. But I know you are a natural doubter.
What if Joseph suspects she did make the whole thing up? After all, it is much more likely and easy that instead of being conceived by the Holy Spirit, she did conceive Jesus in the ordinary way. Joseph is like a doubter.