Bayesianism is fundamentally an epistemic study of belief.
As such you can come to the completely wrong conclusion and then say my belief in the wrong conclusion is justified.
Furthermore it seems to make a lot of assumptions that transcendental realists and empirical idealists make, so it could be said all considering that it's epistemologically flawed but that is a very complicated subject I am not ready to flesh out.
I would argue that this is not the fault of Bayesianism but rather it's an unfortunate fact of reality that we do not have access to all conceivable information. I.e., your world-view is not perfect. Once you learn that there's something that conflicts your current world-view, you build and modify your model.
Science doesn't make proclamations about the truth, but it makes tentative statements based on the best information and models available to us.
Before we learned that Newtonian laws of gravity are incorrect, it made perfect sense for us to assume that Mercury traced an ellipse. Once we discovered new information that conflicted with that world view, we revised our models.
Bayesianism just happens to be the best method, currently, to learn about the natural world. If you can come up with an example of another epistemology which can show that something we both accept as true is true, but which Bayesian analysis states is false, I'll be very surprised.