The type of bearing you're using is not intended for this type of application. It is intended for mainly radial loads with lesser thrust loads. In this case the thrust loads are the primary forces. You need a thrust bearing with a captured center to absorb the off-center loads created by the offset swing radius. Do a Google search for "swivel caster cross section" and look at the images. You'll see what I mean.
Which brings up a question - why are you even designing a device that is in such wide use already? You could probably buy a proven design off the shelf somewhere.