One of the many lessons from my decades of engineering is this: Use set screws with caution. They are very often misused and end up damaging other components and failing.
Look at your first photo. See the bearing locknut with a lockwasher at the top? That is a very common method of retaining components on a shaft. Maybe you can duplicate that. Another way is to use a taper-lock hub or bushing. Or you could just use a bolt in the end of the shaft with a cap plate. If you do that make sure you provide some method to insure that the bolt does not back out.