There isn't just one right way to attach a gun scabbard to your saddle. I have seen rifle scabbards slung on horses many different ways. Just watch a few old John Wayne movies and you will see leather rifle scabbards on the near side of the horse and on the off-side, stock forward and stock facing toward the rear. Whichever way you choose, the most important thing is that it works for you.
I personally hang my Rifle Scabbard on the off-side of the horse with the butt toward the front about horn height and the rifle barrel angled slightly toward the back. Here's why:
One additional note, if you use an open ended leather or nylon rifle scabbard and hang it with the butt toward the rear, your rifle could end up missing. I have heard many stories where folks have found their rifle way back on the trail because they never knew it had been snagged on a branch and yanked out of the gun scabbard.
When hanging a Bow Scabbard, I secure it to the rear of the saddle behind the cantle and angle it slightly forward so that my bow is facing toward the rear with the quiver to the outside. I prefer this attachment method because with the bow scabbard pointed toward the rear, my horse is free to turn tightly to the right without being hindered by the bow. And once again, the weight of the bow and bow scabbard are on the opposite side from me when I mount the horse, helping to offset my weight as I mount.
Pack Saws and Camp Axes
The same applies when I attach my pack saw and its scabbard or my camp axe in its sheath. I prefer to attach my saw scabbard with the saw handle tied to the back strings and the blade coming just under my leg. I usually pack my camp axe on my pack mule to the outside of a mantied load, but you can certainly attach it to your saddle if you prefer an axe. An axe sheath is designed with rings to tie your saddle strings to. This hangs the axe at an angle where it is easy to secure the handle out of your way.
Balance the Load
Wherever you decide to hang your rifle scabbard, bow scabbard or pack saw scabbard, it is important to balance the weight loaded on your horse. You should pack heavier items in the horn bags or saddlebags on the opposite side from your scabbard. Weigh your scabbard so you know how much extra weight it adds and can better offset it.