As its name and type signature suggest, foldRT is a bona fide right fold for your tree (you can convince yourself of that by hand-evaluating it for something like Branch 1 (Branch 0 Leaf Leaf) (Branch 2 Leaf Leaf)). That means implementing Foldable will give you foldRT: foldRT = foldr. Why is that relevant? Because, in this case, it is much easier to implement foldMap than it is to figure out how to write foldr directly:
-- Note that l and r here aren't the subtrees, but the results of folding them.
instance Foldable Tree where
foldMap f = fold (\v l r -> l <> f v <> r) mempty
If you want to write foldRT without relying on a Foldable instance, all you need is expanding the foldMap-based definition of foldr (see the answers to this question for the details I'll gloss over here):
foldRT f z t = appEndo (foldMap (Endo . f) t) z
-- To make things less clumsy, let's use:
foldEndo f = appEndo . foldMap (Endo . f)
-- foldEndo f = flip (foldRT f)
foldEndo f = appEndo . fold (\v l r -> l <> Endo (f v) <> r) mempty
-- As we aren't mentioning foldMap anymore, we can drop the Endo wrappers.
-- mempty @(Endo _) = Endo id
-- Endo g <> Endo f = Endo (g . f)
foldEndo f = fold (\v l r -> l . f v . r) id
-- Bringing it all back home:
foldRT :: (a -> b -> b) -> b -> Tree a -> b
foldRT f z t = fold (\v l r -> l . f v . r) id t z
(Note that we ultimately reached a solution of the sort suggested by Carl, only through an indirect route.)