From experience: HDL code turns out to be easier to port than ARM Code. Because FPGAs come with pins, not peripherals.
The part of code that's portable across ARM suppliers is the core.
Peripherals and their APIs are proprietary. The code that deals with that usually needs a rewrite when moving between devices and brands. In an embedded system that's the bulk of the code.
The R5 is not for Linux, so you don't get the advantage of OS and device driver abstraction.
Because of that, I use the manufacturer's toolchain, when the aim for portability yields low reuse and causes significant effort.
I prefer to spend that time to automate the compile/synthesize, test, deploy activities ...
The short answer is no, you cannot build R5 and PSU binaries without Vitis. You will need Vitis to build - and run - software applications on the R5s and A53s of the ZU+ on the Ultra96-V2 board.
