why binaries from one OS don't work on another

Idea of taking calc.exe and just running it on Linux/whatever is obviously absurd. However, taking a binary from a unix-like system (say, Linux) and running it on a different system (say, FreeBSD) may pose a legitimate question why that would not work without special support.

The list below is by no means complete and I'm too lazy to educate myself more on the subject.

So, let's take a look what's needed to get a binary running.

Of course there is functionality specific to given system (e.g. epoll vs kqueue), extensions to common functionality or minor differences in semantics of common functionality, but let's ignore that.

binary loading

The kernel has to parse the binary. Both systems use ELF format, so headers are readable, but it does not mean either system can make sense of everything found inside.

ELF supports passing additional information to the process (apart from argument vector and the environment), but the information passed is os-specific.

A dynamically linked binary contains a hardcoded path to the linker which is supposed to be used and typically requires some librariers (like libc), but e.g. sizes of various structures can be different or macros can expand to different symbols.

As such even if you loaded glibc along with FreeBSD binary it would not work, and the linker does not like the binary anyway.

So one would have to provide a complete enough environment with all necessary binary files.

system calls

Programs open files, talk over the network etc. by asking the kernel to perform a specific action. This is achieved by the use of syscalls.

The kernel has a table of system calls. Userspace processes tell the kernel which syscall they want and what arguments should be used.

Even common syscalls can (and often do) have different numbers (FreeBSD, Linux). So our binary would end up calling wrong syscalls, which obviously cannot work.

Do you at least do the same thing to call a syscall with given arguments? Well...

On i386 systems FreeBSD expects syscall arguments to be on the stack, while Linux expects them in registers. A way of invoking a syscall is the same though.

On amd64 both systems do the same thing, but one could make them differ for the sake of it.

conclusion

Supporting trivial binaries from other systems, which only use functionality provided by your system is not very hard. You can provide a dedicated system call table and make sure your signal delivery works.

Running non-trivial binaries requires a significant effort.

Such work was performed in FreeBSD (and other BSDs) and allows it to run Linux binaries. Although there are a lot of missing syscalls (e.g. inotify) and there are some terrible hacks, the layer is quite usable.