How Portable Can We Get With The Compiler

27 February 2020

Portable binaries

To increase backward compatibility, I’m putting my bet on native binaries for what is on the left of the boundary of portability.

Why native? Why not Java?

That is also a choice worth considering and in the end I may even change my mind to use Java. Nevertheless, it is not given for free either. I would face two problems:

  1. Older Android ships with older version of the JVM, which means older supported library versions, including the deprecated cryptographic capabilities (worst-case scenario, I have not looked it up yet).
    • C++ may have this problem as well, although, in theory and to my best understanding, a newer compiler for the same architecture may help me out with that. It would be worth doing some research on this. However, if C++ fails, then I still have the option to use C.
  2. Java may consume significantly more of the resources which we don’t have, since, again, we’re using old, abandoned devices.
    • Using Java may exclude from the start some of the non-Android devices I could target

Problems with native code

The Key Container Project requires some cryptographic capabilities and a REST interface.

Shipping portable binary applications for Linux is already challenging enough mostly due to dependency incompatibilities, even on the level of library ABIs or libc versions.

Ownyourbits.com does an amazing job explaining how Linux binaries work, I highly recommend that read.

Linux distributors solve this problem by shipping the software themselves. They compile the packages from the original upstream source and tweak them until they work with that particular OS. Folks at a distribution typically compile together a set of software and library versions in a way that every piece of binary is compatible with their dependencies.

I think the reason that this kind of problem is much less known on a Windows platform is that usually only one or two versions of the operating system is targeted by a specific software. Still, related troubles occur on Windows anyway, and since it does not rely on a sophisticated dependency-based package manager, we’ve got DLL hell.

Time to set up the emulator

My first smart phone running Android was an LG GT540 Optimus, starting from version 1.6. Unfortunately, I have already upgraded it to 2.1 Eclair and later hacked it up to 2.3 Gingerbread, as I desperately wanted to have app2sd support on my phone. It did not have a lot of space for apps.

At the time in high school, I was very enthusiastic about that programmable device right in my pocket. It is true even these days! :)

In order to be able to create an AVD with the old version, I had to download the old SDK:

Downloading the old SDK

After that I could select the old platform and create the AVD. Now, here comes the problem:

AVD Manager of Android Studio

While I am able to start the AVD with Android 7.0, clicking the green play triangle on an AVD with version 1.6 (or even 2.3) just won’t do anything. Not even the console did output any error message. The action in the context menu had the same effect (none).

With a little guidance from Stack Overflow I have downloaded the legacy Android SDK. The executable for the old AVD Manager UI was at

android-sdk-linux_x86-1.6_r1/tools/android

Old AVD Manager UI

Using the old SDK, I could create an AVD with Android version even as low as 1.5 (API Level 3)! For the sake of adventure, as long as I do not encounter serious problems, I’m sticking with that. This is what is looks like:

Emulating Android 1.5 with legacy emulator

I’m already feeling nostalgic. Remember the smart phone that looked mostly like your older dumb one but with the capability of installing applications on it? It was like a mini computer! I wonder why I the mood is different of a modern Android device for me. It’s maybe because of the lack of restrictions at the time.

Creating a bare CMake hello world project

Let’s start small:

# CMakeLists.txt:

project (CMakeHelloWorld)
add_executable (hello hello.cpp)
// hello.cpp:

#include <iostream>

int main() {
  std::cout << "Hello world" << std::endl;
  return 0;
}

Configuring the toolchain

In order to compile C++ for Android, we need the Android Native Development Kit, or NDK for short. Among others it contains the appropriate compiler for the platform.

After some digging on the web and opening a couple zip files, it turns out, that the latest NDK that still supports Android 1.5 is revision 11c.

This and other old versions of the kit can be found among the unsupported NDK downloads.

Creating a Standalone NDK Toolchain

According to the manual, if I would like to use the NDK tools independently of Android Studio, then what I need is a standalone toolchain compiled.

/opt/kuklinistvan/android-ndk-r11c $ ./build/tools/make-standalone-toolchain.sh
HOST_OS=linux
HOST_EXE=
HOST_ARCH=x86_64
HOST_TAG=linux-x86_64
HOST_NUM_CPUS=4
BUILD_NUM_CPUS=8
Auto-config: --arch=arm
Auto-config: --toolchain=arm-linux-androideabi-4.9
Auto-config: --platform=android-3
Copying prebuilt binaries...
Copying sysroot headers and libraries...
Copying c++ runtime headers and libraries...
Creating package file: /tmp/ndk-kuklinistvan/arm-linux-androideabi-4.9.tar.bz2
ERROR: Could not create tarball from /tmp/ndk-kuklinistvan/tmp/build-80038/ndk.3NYb4Dh

Although the archive could not be created for some reason, the directory left behind turns out to be useful on its own.

$ mv /tmp/ndk-kuklinistvan/tmp/build-80038/ndk.3NYb4Dh ../android-ndk-r11c-standalone

Compiling hello world

I’ve set up a rudimentary toolchain file to easily work with CMake:

# ndk-r11c.toolchain:

set(CMAKE_SYSTEM_NAME Android)
set(CMAKE_ANDROID_STANDALONE_TOOLCHAIN /opt/kuklinistvan/android-ndk-r11c-standalone/)
/home/kuklinistvan/Desktop/cmake-hello-world $ cmake . -DCMAKE_TOOLCHAIN_FILE="ndk-r11c.toolchain"
-- The C compiler identification is GNU 4.9.0
-- The CXX compiler identification is GNU 4.9.0
-- Check for working C compiler: /opt/kuklinistvan/android-ndk-r11c-standalone/bin/arm-linux-androideabi-gcc
-- Check for working C compiler: /opt/kuklinistvan/android-ndk-r11c-standalone/bin/arm-linux-androideabi-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/kuklinistvan/android-ndk-r11c-standalone/bin/arm-linux-androideabi-gcc
-- Check for working CXX compiler: /opt/kuklinistvan/android-ndk-r11c-standalone/bin/arm-linux-androideabi-gcc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kuklinistvan/Desktop/cmake-hello-world
$ make -j4
make[1]: Entering directory '/home/kuklinistvan/Desktop/cmake-hello-world'
make[2]: Entering directory '/home/kuklinistvan/Desktop/cmake-hello-world'
Scanning dependencies of target hello
make[2]: Leaving directory '/home/kuklinistvan/Desktop/cmake-hello-world'
make[2]: Entering directory '/home/kuklinistvan/Desktop/cmake-hello-world'
[ 50%] Building C object CMakeFiles/hello.dir/hello.c.o
[100%] Linking C executable hello
make[2]: Leaving directory '/home/kuklinistvan/Desktop/cmake-hello-world'
[100%] Built target hello
make[1]: Leaving directory '/home/kuklinistvan/Desktop/cmake-hello-world'

Will it blend?

$ adb push hello /data/local/hello && adb shell /data/local/hello
hello: 1 file pushed. 3.1 MB/s (4005192 bytes in 1.244s)
Hello world
[1]   Illegal instruction     /data/local/hello

Oops.

Okay, there are some versions and combinations to try, here are the results.

C/C++ NDK Version Android version Result
C++ r11c Android 1.5 [1] Illegal instruction
C++ r10e (oldest available) Android 1.5 Frozen until Ctrl+C was hit
C++ r11c Android 1.6 Worked
C++ r10e Android 1.6 Frozen until Ctrl+C was hit
C r11c Android 1.5 Worked
C r10e Android 1.5 Worked
C r11c Android 1.6 Worked
C r10e Android 1.6 Worked

I pretty much think, this is the point where I should abandon the idea of C++. It is a much heavier language compared to C with a lot of abstraction and it is very hard to find out what goes wrong if my application crashes on Android.

On the other hand, gcc compiled the C-language hello words in a pretty stable manner and I think there is still hope for the native way.

It is funny, that two days earlier I was trying to get C++17 support.

Fighting with compilers for modern C++

(spoiler: I’ve lost the battle)

In the beginning I was more naive and actually tried to get the original gcc cross-compile executables for the ARM architecture, expecting them to run on Android. I did that to get modern C++17, because, you know, why not (actually to ensure modern library compatibility - just in case - and for some convenience)

It took me a couple days to realize that this is not the path worth taking. Running gcc -v just to see how the original NDK compiler was built was enough for me to stop even trying. Note the line starting with “Configured with:”.

$ ./arm-linux-androideabi-gcc -v
Using built-in specs.
COLLECT_GCC=./arm-linux-androideabi-gcc
COLLECT_LTO_WRAPPER=/opt/kuklinistvan/android-ndk-r11c-standalone/bin/../libexec/gcc/arm-linux-androideabi/4.9/lto-wrapper
Target: arm-linux-androideabi
Configured with: /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/configure --prefix=/tmp/002ff922da42f0f80603ab1715cd43d4 --target=arm-linux-androideabi --host=x86_64-linux-gnu --build=x86_64-linux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/buildbot/tmp/build/toolchain/temp-install --with-mpfr=/buildbot/tmp/build/toolchain/temp-install --with-mpc=/buildbot/tmp/build/toolchain/temp-install --with-cloog=/buildbot/tmp/build/toolchain/temp-install --with-isl=/buildbot/tmp/build/toolchain/temp-install --with-ppl=/buildbot/tmp/build/toolchain/temp-install --disable-ppl-version-check --disable-cloog-version-check --disable-isl-version-check --enable-cloog-backend=isl --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --disable-libitm --with-float=soft --with-fpu=vfp --with-arch=armv5te --enable-target-optspace --enable-initfini-array --disable-nls --prefix=/tmp/002ff922da42f0f80603ab1715cd43d4 --with-sysroot=/tmp/002ff922da42f0f80603ab1715cd43d4/sysroot --with-binutils-version=2.25 --with-mpfr-version=3.1.1 --with-mpc-version=1.0.1 --with-gmp-version=5.0.5 --with-gcc-version=4.9 --with-gdb-version=none --with-gxx-include-dir=/tmp/002ff922da42f0f80603ab1715cd43d4/include/c++/4.9 --with-bugurl=http://source.android.com/source/report-bugs.html --enable-languages=c,c++ --disable-bootstrap --enable-plugins --enable-libgomp --enable-gnu-indirect-function --disable-libsanitizer --enable-gold --enable-threads --enable-eh-frame-hdr-for-static --enable-graphite=yes --with-isl-version=0.11.1 --with-cloog-version=0.18.0 --with-arch=armv5te --program-transform-name='s&^&arm-linux-androideabi-&' --enable-gold=default
Thread model: posix
gcc version 4.9 20150123 (prerelease) (GCC)

Here was what I’ve tried and their results:

What I’ve tried Result
Simply use a modern version of the NDK and ignore API incompatibility. Got segfault when running the resulted executable right from the first NDK that has dropped support.
Cross-compile with a gcc-arm-gnueabi and change the interpreter to Android’s libc and interpreter by patching the ELF executable. Segfault Android.
Cross-compile with gcc-arm-gnueabi and copy the host’s glibc and interpreter, then patch ELF executables to use that one. Did not find a toolchain for Arch Linux (yeah, for Arch! really!) that compiles for ARM eabi soft float cpu.
Use Debian ARM Docker image for compiling to solve the upper problem. Segfault on Android. (the beauty in segfaults is that this is pretty much all the information I have on what the problem was - the executable tried to instruct the CPU to do something illegal because of incompatibility)
Install a Debian ARM virtual machine as a compiling environment to solve the upper problem. No description on how to install Debian 10.3 inside QEMU ARM Virt board, only some blog posts for older versions. Turns out neither is this straightforward and probably does not worth the effort.
Create an Alpine runtime environment in a folder with all ELF executables patched to use its libc. Worked on the computer, did not on Android, mostly due to the lack of support for soft float version of the binaries for those ARM CPUs that do not have a hardware FPU.
gcc-arm-none-eabi (the compiler for hardware without OS) Segfault (kind of expected, but tried it anyway)
Creating a toolchain with crosstool-ng Takes a LOT of time (more than an hour) to compile the toolchain and nothing guarantees that it will work. Yeah, I’ve actually interrupted the process.
Looking at the NDK source code and try to compile the latest gcc After realizing that not even the official NDK guarantees that the targeted platform is going to be able to run the compiled C++ code, I think it does not worth the time and I’ll probably encounter obstacles that are far beyond my current skills to solve.

Falling back to C

Next time I’ll see what are my chances to compile C libraries, like mbed TLS. I hope that the ending of my next post won’t be titled “Falling back to Java”.