In the world of smart device development, the diversity of the Android ecosystem presents unique challenges for developers. This article explores the use of Rust for native Android service development, with a particular focus on cross-compilation techniques encountered in automotive environments. Drawing from real-world practices, we share our hands-on experience and solutions.

Cross Compilation

1.1 Why Cross-Compilation

In embedded systems—especially in automotive and industrial settings—GreptimeDB Edge is often deployed as a system service on Android. This requires compiling GreptimeDB Edge into an executable suitable for the Android platform.

While it’s theoretically possible to compile directly on an Android development board by installing the Rust toolchain, this approach comes with several limitations:

• Complex environment setup: Setting up Rust on Android boards can be tedious and error-prone.
• Low build efficiency: Android boards typically have underpowered CPUs, making it impractical to build large Rust projects.
• Version mismatch risks: The Android API version or CPU architecture on the dev board may differ from the actual target device, potentially introducing compatibility issues.

Cross-compilation offers a more efficient and scalable solution. It allows developers to compile code on one platform (e.g., x86_64 Linux/macOS) targeting another (e.g., ARM Android)—particularly useful when direct compilation on the target device is infeasible.

1.2 Cross-Compilation Support in Rust

Rust has excellent cross-compilation support and Android NDK provides the necessary toolchain and libs, which further simplifies the cross-compilation.

Let’s briefly understand how Rust compiles code. Rustc first compiles Rust code into LLVM-IR, then LLVM compiles LLVM-IR into binaries for the corresponding platform, and finally the Linker links them together to generate the final binary file.

Rustc is a compiler for Rust, with LLVM as the backend (you can also say that Rustc is the front end of LLVM).

Here is a simplified diagram of the Rust compiler architecture.

（Figure 1: The Compiling Architecture of Rust）

1.3 Cross-Compiling GreptimeDB

We’ll walk through how to cross-compile an open-source version of GreptimeDB on an x86 Linux machine to generate an executable targeting the aarch64-linux-android architecture.

Step 1: Install Android NDK

Download link of NDK see here. In order to facilitate subsequent operations, an environment variable is set. As shown below:

export ANDROID_NDK_HOME=<YOUR_NDK_ROOT>

# example
# export ANDROID_NDK_HOME=/home/fys/soft/ndk/android-ndk-r25c

Step 2: Clone the GreptimeDB Repository

git clone https://github.com/GreptimeTeam/greptimedb.git --depth 1

Step 3: Add Target to Rust Toolchain

Adding Target to the Rust toolchain is a key step to achieve cross-platform compilation. This allows Rustc to compile the intermediate representation (LLVM-IR) code into machine language for the target platform. In this example, the target platform architecture is aarch64-linux-android. Execute the following command in the root directory of the GreptimeDB project:

rustup target add aarch64-linux-android

Full list of supported targets here.

At this time, you may get an error -lgcc not found when trying to compile. The reason is libgcc.a of Android NDK has been replaced by libunwind.a. The workaround is to copy libunwind.a and rename it to libgcc.a. For more details, see the Rust official blog.

cd $ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/lib/clang/17/lib/linux/aarch64/
cp libunwind.a libgcc.a

Step 4: Simplify with cargo-ndk

When your Rust project integrates with C or C++ code via build.rs, things can get tricky. This usually requires providing some necessary information to the compilation tool (such as cc or cmake, including the compiler path (CC and CXX), the paths of libs and header files, etc. Fortunately, cargo-ndk wraps up much of this complexity:

cargo ndk --platform 30 -t aarch64-linux-android build --bin greptime --release

For crates incompatible with the target architecture, you can either:

• Replace them with alternatives
• Use feature guards to exclude them during compilation

If errors mention missing libraries like protobuf, ensure they are properly installed for the target environment.

1.4 Challenges: LTO and Symbol Conflicts

We once encountered a failure when building with LTO (Link Time Optimization) enabled. The error:

ld.lld: error: duplicate symbol: pthread_atfork

This occurs because both Android's native libraries and tikv-jemallocator define pthread_atfork as strong symbols. LTO merges these, causing conflicts.

Fix: Use a weak symbol for pthread_atfork in tikv-jemallocator. Recent versions already address this—details here.

Backtrace on Android

While working on GreptimeDB Edge, we found that Rust's backtrace support on Android is unreliable. Specifically, panic stack traces show up as unknown, making debugging difficult.

2.1 Reproducing the Problem

To reproduce the issue, we wrote a minimal example:

1. Trigger a panic in the main function to simulate an exception:

fn main() {
    panic!("Panic here.");
}

2. Set the rust-toolchain to stable 1.81 or an earlier version:

[toolchain]
channel = "1.81"

3. Cross-compile the program to produce a binary for Android. This step revisits what we've covered earlier:

export ANDROID_NDK_HOME=<YOUR_NDK_ROOT>
rustup target add aarch64-linux-android
cargo ndk --platform 28 -t aarch64-linux-android build --release

4. Push the binary to an Android emulator and run:

RUST_BACKTRACE=full ./<path-to-binary>

5. The result shows that the expected backtrace information is missing. The issue is successfully reproduced:

thread 'main' panicked at src/main.rs:2:5:
        Panic here
        stack backtrace:
           0:     0x5d908f7a7535 - <unknown>
           1:     0x5d908f7b336b - <unknown>
           ...

2.2 Solution

We’ll start with the fix in case you're not interested in the root cause.

Upgrade the Rust toolchain:

Upgrade rust-toolchain to Rust 1.82 or later. The issue has already been fixed in this version (details are explained in the next section).

Custom Panic Hook:

If upgrading is not an option, you can register a custom panic hook using the backtrace-rs crate.

Rust’s default panic hook prints messages to stderr, which may not be ideal in environments like Android, where logging to files or logcat is preferred. In such cases, customizing the panic hook is often necessary.

Here's a simple implementation:

pub fn set_panic_hook() {
    #[cfg(windows)]
    const LINE_ENDING: &str = "\r\n";
    #[cfg(not(windows))]
    const LINE_ENDING: &str = "\n";
    std::panic::set_hook(Box::new(move |panic| {
        let backtrace = backtrace::Backtrace::new();
        let Some(l) = panic.location() else {
            log::error!(
                "Panic: {:?}, backtrace: {}{:#?}",
                panic, LINE_ENDING, backtrace
            );
            return;
        };
        log::error!(
            "Panic: {:?}, file: {}, line: {}, col: {}, backtrace: {}{:#?}",
            panic,
            l.file(),
            l.line(),
            l.column(),
            LINE_ENDING,
            backtrace,
        );
    }));
}

The output stack information is as follows (with debug info removed from the compilation options and the symbol table preserved):

pub fn set_panic_hook() {
    #[cfg(windows)]
    const LINE_ENDING: &str = "\r\n";
    #[cfg(not(windows))]
    const LINE_ENDING: &str = "\n";

    std::panic::set_hook(Box::new(move |panic| {
        let backtrace = backtrace::Backtrace::new();

        let Some(l) = panic.location() else {
            log::error!(
                "Panic: {:?}, backtrace: {}{:#?}",
                panic, LINE_ENDING, backtrace
            );
            return;
        };

        log::error!(
            "Panic: {:?}, file: {}, line: {}, col: {}, backtrace: {}{:#?}",
            panic,
            l.file(),
            l.line(),
            l.column(),
            LINE_ENDING,
            backtrace,
        );
    }));
}

⚠️ Note: The availability and detail of backtrace information also depend on compilation settings. Stripping both symbol tables and debug info will result in a stack trace with <unknown> entries. Keeping debug info provides more detailed traces but increases binary size.

2.3 Root Cause Analysis

Now let’s dig into the underlying cause, based on Rust 1.81.

Background knowledge:

• Rust’s standard library backtrace functionality depends on the backtrace-rs crate, which is included as a git submodule.
• backtrace-rs checks the Android API level during build time. If the API level is ≥ 21, it enables the dl_iterate_phdr feature. However, because Rust integrates backtrace-rs as a submodule without executing its build.rs, the dl_iterate_phdr feature is not enabled. As a result, std::backtrace doesn't work properly on Android.

Mystery solved!

So how do we fix it?

We need to enable the dl_iterate_phdr feature in the standard library's version of backtrace-rs. Fortunately, this was addressed starting with #120593, which raised Rust's minimum Android API level from 19 to 21. Since Android 21 supports dl_iterate_phdr, the feature can now be enabled by default in backtrace-rs. Rust 1.82 adopted exactly this approach, resolving the issue.

🔗 Related PRs:

• backtrace-rs#656
• rust-lang/rust#129305

Binary Size Optimization Practices

3.1 Overview

When developing foundational services on the Android platform, binary size is a critical concern. An oversized binary not only slows down OTA (over-the-air) updates but also reduces the efficiency of hot updates and consumes more storage space. So, how can we effectively reduce the size of a Rust program?

Fortunately, the GitHub project Minimizing Rust Binary Size has comprehensively documented nearly all effective techniques for this purpose. Interested readers can explore that project for in-depth information — we won’t repeat those details here.

Among these techniques, one in particular deserves extra attention: stripping symbol tables and debug info from the binary. While this can dramatically shrink the binary size, it comes at a cost. For example, if the program panics, the stack trace becomes unreadable—making debugging significantly more difficult. So, how do we strike a balance between reducing binary size and retaining readable stack traces in the event of a panic?

GreptimeDB Edge, which runs on in-vehicle systems, offers a solution. The idea is to build two versions of the executable:

• One is stripped of symbols and debug info to achieve minimal size and is used in production on the vehicle system.
• The other retains symbols and debug info and is stored in the cloud.

If the vehicle system encounters an error (e.g., panic or crash), we can retrieve the corresponding version of the binary from the cloud and use its symbols and debug info to reconstruct a human-readable stack trace — providing valuable support for diagnosing and resolving issues.

（Figure 2: Binary size optimization using GreptimeDB Edge as an example）

3.2 Stack Trace Recovery

So, how exactly does stack trace recovery work? Let’s go through a simple demo program to illustrate.

The program below is designed to panic at runtime. It sets up a custom panic hook that prints the memory addresses of loaded objects and captures the stack trace.

main.rs

use std::panic;
use backtrace::Backtrace;
use tracing::error;
fn main() {
    tracing_subscriber::fmt::init();
    set_panic_hook();
    a();
}
#[inline(never)]
fn a() {
    b();
}
#[inline(never)]
fn b() {
    // panic here!!!
    panic!("Oh no!");
}
pub fn set_panic_hook() {
    log_base_addr();
    #[cfg(windows)]
    const LINE_ENDING: &str = "\r\n";
    #[cfg(not(windows))]
    const LINE_ENDING: &str = "\n";
    panic::set_hook(Box::new(move |panic| {
        let backtrace = Backtrace::new();
        let Some(l) = panic.location() else {
            error!(
                "Panic: {:?}, backtrace: {}{:#?}",
                panic, LINE_ENDING, backtrace
            );
            return;
        };
        error!(
            "Panic: {:?}, file: {}, line: {}, col: {}, backtrace: {}{:#?}",
            panic,
            l.file(),
            l.line(),
            l.column(),
            LINE_ENDING,
            backtrace,
        );
    }));
}
fn log_base_addr() {
    for o in phdrs::objects() {
        error!("Object name: {:?}, base addr: {:#x?}", o.name(), o.addr());
    }
}

Cargo.toml

[package]
name = "panic_demo"
version = "0.1.0"
edition = "2021"
[dependencies]
backtrace = "0.3"
phdrs = { git = "https://github.com/softdevteam/phdrs.git", rev = "86992b1e8e896a495387d931072c6088086eeabd" }
tracing = "0.1"
tracing-subscriber = "0.3"
[profile.release]
debug = true
opt-level = "z"
lto = true
codegen-units = 1

First, build the binary with symbols and debug info:

cargo build --release

Note: We’ve overridden the default release profile settings in Cargo.toml.

After building, go to the target/release directory and make a copy of the binary named panic_demo_with_strip, then use the strip tool to remove symbols and debug info:

cd target/release/
cp panic_demo panic_demo_with_strip
strip -s panic_demo_with_strip

Now you have two binaries in target/release:

• panic_demo — with full symbols and debug info
• panic_demo_with_strip — stripped version

When you run panic_demo, you’ll see a readable stack trace because it includes debug symbols:

./panic_demo

➜ ./panic_demo
2025-03-10T03:05:27.843007Z ERROR panic_demo: Object name: "", base addr: 0x5621ce28b000
2025-03-10T03:05:27.843075Z ERROR panic_demo: Object name: "linux-vdso.so.1", base addr: 0x738d45df3000
2025-03-10T03:05:27.843094Z ERROR panic_demo: Object name: "/usr/lib/libc.so.6", base addr: 0x738d45bd8000
2025-03-10T03:05:27.843107Z ERROR panic_demo: Object name: "/usr/lib/libgcc_s.so.1", base addr: 0x738d45baa000
2025-03-10T03:05:27.843120Z ERROR panic_demo: Object name: "/lib64/ld-linux-x86-64.so.2", base addr: 0x738d45df5000
2025-03-10T03:05:27.850986Z ERROR panic_demo: Panic: PanicHookInfo { payload: Any { .. }, location: Location { file: "src/main.rs", line: 20, col: 5 }, can_unwind: true, force_no_backtrace: false }, file: src/main.rs, line: 20, col: 5, backtrace:
   0:     0x5621ce2b4ba5 - panic_demo::set_panic_hook::{{closure}}::hfc3c6c380e6b0be5
                               at /home/fys/rust_target/projects/panic_demo/src/main.rs:33:25
   1:     0x5621ce2f0312 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::ha3fed88e6e913722
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/alloc/src/boxed.rs:1984:9
                           std::panicking::rust_panic_with_hook::h1df75c095a4f3488
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/std/src/panicking.rs:825:13
   2:     0x5621ce31b5a5 - std::panicking::begin_panic_handler::{{closure}}::hf3afa20cd541c11f
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/std/src/panicking.rs:683:13
   3:     0x5621ce31b539 - std::sys::backtrace::__rust_end_short_backtrace::h3ced788cfddd85e3
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/std/src/sys/backtrace.rs:168:18
   4:     0x5621ce31bebc - rust_begin_unwind
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/std/src/panicking.rs:681:5
   5:     0x5621ce2d41ef - core::panicking::panic_fmt::ha07a50819406191f
                               at /rustc/a4cb3c831823d9baa56c3d90514b75b2660116fa/library/core/src/panicking.rs:75:14
   6:     0x5621ce2b4b81 - panic_demo::b::h9ebbf8c80464f859
                               at /home/fys/rust_target/projects/panic_demo/src/main.rs:20:5
   7:     0x5621ce2b4b4b - panic_demo::a::he4eee93f5289646a
                               at /home/fys/rust_target/projects/panic_demo/src/main.rs:14:5
   8:     0x5621ce2b47dd - panic_demo::main::he3522539407f83f5
                               at /home/fys/rust_target/projects/panic_demo/src/main.rs:9:5
   9:     0x5621ce2b2d4d - core::ops::function::FnOnce::call_once::h3e6e811b791b14aa
                               at /home/fys/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
                           std::sys::backtrace::__rust_begin_short_backtrace::h00639e0e41301441
                               at /home/fys/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:152:18
  10:     0x5621ce2b5286 - main
  11:     0x738d45bff488 - <unknown>
  12:     0x738d45bff54c - __libc_start_main
  13:     0x5621ce2b2c55 - _start
  14:                0x0 - <unknown>

However, running panic_demo_with_strip will result in an unreadable stack trace due to the missing symbols and debug info:

./panic_demo_with_strip

❯ ./panic_demo_with_strip
2025-03-10T03:05:49.818147Z ERROR panic_demo: Object name: "", base addr: 0x59bba9fd8000
2025-03-10T03:05:49.818207Z ERROR panic_demo: Object name: "linux-vdso.so.1", base addr: 0x77bf1596d000
2025-03-10T03:05:49.818216Z ERROR panic_demo: Object name: "/usr/lib/libc.so.6", base addr: 0x77bf15752000
2025-03-10T03:05:49.818221Z ERROR panic_demo: Object name: "/usr/lib/libgcc_s.so.1", base addr: 0x77bf15724000
2025-03-10T03:05:49.818226Z ERROR panic_demo: Object name: "/lib64/ld-linux-x86-64.so.2", base addr: 0x77bf1596f000
2025-03-10T03:05:49.818808Z ERROR panic_demo: Panic: PanicHookInfo { payload: Any { .. }, location: Location { file: "src/main.rs", line: 20, col: 5 }, can_unwind: true, force_no_backtrace: false }, file: src/main.rs, line: 20, col: 5, backtrace:
   0:     0x59bbaa001ba5 - <unknown>
   1:     0x59bbaa03d312 - <unknown>
   2:     0x59bbaa0685a5 - <unknown>
   3:     0x59bbaa068539 - <unknown>
   4:     0x59bbaa068ebc - <unknown>
   5:     0x59bbaa0211ef - <unknown>
   6:     0x59bbaa001b81 - <unknown>
   7:     0x59bbaa001b4b - <unknown>
   8:     0x59bbaa0017dd - <unknown>
   9:     0x59bba9fffd4d - <unknown>
  10:     0x59bbaa002286 - <unknown>
  11:     0x77bf15779488 - <unknown>
  12:     0x77bf1577954c - __libc_start_main
  13:     0x59bba9fffc55 - <unknown>
  14:                0x0 - <unknown>

How to recover the stack trace? Let’s take the address 0x59bbaa001ba5 as an example to demonstrate the process of stack trace recovery.

First, in the panic hook, we printed the memory ranges where each object was loaded. From this information, we can determine that 0x59bbaa001ba5 falls within the memory range of the panic_demo_with_strip object. Therefore, we can calculate the offset from the base address as follows:

0x59bbaa001ba5 - 0x59bba9fd8000 - 1 = 0x29ba4

Next, we use the addr2line tool to convert this address into a human-readable file name and line number in the source code.

⚠️ Note: If the lto = true option is set during compilation, GNU addr2line may not work correctly. In this case, you’ll need to use gimli-rs/addr2line.

addr2line -e panic_demo 0x29ba4

Output:

➜ addr2line -e panic_demo 0x29ba4
/home/fys/rust_target/projects/panic_demo/src/main.rs:33

Following the same approach, we can convert other addresses into corresponding file names and line numbers in the source code. And just like that, we've recovered a human-readable stack trace!

Conclusion

Cross-compilation has always been a challenging task. There’s no one-size-fits-all solution, and various problems can arise along the way. In most cases, we need to handle issues on a case-by-case basis. Fortunately, tools like Cargo NDK and the Android NDK provide a convenient set of solutions that help us effectively address most of the compilation challenges.

Through the discussion in this article, we’ve come to understand the importance of cross-compilation in the Android environment, as well as the advantages of Rust’s build system. While the ideal compilation process often encounters many real-world obstacles, we hope that our experience can offer some practical insights and inspiration for future development.

---

About Greptime

Greptime offers industry-leading time series database products and solutions to empower IoT and Observability scenarios, enabling enterprises to uncover valuable insights from their data with less time, complexity, and cost.

GreptimeDB is an open-source, high-performance time-series database offering unified storage and analysis for metrics, logs, and events. Try it out instantly with GreptimeCloud, a fully-managed DBaaS solution—no deployment needed!

The Edge-Cloud Integrated Solution combines multimodal edge databases with cloud-based GreptimeDB to optimize IoT edge scenarios, cutting costs while boosting data performance.

Star us on GitHub or join GreptimeDB Community on Slack to get connected.