If you tried to build Iroha on Raspberry from source, you might encounter a problem of virtual memory exhaustion when building irohad (specifically during the build of on_demand_ordering_init.o), or one of the errors listed in this page: Common Errors when building Hyperledger Iroha from source code.
virtual memory exhausted: Cannot allocate memory make[3]: *** [irohad/main/CMakeFiles/application.dir/build.make:76: irohad/main/CMakeFiles/application.dir/impl/on_demand_ordering_init.o] Error 1
If you haven’t yet installed all the dependencies for iroha, you may also want to have a look at this page: Install dependencies of Hyperledger Iroha manually.
At this moment, there are no official Hyperledger docker images of Iroha for Raspberry PI. There are some implementations out there, such as the iroha-pi (https://github.com/tkyonezu/iroha-pi), an environment for building and running hyperledger/iroha more easily, but you need to trust where these implementations come from. I have personally not tried it, so I can’t really say much about it.
The following shows how to build Hyperledger Iroha on Raspberry PI. No custom software of docker images involved.
USE THIS INFORMATION AT YOUR RISK. THE FOLLOWING IS PROVIDED ON A “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND.
Disclaimer: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this article are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
Tested on the following hardware/software:
- Raspberry PI 3 Model B+ (armv7l)
- Raspbian GNU/Linux 10 (buster) – also tested on Raspbian 9 (stretch)
- Hyperledger Iroha v1.1.3 – also tested on 1.1.1
Table of Contents
- Executive Summary
- On “virtual memory exhausted: Cannot allocate memory”
- Step by step Building
- Dependencies
- Build on Raspberry armv7l
- Build on Debian x86_64
- Cross-compile for armv7l on x86_64
- Quick check that the cross-compiling was successful and correct
- Conclusion
- References & Useful links
0) Straight to the solution: Cross-compiling for armv7l on x86_64
If you want to go straight to the point (this works for me, hope it does for you too) on how to compile on_demand_ordering_init.o for armv7l on x86_64, then here it is.
In the following, we use the arm-linux-gnueabihf-g++-8 compiler (because we are compiling on stretch), which you may need to install it first on your machine with: sudo apt-get install g++-arm-linux-gnueabihf gdb-multiarch. In the command below, the secret sauce, the magic formula, the abracadabra is provided by these options: -mfloat-abi=hard -mfpu=vfp -mtls-dialect=gnu -marm -march=armv6+fp.
The following is to be run from the Debian desktop and it assumes that the iroha source code is contained in /home/user/PKG/CROSS/iroha.
Before launching the build, move to the build/irohad/main directory:cd /home/pi/PKG/CROSS/iroha/build/irohad/main and then run:
arm-linux-gnueabihf-g++-8 -mfloat-abi=hard -mfpu=vfp -mtls-dialect=gnu -marm -march=armv6+fp -DBOOST_ALL_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_NO_IOSTREAM -DBOOST_NO_RTTI -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_HEADER_ONLY -I/home/pi/PKG/CROSS/iroha/irohad -I/home/pi/PKG/CROSS/iroha/libs -I/home/pi/PKG/CROSS/iroha/shared_model -I/home/pi/PKG/CROSS/iroha/build/schema -isystem /home/pi/PKG/CROSS/grpc/include -isystem /usr/local/include/soci -isystem /usr/include/postgresql -isystem /usr/include -Wall -fdiagnostics-color=always -g -g -Wextra -Wno-unused-parameter -Wno-deprecated-declarations -O0 -fPIC -std=gnu++14 -o CMakeFiles/application.dir/impl/on_demand_ordering_init.o -c /home/pi/PKG/CROSS/iroha/irohad/main/impl/on_demand_ordering_init.cpp
An easy way to get this command right is given by simply building iroha on the Debian desktop using the default build command (iroha uses cmake) and add the abracadabra above. You can see the precise command by running ps ax | grep c++ while on_demand_ordering_init is being built.
1) Executive Summary
Very briefly, the method consists in:
- building iroha on Raspberry (which is armv7l) until (most likely) the virtual memory of the Raspberry is exhausted,
- cross-compiling the file on_demand_ordering_init.o for arm on a x86_64 Debian Linux machine,
- copying the generated on_demand_ordering_init.o back to the Raspberry and keep the building process of irohad.
A FEW IMPORTANT POINTS TO KEEP IN MIND (namely, taking into account the following items has worked for me):
- The very same Debian distribution and version is used on both the Raspberry (Raspbian 10 buster) and the Linux machine (Debian 10 buster). This is just to ensure that the versions of the libraries used are the same.
- The version of the compiler used on Raspbian and Debian should be the same (actually this is not really necessary; the important thing is that the cross-compiler can generate a file compatible with the files built on the Raspberry. For more info, please refer to the links at the end of the page.
- In the following of this page, iroha is built on both the armv7l raspberry and the x64 debian machine, then only on_demand_ordering_init.o is cross-compiled. Another way, potentially faster is to run the gcc preprocessor on the file (-E flag) in Raspberry and then cross-compile only this file on the x64 Debian, as described here (https://raspberrypi.stackexchange.com/questions/93610/how-to-compile-project-on-raspberry-pi-even-when-gs-virtual-memory-exhausted/93612). Have not tried this way.
2) On “virtual memory exhausted: Cannot allocate memory”
Compiling iroha on Raspberry fails because of the large virtual memory needed to build the iroha daemon irohad. In fact, the build fails even when the Raspberry PI is configured with very large swaps (4GB or even 8GB are not enough), so that’s not a way out.
It is still recommended to configure the Raspberry with 2GB of swap.
Actually, while compiling on_demand_ordering_init.o, the Raspberry does not use more than 1300-1400MB and when the compilation error occurs, there is still free memory available:
pi@raspberry:~ $ free -m
total used free shared buff/cache available
Mem: 926 862 25 0 37 20
Swap: 2047 1376 671
virtual memory exhausted: Cannot allocate memory make[3]: *** [irohad/main/CMakeFiles/application.dir/build.make:76: irohad/main/CMakeFiles/application.dir/impl/on_demand_ordering_init.o] Error 1
3) Step by step Building
3.1) Dependencies
Iroha currently uses the vcpkg dependency manager (https://iroha.readthedocs.io/en/master/build/index.html). If this method does not work on your raspberry, you may want to install each dependency manually. Luckily, it is easy to find what iroha dependencies are: they are simply listed in this file: iroha/vcpkg/build_iroha_deps.sh. If you look into it, then you’ll see that the dependencies are contained in just two files:
- VCPKG_DEPS_LIST
- VCPKG_HEAD_DEPS_LIST
While installing each dependency, you just need to be careful to install the right version for iroha. You can find here Install dependencies of Hyperledger Iroha manually a step-by-step guide on how to install each dependency manually. Some care must be taken if you plan on building the dependencies from source, as some need very specific options at build time.
3.2) Build on Raspberry armv7l
On Raspberry we disable the tests (-DTESTING=OFF) to build iroha faster (it will still take around one day or so, so be patient…)
cmake -H. -Bbuild -DTESTING=OFF
Keep an eye on the errors, if any. In some cases you may need to explicitly pass the directory where you’ve installed the oneTBB dependency:
cmake -H. -Bbuild -DTESTING=OFF -Dtbb_INCLUDE_DIR=/home/pi/PKG/oneTBB/include -Dtbb_LIBRARY=/home/pi/PKG/oneTBB/build/linux_armv7_gcc_cc8.3.0_libc2.28_kernel4.19.97_release/libtbb.so -LA
then
mkdir build
cd build
cmake ..
make -j4
If the building stops and exits with the infamous “virtual memory exhausted: Cannot allocate memory”, from your x86_64 linux machine copy the glorious cross-compiled on_demand_ordering_init.o to build/irohad/main/CMakeFiles/application.dir/impl/on_demand_ordering_init.o and simply re-launch from the build/ directory cmake -j4 and now it should be OK.
Building iroha-cli should not throw errors.
3.3) Build on Debian x86_64
The command to build on x86 is the same. Depending on how you’ve installed the dependencies, you may need to configure the library locations explicitly, or not, so always keep an eye on the output of cmake -H. -Bbuild -DTESTING=OFF if you see any “library not found” message.
For instance, the following command explicitly specifies a number of libraries needed by iroha: tbb, grpc, gpr, SOCI, SOCI_postgresql, address_sorting.
cmake -H. -Bbuild -DTESTING=OFF -Dtbb_INCLUDE_DIR=/home/pi/PKG/CROSS/oneTBB/include -Dtbb_LIBRARY=/home/pi/PKG/CROSS/oneTBB/ -Dgrpc_LIBRARY=/home/pi/PKG/CROSS/grpc/cmake/build/libgrpc.so -Dgrpc_INCLUDE_DIR=/home/pi/PKG/CROSS/grpc/include -Dgrpc_CPP_PLUGIN=/home/pi/PKG/CROSS/grpc/cmake/build/grpc_cpp_plugin -Dgrpc_grpc++_LIBRARY=/home/pi/PKG/CROSS/grpc/cmake/build/libgrpc++.so -Dgpr_LIBRARY=/home/pi/PKG/CROSS/grpc/cmake/build/libgpr.so -DSOCI_LIBRARY=/usr/local/lib64/libsoci_core.a -DSOCI_postgresql_PLUGIN=/usr/local/lib64/libsoci_postgresql.so -Daddress_sorting_LIBRARY=/home/pi/PKG/CROSS/grpc/cmake/build/libaddress_sorting.so -LA
3.4) Cross-compile for armv7l on x86_64
This command builds for armv7l on x86_64: arm-linux-gnueabihf-g++-8 -mfloat-abi=hard -mfpu=vfp -mtls-dialect=gnu -marm -march=armv6+fp.
The remaining part of the command is the same as the default building process. It can be seen during the build process on your Debian Linux machine by launching a ps ax | grep c++ while on_demand_ordering_init is being built.
The file you need will be saved to CMakeFiles/application.dir/impl/on_demand_ordering_init.o and this is the file you’ll need to copy to the same path on your raspberry.
3.5) Quick check that the cross-compiling was successful and correct
Run this on your Debian Linux machine (the iroha source code is assumed to be in this directory: /home/user/PKG/CROSS/iroha).
user@linux:~$ arm-linux-gnueabihf-readelf -A PKG/CROSS/iroha/build/irohad/main/CMakeFiles/application.dir/impl/on_demand_ordering_init.o
Attribute Section: aeabi
File Attributes
Tag_CPU_name: “6”
Tag_CPU_arch: v6
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_ABI_optimization_goals: Aggressive Debug
Tag_CPU_unaligned_access: v6
If you run the same command readelf -A on the raspberry, on any .o file of the iroha build, you should get the same output:
pi@raspberry:~ $ readelf -A PKG/iroha/build/shared_model/backend/protobuf/CMakeFiles/shared_model_proto_backend.dir/queries/impl/proto_get_peers.o
Attribute Section: aeabi
File Attributes
Tag_CPU_name: “6”
Tag_CPU_arch: v6
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_ABI_optimization_goals: Aggressive Debug
Tag_CPU_unaligned_access: v6
4) Conclusion
Done!
5) References and useful links
- Hyperledger Iroha: https://github.com/hyperledger/iroha
- Build for ARMv6 with gnueabihf: https://stackoverflow.com/questions/35132319/build-for-armv6-with-gnueabihf
- How to compile project on Raspberry Pi, even when g++’s virtual memory exhausted despide big SWAP: https://raspberrypi.stackexchange.com/questions/93610/how-to-compile-project-on-raspberry-pi-even-when-gs-virtual-memory-exhausted/93612
- Cross compiling for ARM with Ubuntu 16.04 LTS: http://jensd.be/800/linux/cross-compiling-for-arm-with-ubuntu-16-04-lts
- https://github.com/rust-lang/rust/issues/45284
- Making a Raspbian Cross Compilation SDK: https://medium.com/@zw3rk/making-a-raspbian-cross-compilation-sdk-830fe56d75ba
- iroha-pi is an environment for building and running hyperledger iroha more easily: https://github.com/tkyonezu/iroha-pi (not tested)
Was this useful?
Let us know at: 
