How to Resolve The Error “Illegal instruction (core dumped)” when Running “import tensorflow” in a Python Program

There are several ways to install TensorFlow on Ubuntu. The easiest way is to install via pip. Unfortunately, this easy installation may result in a bumpy first time experience of running TensorFlow. Consider the following one line Python script:

$ python -c 'import tensorflow as tf;'

This should be where the excitement begins, the moment where conviction about the new era of AI-powered banalities starts to bloom. Yet, the reality can be unexpectedly different. Executing the command may immediately raise this very infamous error:

Illegal instruction (core dumped)

This means that TensorFlow has crashed even before it does anything. What a surprise!

The good thing is that we can run gdb to debug Python and start analyzing the call stack. But what’s even better is that we can save the brilliance for later. This error has been repeatedly reported and has conveniently sit on its fame for a while, as reflected on the issue page.

In brief, the error will be thrown if we’re running recent TensorFlow binaries on CPU(s) that do not support Advanced Vector Extensions (AVX), an instruction set that enables faster computation especially for vector operations. Starting from TensorFlow 1.6, pre-built TensorFlow binaries use AVX instructions. An excerpt from TensorFlow 1.6 release announcement:

...
Breaking Changes
Prebuilt binaries are now built against CUDA 9.0 and cuDNN 7.
Prebuilt binaries will use AVX instructions. This may break TF on older CPUs.
...

This implicitly means that the latest version, which is TensorFlow 1.10 by the time this article is written, also use AVX instructions. Hence, the crash on older CPUs.

Verifying AVX Support

Assumption can mislead and should be proven with validation. Despite presuming that the crash might be caused by the absence of AVX support in the CPU, we may need to verify this. We do so by obtaining the information about CPU features and filtering to the features to find out about AVX availability. We use the following command to list all the CPU features.

$ more /proc/cpuinfo | grep flags

Compare these two sample outputs, the first is from a machine with old CPU and the latter is from another machine with more modern CPU.

Older CPU:

...
flags   : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl pni cx16 hypervisor lahf_lm
...

Newer CPU:

...
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts
...

As you can immediately observe, the newer CPU is shipped with many more features, including AVX and AVX2 instruction sets. This is a stark contrast compared to the older CPU, which provides more limited features, hence less performance optimization opportunity.

So, what is the resolve if the CPU in your current machine does not support AVX? We have two possible options:
1. Downgrading to TensorFlow 1.5, which does not use AVX instruction in the binaries
2. Compile TensorFlow and install with only possible CPU optimization

Downgrading to TensorFlow 1.5

The downgrade process is very simple as outlined below.

Step 1: Uninstall the currently installed TensorFlow module
$ pip uninstall tensorflow

Step 2: Install TensorFlow 1.5
$ pip install tensorflow==1.5

Step 3: Verify that the error disappears when running TensorFlow
$ python -c 'import tensorflow as tf; print(tf.__version__)'

After completing the downgrade, we will now be able to run TensorFlow code for serving a model. However, we are running somewhat older version of TensorFlow and may suffer from other bugs or issues that have not been resolved for that version. If running the latest TensorFlow is a preferred route, you can take a look at the explanation in the next section

Installing TensorFlow from Source (Custom Build)

The TensorFlow official documentation provides the guide for building TensorFlow from source. However, the documentation did not really specify how to optimize the source build especially for TensorFlow with CPU support-only. Build optimization takes place by appending additional flags to -march=native, which is the default optimization flag. Optimization flags vary from one machine to another.

To find possible optimization flags in the current machine, we run this command on the command line prompt:


$ grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]' | { read FLAGS; OPT="-march=native"; for flag in $FLAGS; do case "$flag" in "sse4_1" | "sse4_2" | "ssse3" | "fma" | "cx16" | "popcnt" | "avx" | "avx2") OPT+=" -m$flag";; esac; done; MODOPT=${OPT//_/\.}; echo "$MODOPT"; }

From the previous two CPU samples, we will have “-march=native -mcx16” flags for the older CPU and “-march=native -mssse3 -mfma -mcx16 -msse4_1 -msse4_2 -mpopcnt -mavx -mavx2” flags for the more modern CPU. We supply these flags to the build configurator and then generate the WHL file that can be installed to the system.

The full instruction for building TensorFlow that targets CPU optimization is explained in this article.

Github Repository TensorFlow CPU Custom Build

For some reasons, you may find it less practical to build from source and will prefer a more convenient way to install the custom TensorFlow CPU build. If you want to go this route, you can check this Github repository that hosts TensorFlow custom builds, starting from TensorFlow 1.10. Custom builds for future releases will also be added to this repository, subject to the availability of machines and other resources to perform the source build.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.