How to Resolve The Error “Illegal instruction (core dumped)” when Running “import tensorflow” in a Python Program

There are several ways to install TensorFlow on Ubuntu. The easiest way is to install via pip. Unfortunately, this easy installation may result in a bumpy first time experience of running TensorFlow. Consider the following one line Python script:

$ python -c 'import tensorflow as tf;'

This should be where the excitement begins, the moment where conviction about the new era of AI-powered banalities starts to bloom. Yet, the reality can be unexpectedly different. Executing the command may immediately raise this very infamous error:

Illegal instruction (core dumped)

This means that TensorFlow has crashed even before it does anything. What a surprise!

The good thing is that we can run gdb to debug Python and start analyzing the call stack. But what’s even better is that we can save the brilliance for later. This error has been repeatedly reported and has conveniently sat on its fame for a while, as reflected on the issue page.

In brief, the error will be thrown if we’re running recent TensorFlow binaries on CPU(s) that do not support Advanced Vector Extensions (AVX), an instruction set that enables faster computation especially for vector operations. Starting from TensorFlow 1.6, pre-built TensorFlow binaries use AVX instructions. An excerpt from TensorFlow 1.6 release announcement:

...
Breaking Changes
Prebuilt binaries are now built against CUDA 9.0 and cuDNN 7.
Prebuilt binaries will use AVX instructions. This may break TF on older CPUs.
...

This implicitly means that the latest version, which is TensorFlow 1.10 by the time this article is written, also use AVX instructions. Hence, the crash on older CPUs.

Verifying AVX Support

Assumption can mislead and should be proven with validation. Despite presuming that the crash might be caused by the absence of AVX support in the CPU, we may need to verify this. We do so by obtaining the information about CPU features and filtering to the features to find out about AVX availability. We use the following command to list all the CPU features.

$ more /proc/cpuinfo | grep flags

Compare these two sample outputs, the first is from a machine with old CPU and the latter is from another machine with more modern CPU.

Older CPU:


...
flags   : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl pni cx16 hypervisor lahf_lm
...

Newer CPU:


...
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts
...

As you can immediately observe, the newer CPU is shipped with many more features, including AVX and AVX2 instruction sets. This is a stark contrast compared to the older CPU, which provides more limited features, hence less performance optimization opportunity.

So, what is the resolve if the CPU in your current machine does not support AVX? We have two possible options:
1. Downgrading to TensorFlow 1.5, which does not use AVX instruction in the binaries
2. Compile TensorFlow and install with only possible CPU optimization

Downgrading to TensorFlow 1.5

The downgrade process is very simple as outlined below.

Step 1: Uninstall the currently installed TensorFlow module

$ pip uninstall tensorflow

Step 2: Install TensorFlow 1.5

$ pip install tensorflow==1.5

Step 3: Verify that the error disappears when running TensorFlow

$ python -c 'import tensorflow as tf; print(tf.__version__)'

After completing the downgrade, we will now be able to run TensorFlow code for serving a model. However, we are running somewhat older version of TensorFlow and may suffer from other bugs or issues that have not been resolved for that version. If running the latest TensorFlow is a preferred route, you can take a look at the explanation in the next section

Installing TensorFlow from Source (Custom Build)

The TensorFlow official documentation provides the guide for building TensorFlow from source. However, the documentation did not really specify how to optimize the source build especially for TensorFlow with CPU support-only. Build optimization takes place by appending additional flags to -march=native, which is the default optimization flag. Optimization flags vary from one machine to another.

To find possible optimization flags in the current machine, we run this command on the command line prompt:

$ grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]' | { read FLAGS; OPT="-march=native"; for flag in $FLAGS; do case "$flag" in "sse4_1" | "sse4_2" | "ssse3" | "fma" | "cx16" | "popcnt" | "avx" | "avx2") OPT+=" -m$flag";; esac; done; MODOPT=${OPT//_/\.}; echo "$MODOPT"; }

From the previous two CPU samples, we will have “-march=native -mcx16” flags for the older CPU and “-march=native -mssse3 -mfma -mcx16 -msse4_1 -msse4_2 -mpopcnt -mavx -mavx2” flags for the more modern CPU. We supply these flags to the build configurator and then generate the WHL file that can be installed to the system.

The full instruction for building TensorFlow that targets CPU optimization is explained in this article.

Github Repository TensorFlow CPU Custom Build

For some reasons, you may find it less practical to build from source and will prefer a more convenient way to install the custom TensorFlow CPU build. If you want to go this route, you can check this Github repository that hosts TensorFlow custom builds, starting from TensorFlow 1.10. Custom builds for future releases will also be added to this repository, subject to the availability of machines and other resources to perform the source build.

12 thoughts on “How to Resolve The Error “Illegal instruction (core dumped)” when Running “import tensorflow” in a Python Program

  1. Lina

    I just had to run a project on a very old laptop and your article really helped with debugging and fixing, thank you for taking the time to share!

    Reply
    1. Mikael Fernandus Simalango Post author

      Glad to know that the article helps prolong the laptop shelf life.

  2. Pingback: "Illegal instruction (core dumped)" on tensorflow >1.6 – Docker Questions

  3. Ankita

    I have a python v3.7. Will installing from source work for me? for tensorflow v1.5 as only v1.13 (tf) and above are available for py3.7
    I have been juggling between python3.8 to 2.7 and tried and tested most of the remedies for this issue. Also have other dependency issues, hence i switched back to python3.7 now but seems its a viscous circle and I am back to the same error.

    Reply
    1. Mikael Fernandus Simalango Post author

      My recommendation is to test in a new environment, let’s say a freshly created VM, and try to build Tensorflow from sources within this environment. Things have changed from the date of the original posts, including the versions of dependencies. By isolating the environment where the build is executed, it will be easier to work around the errors.

  4. Pingback: problems compilling tensorflow from source with bazel – Ask python questions

  5. ucsenime

    hello, I am in a similar position trying to import tensorflow and failing in python 3.8 .
    Could you please explain how you came to the conclusion that the avx/avx2 is the cause of this failure ? I tried to use python-dbg and use gdb to see what is failing but the gdb only shows me some issue in nsync::nsync_mu_init(nsync::nsync_mu_s_*) () from /usr/local/lib/python3.8/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so but does not specify what is trying to do so that I can search for what it is.

    Thanks in advance

    Reply
    1. Mikael Fernandus Simalango Post author

      For more recent versions of Python and Tensorflow, avx/avx2 is not always the culprit. There can be other incompatibility issue or unsafe system call leading to the error. I recommend that you raise about this specific failure to Tensorflow team.

  6. Atlas Liet

    I’m a Windows 7 32bits user.
    I had to install mingw32 to be able to install virtualenv, and after creating the virtual environment and trying to install tensorflow, I find that tensorflow doesn’t work on 32bit systems, and so begins my battle to install rufus and a linux mint 64bit distro, I went to the store and bought a flash drive, made it bootable, spent hours trying to figure out how to handle partitions and not delete my windows, I managed to make a dualboot work with grub, with mint in hand my struggle to install virtualenv began again, but this time having to install the dependencies that don’t come by default in the native python2 version of mint 1.9.3, after installing the dependencies I was able to install virtualenv, and then I try to install tensorflow and find that it only works with python3, another battle starts for me trying to install python3, I manage to install it and so virtualenv can access and clone it, finally it’s time to train my model with tensorflow, I suffer a little more to install it, it apparently works, and then I prepare my entire dataset and when executing the train command….
    boom, your cpu is too old you s*cker.

    I had to crash my face into at least 3 walls to find out that something as fundamental as my CPU is not supported by Tensorflow, really?
    just put “tensorflow doesn’t work on 32bit systems, python 2, or older cpus” at the top of the website and it will save a lot of people’s time, for god’s sake.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *