In the previous posts, we have gone through the installation processes for deep learning infrastructure, such as Docker, nvidia-docker, CUDA Toolkit and cuDNN. With the infrastructure setup, we may conveniently start delving into deep learning: building, training, and validating deep neural network models, and applying the models into a certain problem domain. Translating deep learning primitives into low level bytecode execution can be an enormous task, especially for practitioners without interests in the deep learning calculus. Fortunately, there are several deep learning frameworks that provide the high level programming interface to assist in performing deep learning tasks.
In this post, we will go through the installation of Caffe2, one of the major deep learning frameworks. Caffe2 is adopted from Caffe, a deep learning framework developed by the Barkeley Vision and Learning Center (BVLC) of UC Berkeley. Caffe2 was started with the aim to improve Caffe especially to better support large-scale distributed model training, mobile deployment, reduced precision computation, new hardware, and flexibility of porting to multiple platforms.
Why Installing from Source?
It is important to note that instead of installation by building from source, there are some easier options, as for example by pulling and running Caffe2 Docker images. The Docker repository for Caffe2 images is accessible here. Nonetheless, if you observe the list, the images are not recent and they were built against older version of CUDA toolkit and cuDNN. Therefore, one who intends to fiddle with latest Caffe2, CUDA, or cuDNN features may seriously consider installing from source.
Caffe2 Is Now A Part of Pytorch
In the past, Caffe2 source was maintained as an independent repository on Github. This changed from the end of March 2018 since Caffe2 repository was merged into Pytorch repository. Consequently, the common build process is now integrated into that of Pytorch.
There can be questions about the motivation of the source code merging. As explained in the announcement post, the merging was preceded by the development infrastructure sharing between Caffe2 and Pytorch that led into increasing amount of shared code and common libraries. It was then concluded that merging the two projects into one codebase would result in better engineering efficiency and increase the robustness of the framework especially in the area of model deployment.
What if Caffe2 has been previously installed? When installing Caffe2 prior to the source code merging, the build process would output header files, dynamic library and python library that subsequently copied to designated directories. In theory, it would suffice to replace the old header and library files with the one from the newest build. If it does not work, a clean up is necessary by first removing the old files from their installed directories prior to proceeding with the installation
Pre-Requisites
We will be installing Caffe2 with GPU support. This article only covers the installation process for machines with NVIDIA GPUs. Verify that these items are satisfied prior to going through the installation.
1. Verify that NVIDIA graphics driver has been properly installed.
You can refer to this article for NVIDIA graphics driver installation and functionality checking.
2. Verify that CUDA toolkit has been installed.
If you have not installed CUDA toolkit, are unsure whether it is already installed, or want to install the newer version of CUDA toolkit, refer to this article that provides more elaboration about CUDA toolkit installation.
3. Verify that cuDNN has been installed.
You can refer to this article for more details about cuDNN installation
4. Remove files from previous Caffe2 installation
If you have installed Caffe2 earlier and want to do an upgrade, you may need to clean the files created and copied in the previous Caffe 2 installation.
Note: this can usually be skipped as the build tool will first check the existing version of Caffe2 installed in the system prior to replacing it with a new one
$ sudo rm -vRf /usr/local/include/caffe2
$ sudo rm -vf /usr/local/lib/libcaffe2*Additionally, we may need to remove the python APIs from the Python package directory (path may vary depending on the installed Python version).
$ sudo rm -vRf /usr/local/lib/python2.7/dist-packages/caffe2After checking all the prerequisites, we can now proceed with the installation.
Installation Steps
The installation can be accomplished by going through these steps in order.
Step 1. Update apt package index
$ sudo apt-get updateStep 2. Install apt package dependencies
remote$ sudo apt-get install -y --no-install-recommends build-essential cmake git libgoogle-glog-dev libgtest-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev openmpi-bin openmpi-doc protobuf-compiler python-dev python-pipStep 3. Install pip dependencies
$ sudo pip install --upgrade pip
$ sudo pip install setuptools future numpy protobuf
$ sudo apt-get install -y --no-install-recommends libgflags-devStep 4. Clone Caffe 2 into a local directory
Note: We create a directory named caffe2-pytorch and clone Pytorch git repository into this directory.
$ mkdir caffe2-pytorch && cd caffe2-pytorch
$ git clone --recursive https://github.com/pytorch/pytorch.git ./
$ git submodule update --initStep 5. Build Caffe 2
$ mkdir build && cd build
$ cmake ..
$ sudo make -j"$(nproc)" installNote: when building the source, we supply the -j flag. The flag refers to the number of threads that can be spawned by the compiler (i.e. GCC) when building the source code. The nproc command itself will print the number of the CPUs available. In short, we want to speed up the compilation time by creating a number of threads that amount to the number of CPUs to perform the compilation in parallel.
Step 6. Create the symbolic link for Caffe 2 shared library
$ sudo ldconfigStep 7. Verify that Caffe 2 library and the headers have been installed in the proper directory
– Update the locate database
$ sudo updatedb– Ensure that libcaffe2.so is located in /usr/local/lib
$ locate libcaffe2.so– Ensure that the header files have been copied into /usr/local/include
$ locate caffe2 | grep /usr/local/include/caffe2Step 8. Add Caffe2 into python and library paths so that it can be immediately discovered and linked by other applications.
$ vim ~/.profile...
# set python path
if [ -z "$PYTHONPATH" ]; then
PYTHONPATH=/usr/local
else
PYTHONPATH=/usr/local:$PYTHONPATH
fi
#set library path
if [ -z "$LD_LIBRARY_PATH" ]; then
LD_LIBRARY_PATH=/usr/local/lib
else
LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
fi
...We can also set the library paths using parameter expansion as follows:
...
PYTHONPATH=/usr/local${PYTHONPATH:+:${PYTHONPATH}}
LD_LIBRARY_PATH=/usr/local/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
...We then immediately export the new paths by invoking the source command as follows.
$ source ~/.profileAlternatively, we can use the dot (.) command, to execute ~/.profile file and enable the new environment variables in the current shell.
$ . ~/.profileStep 9. Verify that the Caffe2 python module can be properly invoked
$ python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"Step 10. Verify that Caffe2 can run with GPU support
$ python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'The installation is now complete. We can now start using Caffe2 for deep learning modeling and implementation. If you have problem with your Caffe2 installation, simply write it in the comment section.
Pingback: How to Properly Install NVIDIA Graphics Driver on Ubuntu 16.04 « Amikelive | Technology Blog
Hi, I have a problem in verification
python -c ‘from caffe2.python import core’ 2>/dev/null && echo “Success” || echo “Failure”
Failure
python2 -c ‘from caffe2.python import workspace; print(workspace.NumCudaDevices())’
WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu
CRITICAL:root:Cannot load caffe2.python. Error: No module named caffe2_pybind11_state
As the error hinted, your GPU may not have the support for CUDA. You can invoke the command “nvidia-smi” to get the GPU model and check if it is CUDA-enabled.
If you’re working with a desktop, the list of CUDA-enabled GPUs can be found in this article: https://tech.amikelive.com/node-685/list-of-nvidia-desktop-graphics-card-models-for-building-deep-learning-ai-system/
# Caffe2 python run does not have GPU support
### system info
– PyTorch or Caffe2: Caffe2
– OS: ubuntu16.04
– How I installed Caffe2: source
– Python version: python 3.6.0
– CUDA/cuDNN version:8.0.61/7.0.5
– GPU models and configuration:Nvidia GTX 1080
– GCC version:5.4.0
– cmake version: 3.5.1
– miniconda
### What I did
– I have tried to build caffe2 from source (following [this](https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile#install-with-gpu-support)):
After I installed the dependencies on that page and clone the repository, I run this command `FULL_CAFFE2=1 python setup.py install`. I test `from caffe2.python import core` and I got `Failure`
– Then I find a solution in github [following this(@kaisark)](https://github.com/pytorch/pytorch/issues/6259)
I passed the test and got Success but I found it doesn’t support GPU.
### Test caffe2 python core
“`
(cf2) caffe@learner-pc:~$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> from caffe2.python import core
WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
>>>
“`
### Relavent Information
“`
(cf2) caffe@learner-pc:~$ which python
/home/caffe/miniconda3/envs/cf2/bin/python
(cf2) caffe@learner-pc:~$ python –version
Python 3.6.0 :: Continuum Analytics, Inc.
(cf2) caffe@learner-pc:~$ echo $PYTHONPATH
(cf2) caffe@learner-pc:~$ echo $PATH
/home/caffe/miniconda3/envs/cf2/bin:/home/caffe/bin:/home/caffe/.local/bin:/sbin:/home/caffe/miniconda3/bin:/usr/local/cuda-8.0/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/usr/local/protobuf/bin
(cf2) caffe@learner-pc:~$ echo $LD_LIBRARY_PATH
/usr/local/lib:/usr/local/cuda-8.0/lib64::/usr/local/protobuf/lib:/home/caffe/github/nccl/build/lib
(cf2) caffe@learner-pc:~$
“`
### Thanks for any help!
Any reason that prevented you from upgrading to CUDA 9? I haven’t tested the build for CUDA 8, so this might not be the best suggestion that you can get.
You may need to check the location of libcaffe2_gpu.so in the system using this command:
$ locate libcaffe2 | grep libcaffe2_gpuMake sure that this library is located in the same directory with libcaffe2.so. If the library does not exist, you need to rebuild from source again.
Hi, Thanks for the detailed installing instruction. I am trying to install caffe2 on my machine, but keep getting this error. Could you guide me to resolve the issue.
[ 88%] Linking CXX shared module python/caffe2_pybind11_state_gpu.so
[ 88%] Built target caffe2_pybind11_state_gpu
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
Error code 2 means that the file (caffe2_pybind11_state_gpu.so) does not exist. If the source was building properly the .so file should have beeen located under SRC_HOME/build/caffe2/python/ directory. A possible cause is that cmake could not find CUDA. Could you paste the output of “echo $LD_LIBRARY_PATH”, “echo $PYTHONPATH”, and the output of cmake for CUDA:
USE_CUDA : ON/OFF
CUDA version : x.x.x
If you follow the steps explained in the article, USE_CUDA must be ON and CUDA version must be 9.1.
Hi Thank for the prompt respond :
the output for cmake is
— ******** Summary ********
— General:
— CMake version : 3.5.1
— CMake command : /usr/bin/cmake
— Git version : v0.1.11-8684-g36c3859
— System : Linux
— C++ compiler : /usr/bin/c++
— C++ compiler version : 5.4.0
— BLAS : Eigen
— CXX flags : -fvisibility-inlines-hidden -DONNX_NAMESPACE=onnx_c2 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-unused-parameter -Wno-unknown-warning-option -Wno-unknown-pragmas
— Build type : Release
— Compile definitions :
—
— BUILD_CAFFE2 : ON
— BUILD_ATEN : OFF
— BUILD_BINARY : ON
— BUILD_CUSTOM_PROTOBUF : ON
— Link local protobuf : ON
— BUILD_DOCS : OFF
— BUILD_PYTHON : ON
— Python version : 2.7.12
— Python includes : /usr/include/python2.7
— BUILD_SHARED_LIBS : ON
— BUILD_TEST : ON
— USE_ASAN : OFF
— USE_CUDA : ON
— CUDA static link : OFF
— USE_CUDNN : ON
— CUDA version : 9.2
— cuDNN version : 6.0.21
— CUDA root directory : /usr/local/cuda
— CUDA library : /usr/local/cuda-9.2/lib64/stubs/libcuda.so
— cudart library : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
— cublas library : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
— cufft library : /usr/local/cuda/lib64/libcufft.so
— curand library : /usr/local/cuda/lib64/libcurand.so
— cuDNN library : /home/paperspace/anaconda3/lib/libcudnn.so
— nvrtc : /usr/local/cuda-9.2/lib64/libnvrtc.so
— CUDA include path : /usr/local/cuda/include
— NVCC executable : /usr/local/cuda/bin/nvcc
— CUDA host compiler : /usr/bin/cc
— USE_TENSORRT : OFF
— USE_ROCM : OFF
— USE_EIGEN_FOR_BLAS : ON
— USE_FFMPEG : OFF
— USE_GFLAGS : ON
— USE_GLOG : ON
— USE_GLOO : ON
— USE_GLOO_IBVERBS : OFF
— USE_LEVELDB : ON
— LevelDB version : 1.20
— Snappy version : ..
— USE_LITE_PROTO : OFF
— USE_LMDB : ON
— LMDB version : 0.9.21
— USE_METAL : OFF
— USE_MKL :
— USE_MOBILE_OPENGL : OFF
— USE_MPI : ON
— USE_NCCL : ON
— USE_SYSTEM_NCCL : OFF
— USE_NERVANA_GPU : OFF
— USE_NNPACK : ON
— USE_OBSERVERS : ON
— USE_OPENCL : OFF
— USE_OPENCV : ON
— OpenCV version : 3.3.1
— USE_OPENMP : OFF
— USE_PROF : OFF
— USE_REDIS : OFF
— USE_ROCKSDB : OFF
— USE_ZMQ : OFF
and echo returns are
echo $LD_LIBRARY_PATHâ, âecho $PYTHONPATH
/home/paperspace/src/torch/install/lib:/home/paperspace/src/torch/install/lib:/usr/local/lib:/home/paperspace/src/torch/install/lib:â, âecho /home/paperspace/src/caffe/python:
and
paperspace@psx6p289c:~$ echo $PYTHONPATH
/home/paperspace/src/caffe/python:
At first glance, that configuration looks fine. I suppose CUDA 9.2 did not play nice with cuDNN 6.0, but this should be further validated because i have yet to configure with that kind of CUDA-cuDNN version pair.
If you can also provide the output for “nvidia-smi” and “nvcc –version” commands, it can add more datapoints for the root cause analysis.
Additionally, you may consider upgrading cuDNN to 7.1.x and recompile again.
Hi, so I redid the installation again :
paperspace@psx6p289c:~$ nvidia-smi
Mon Jun 4 11:53:05 2018
+—————————————————————————–+
| NVIDIA-SMI 396.26 Driver Version: 396.26 |
|——————————-+———————-+———————-+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P6000 Off | 00000000:00:05.0 On | Off |
| 26% 31C P8 17W / 250W | 586MiB / 24449MiB | 1% Default |
+——————————-+———————-+———————-+
+—————————————————————————–+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2436 G /usr/lib/xorg/Xorg 372MiB |
| 0 2766 G /usr/bin/gnome-shell 120MiB |
| 0 3484 G …-token=480B4330CA48007D74D9DD70DFC02263 88MiB |
+—————————————————————————–+
paperspace@psx6p289c:~$ nvcc –version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Wed_Apr_11_23:16:29_CDT_2018
Cuda compilation tools, release 9.2, V9.2.88
+——————————————————————————–+
Cmake :
— Python version : 2.7.12
— Python includes : /usr/include/python2.7
— BUILD_SHARED_LIBS : ON
— BUILD_TEST : ON
— USE_ASAN : OFF
— USE_CUDA : ON
— CUDA static link : OFF
— USE_CUDNN : ON
— CUDA version : 9.2
— cuDNN version : 7.1.4
+———————————————————————————-+
echo $PYTHONPATH
/home/paperspace/caffe2-pytorch/caffe2/python:
echo $LD_LIBRARY_PATH
/usr/local/lib:
+————————————————————————————–+
error:
/usr/bin/ld: CMakeFiles/mpi_gpu_test.dir/mpi/mpi_gpu_test.cc.o: undefined reference to symbol ‘_ZN3MPI8Datatype4FreeEv’
//usr/lib/libmpi_cxx.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
caffe2/CMakeFiles/mpi_gpu_test.dir/build.make:110: recipe for target ‘bin/mpi_gpu_test’ failed
make[2]: *** [bin/mpi_gpu_test] Error 1
CMakeFiles/Makefile2:2319: recipe for target ‘caffe2/CMakeFiles/mpi_gpu_test.dir/all’ failed
make[1]: *** [caffe2/CMakeFiles/mpi_gpu_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs….
[ 88%] Linking CXX executable ../bin/predictor_test
[ 88%] Linking CXX executable ../bin/string_ops_test
[ 88%] Linking CXX executable ../bin/event_gpu_test
[ 88%] Built target predictor_test
[ 88%] Built target string_ops_test
[ 88%] Built target event_gpu_test
[ 88%] Linking CXX executable ../bin/math_gpu_test
[ 88%] Built target math_gpu_test
[ 88%] Linking CXX executable ../bin/blob_test
CMakeFiles/blob_test.dir/core/blob_test.cc.o: In function `caffe2::(anonymous namespace)::ContentChunks_Serialization_Test::TestBody()’:
blob_test.cc:(.text+0x3b356): warning: the use of `tmpnam’ is dangerous, better use `mkstemp’
[ 88%] Built target blob_test
[ 88%] Linking CXX shared module python/caffe2_pybind11_state_gpu.so
[ 88%] Built target caffe2_pybind11_state_gpu
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
+—————————————————————————————————+
I notice that I don’t have “caffe2_pybind11_state_gpu.so” in my ~/caffe2-pytorch/caffe2/python what you think I should do to add that libarary to the path.
Looking at the log this time, it’s more obvious that the error was caused by Open MPI. You may have not installed Open MPI or the library files are not linked properly.
Quick resolve: Build without Open MPI support
$ cmake .. -DUSE_MPI=OFFRobust resolve: Track the Open MPI status, install if it has not been installed or fix the library linking
– To check if Open MPI has been installed
$ mpirun --version– To check where the library is located
$ locate libmpi– To search Open MPI in apt repositories
$ apt-cache search openmpi– To install relevant Open MPI libraries
$ sudo apt-get install libopenmpi-dev libopenmpi1.10 openmpi-bin openmpi-common openmpi-docThis issue is a known Caffe2 issue. You can find more info about it here: https://github.com/caffe2/caffe2/issues/2144
thank you man, I had openmpi 1-10-2, But disabling it resolved the issue.
Glad to know it resolved your issue. As a side note, if you’re building a cluster (GPU/CPU), it’s suggested to compile with Open MPI support. -Mike-
I followed the steps and when I tried from caffe2.python import core, I get ImportError: No module named caffe2.python
The installation may have not been completed properly. Would you provide the output of these commands when invoked from the terminal?
$ echo $LD_LIBRARY_PATH$ echo $PYTHONPATH$ locate libcaffe2Also, if you can provide the output of cmake, it will also help.
I did a fresh install again but this time, make install won’t finish. Here are my output.
[ 78%] Linking CXX shared module python/caffe2_pybind11_state.so
[ 78%] Built target caffe2_pybind11_state
CMakeFiles/Makefile2:1384: recipe for target ‘caffe2/CMakeFiles/caffe2_gpu.dir/all’ failed
make[1]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/all] Error 2
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
PYTHONPATH is
/opt/ros/kinetic/lib/python2.7/dist-packages
LD_LIBRARY_PATH is
/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/usr/local/cuda-9.0/lib64
locate libcaffe2 is
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_observers.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_observers.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_module_test_dynamic.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_observers.so
/home/dexai/exp/densepose/build/libcaffe2_detectron_custom_ops.so
/home/dexai/exp/densepose/build/libcaffe2_detectron_custom_ops_gpu.so
/home/dexai/exp/pytorch/build/lib/libcaffe2.so
/home/dexai/exp/pytorch/build/lib/libcaffe2_observers.so
/home/dexai/exp/pytorch/build/lib/libcaffe2_protos.a
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_gpu.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_observers.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_gpu.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_observers.so
/usr/local/lib/libcaffe2.so
/usr/local/lib/libcaffe2_detectron_ops_gpu.so
/usr/local/lib/libcaffe2_gpu.so
/usr/local/lib/libcaffe2_observers.so
>>> from caffe2.python import core
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python2.7/dist-packages/caffe2/python/core.py”, line 14, in
from caffe2.proto import caffe2_pb2
File “/usr/local/lib/python2.7/dist-packages/caffe2/proto/caffe2_pb2.py”, line 23, in
t\x12\x17\n\x0freport_interval\x18\x08 \x01(\x05\x12\x14\n\x0crun_every_ms\x18\x0b \x01(\x03\x12\x1b\n\x13\x63oncurrent_substeps\x18\x06 \x01(\x08\x12\x18\n\x10should_stop_blob\x18\t \x01(\t\x12\x11\n\tonly_once\x18\n \x01(\x08\x12\x18\n\x10\x63reate_workspace\x18\x0c \x01(\x08\x12 \n\x18num_concurrent_instances\x18\r \x01(\x05\”g\n\x07PlanDef\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x1f\n\x07network\x18\x02 \x03(\x0b\x32\x0e.caffe2.NetDef\x12-\n\x0e\x65xecution_step\x18\x03 \x03(\x0b\x32\x15.caffe2.ExecutionStep\”\xba\x01\n\tBlobProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0c\n\x04type\x18\x02 \x01(\t\x12#\n\x06tensor\x18\x03 \x01(\x0b\x32\x13.caffe2.TensorProto\x12\x0f\n\x07\x63ontent\x18\x04 \x01(\x0c\x12%\n\x07qtensor\x18\x05 \x01(\x0b\x32\x14.caffe2.QTensorProto\x12\x1a\n\x12\x63ontent_num_chunks\x18\x06 \x01(\x05\x12\x18\n\x10\x63ontent_chunk_id\x18\x07 \x01(\x05\”K\n\rDBReaderProto\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0e\n\x06source\x18\x02 \x01(\t\x12\x0f\n\x07\x64\x62_type\x18\x03 \x01(\t\x12\x0b\n\x03key\x18\x04 \x01(\t*\x90\x01\n\nDeviceType\x12\x07\n\x03\x43PU\x10\x00\x12\x08\n\x04\x43UDA\x10\x01\x12\n\n\x06MKLDNN\x10\x02\x12\n\n\x06OPENGL\x10\x03\x12\n\n\x06OPENCL\x10\x04\x12\t\n\x05IDEEP\x10\x05\x12\x07\n\x03HIP\x10\x06\x12!\n\x1d\x43OMPILE_TIME_MAX_DEVICE_TYPES\x10\x07\x12\x14\n\rONLY_FOR_TEST\x10\xc5\xde\xfb\t’)
TypeError: __init__() got an unexpected keyword argument ‘syntax’
I successfully finished the build process, but now I have this new error.
This could be a problem with protobuf version. You may need to check the current protobuf lib version installed.
$ pip show protobufThere can be incompatibility issue with the most recent protobuf installed via pip against your caffe2 environment. i also noticed that you have CUDA 9.0 instead of the most recent 9.2 installed.
Thanks, yeah it was a protobuf version problem. Everything’s fixed now.
I did a new fresh installation of Caffe2 again, and this time, sudo make install is outputting errors.
[ 78%] Linking CXX shared module python/caffe2_pybind11_state.so
[ 78%] Built target caffe2_pybind11_state
CMakeFiles/Makefile2:1384: recipe for target ‘caffe2/CMakeFiles/caffe2_gpu.dir/all’ failed
make[1]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/all] Error 2
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
My LD_LIBRARY_PATH is:
/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/usr/local/cuda-9.0/lib64
MY PYTHONPATH is:
/opt/ros/kinetic/lib/python2.7/dist-packages/
locate libcaffe2 gives:
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/envs/caffe2/lib/libcaffe2_observers.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/envs/densepose/lib/libcaffe2_observers.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda-cudnn-full-0.8.dev-py27hfdda352_0/lib/libcaffe2_module_test_dynamic.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_gpu.so
/home/dexai/anaconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_observers.so
/home/dexai/exp/densepose/build/libcaffe2_detectron_custom_ops.so
/home/dexai/exp/densepose/build/libcaffe2_detectron_custom_ops_gpu.so
/home/dexai/exp/pytorch/build/lib/libcaffe2.so
/home/dexai/exp/pytorch/build/lib/libcaffe2_observers.so
/home/dexai/exp/pytorch/build/lib/libcaffe2_protos.a
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_gpu.so
/home/dexai/miniconda2/envs/newcaffe2/lib/libcaffe2_observers.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_detectron_ops_gpu.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_gpu.so
/home/dexai/miniconda2/pkgs/caffe2-cuda9.0-cudnn7-0.8.dev-py27_2018.07.20/lib/libcaffe2_observers.so
/usr/local/lib/libcaffe2.so
/usr/local/lib/libcaffe2_detectron_ops_gpu.so
/usr/local/lib/libcaffe2_gpu.so
/usr/local/lib/libcaffe2_observers.so
This is my cmake output
— The CXX compiler identification is GNU 5.4.0
— The C compiler identification is GNU 5.4.0
— Check for working CXX compiler: /usr/bin/c++
— Check for working CXX compiler: /usr/bin/c++ — works
— Detecting CXX compiler ABI info
— Detecting CXX compiler ABI info – done
— Detecting CXX compile features
— Detecting CXX compile features – done
— Check for working C compiler: /usr/bin/cc
— Check for working C compiler: /usr/bin/cc — works
— Detecting C compiler ABI info
— Detecting C compiler ABI info – done
— Detecting C compile features
— Detecting C compile features – done
— Not forcing any particular BLAS to be found
— Build type not set – defaulting to Release
— Performing Test CAFFE2_LONG_IS_INT32_OR_64
— Performing Test CAFFE2_LONG_IS_INT32_OR_64 – Success
— Does not need to define long separately.
— Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
— Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED – Success
— std::exception_ptr is supported.
— Performing Test CAFFE2_IS_NUMA_AVAILABLE
— Performing Test CAFFE2_IS_NUMA_AVAILABLE – Success
— NUMA is available
— Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
— Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING – Success
— Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
— Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS – Success
— Current compiler supports avx2 extention. Will build perfkernels.
— Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
— Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY – Success
— Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
— Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY – Success
— Building using own protobuf under third_party per request.
— Use custom protobuf build.
— Looking for pthread.h
— Looking for pthread.h – found
— Looking for pthread_create
— Looking for pthread_create – not found
— Looking for pthread_create in pthreads
— Looking for pthread_create in pthreads – not found
— Looking for pthread_create in pthread
— Looking for pthread_create in pthread – found
— Found Threads: TRUE
— Caffe2 protobuf include directory: $$
— Found Git: /usr/bin/git (found version “2.7.4”)
— The BLAS backend of choice:Eigen
— Brace yourself, we are building NNPACK
— The ASM compiler identification is GNU
— Found assembler: /usr/bin/cc
— Found PythonInterp: /usr/bin/python (found version “2.7.12”)
— Check if compiler accepts -pthread
— Check if compiler accepts -pthread – yes
— Caffe2: Cannot find gflags automatically. Using legacy find.
— Found gflags: /usr/include
— Caffe2: Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
— Caffe2: Cannot find glog automatically. Using legacy find.
— Found glog: /usr/include
— Caffe2: Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
— Found LMDB: /usr/include
— Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
— Found LevelDB: /usr/include
— Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
— Found Snappy: /usr/include
— Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
— Found Numa: /usr/include
— Found Numa (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnuma.so)
— OpenCV found (/opt/ros/kinetic/share/OpenCV-3.3.1-dev)
— Found system Eigen at /usr/local/include/eigen3
Python 2.7.12
— Setting Python’s include dir to /usr/include/python2.7 from distutils.sysconfig
— Setting Python’s library to /usr/lib/python2.7
— Found PythonInterp: /usr/bin/python (found suitable version “2.7.12”, minimum required is “2.7”)
— Found PythonLibs: /usr/lib/python2.7 (found suitable version “2.7.12”, minimum required is “2.7”)
— Found NumPy: /home/dexai/.local/lib/python2.7/site-packages/numpy/core/include (found version “1.14.5”)
— NumPy ver. 1.14.5 found (include: /home/dexai/.local/lib/python2.7/site-packages/numpy/core/include)
— Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)
— Unable to determine MPI from MPI driver /usr/local/bin/mpicc
— Could NOT find MPI_C (missing: MPI_C_LIBRARIES MPI_C_INCLUDE_PATH)
— Unable to determine MPI from MPI driver /usr/local/bin/mpicxx
— Could NOT find MPI_CXX (missing: MPI_CXX_LIBRARIES MPI_CXX_INCLUDE_PATH)
CMake Warning at cmake/Dependencies.cmake:439 (message):
Not compiling with MPI. Suppress this warning with -DUSE_MPI=OFF
Call Stack (most recent call first):
CMakeLists.txt:184 (include)
— Found CUDA: /usr/local/cuda (found suitable version “9.0”, minimum required is “7.0”)
— Caffe2: CUDA detected: 9.0
— Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
— Caffe2: CUDA toolkit directory: /usr/local/cuda
— Caffe2: Header version is: 9.0
— Found CUDNN: /usr/local/cuda/include
— Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
— Autodetected CUDA architecture(s): 6.1
— Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
— Found NCCL: /usr/include
— Determining NCCL version from the header file: /usr/include/nccl.h
— NCCL_MAJOR_VERSION: 2
— Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so)
— Could NOT find CUB (missing: CUB_INCLUDE_DIR)
— Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY)
— CUDA detected: 9.0
— Found libcuda: /usr/local/cuda-9.0/lib64/stubs/libcuda.so
— Found libnvrtc: /usr/local/cuda-9.0/lib64/libnvrtc.so
— Determining NCCL version from the header file: /usr/include/nccl.h
— NCCL_MAJOR_VERSION: 2
— Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so)
CMake Warning at cmake/Dependencies.cmake:661 (message):
mobile opengl is only used in android or ios builds.
Call Stack (most recent call first):
CMakeLists.txt:184 (include)
CMake Warning at cmake/Dependencies.cmake:737 (message):
Metal is only used in ios builds.
Call Stack (most recent call first):
CMakeLists.txt:184 (include)
— GCC 5.4.0: Adding gcc and gcc_s libs to link line
— Include NCCL operators
— Excluding ideep operators as we are not using ideep
— Including image processing operators
— Excluding video processing operators due to no opencv
— Excluding mkl operators as we are not using mkl
— MPI operators skipped due to no MPI support
— Include Observer library
— Using lib/python2.7/dist-packages as python relative installation path
— Automatically generating missing __init__.py files.
CMake Warning at CMakeLists.txt:348 (message):
Generated cmake files are only fully tested if one builds with system glog,
gflags, and protobuf. Other settings may generate files that are not well
tested.
—
— ******** Summary ********
— General:
— CMake version : 3.5.1
— CMake command : /usr/bin/cmake
— Git version : v0.1.11-9501-g6f10944
— System : Linux
— C++ compiler : /usr/bin/c++
— C++ compiler version : 5.4.0
— BLAS : Eigen
— CXX flags : -fvisibility-inlines-hidden -DONNX_NAMESPACE=onnx_c2 -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations
— Build type : Release
— Compile definitions :
— CMAKE_PREFIX_PATH :
— CMAKE_INSTALL_PREFIX : /usr/local
—
— BUILD_CAFFE2 : ON
— BUILD_ATEN : OFF
— BUILD_BINARY : ON
— BUILD_CUSTOM_PROTOBUF : ON
— Link local protobuf : ON
— BUILD_DOCS : OFF
— BUILD_PYTHON : ON
— Python version : 2.7.12
— Python executable : /usr/bin/python
— Pythonlibs version : 2.7.12
— Python library : /usr/lib/python2.7
— Python includes : /usr/include/python2.7
— Python site-packages: lib/python2.7/dist-packages
— BUILD_SHARED_LIBS : ON
— BUILD_TEST : OFF
— USE_ASAN : OFF
— USE_ATEN : OFF
— USE_CUDA : ON
— CUDA static link : OFF
— USE_CUDNN : ON
— CUDA version : 9.0
— cuDNN version : 7.1.4
— CUDA root directory : /usr/local/cuda
— CUDA library : /usr/local/cuda-9.0/lib64/stubs/libcuda.so
— cudart library : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
— cublas library : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
— cufft library : /usr/local/cuda/lib64/libcufft.so
— curand library : /usr/local/cuda/lib64/libcurand.so
— cuDNN library : /usr/local/cuda/lib64/libcudnn.so
— nvrtc : /usr/local/cuda-9.0/lib64/libnvrtc.so
— CUDA include path : /usr/local/cuda/include
— NVCC executable : /usr/local/cuda/bin/nvcc
— CUDA host compiler : /usr/bin/cc
— USE_TENSORRT : OFF
— USE_ROCM : OFF
— USE_EIGEN_FOR_BLAS : ON
— USE_FFMPEG : OFF
— USE_GFLAGS : ON
— USE_GLOG : ON
— USE_GLOO : ON
— USE_GLOO_IBVERBS : OFF
— USE_LEVELDB : ON
— LevelDB version : 1.18
— Snappy version : 1.1.3
— USE_LITE_PROTO : OFF
— USE_LMDB : ON
— LMDB version : 0.9.17
— USE_METAL : OFF
— USE_MKL :
— USE_MOBILE_OPENGL : OFF
— USE_MPI : OFF
— USE_NCCL : ON
— USE_SYSTEM_NCCL : OFF
— USE_NERVANA_GPU : OFF
— USE_NNPACK : ON
— USE_OBSERVERS : ON
— USE_OPENCL : OFF
— USE_OPENCV : ON
— OpenCV version : 3.3.1
— USE_OPENMP : OFF
— USE_PROF : OFF
— USE_REDIS : OFF
— USE_ROCKSDB : OFF
— USE_ZMQ : OFF
— Public Dependencies : Threads::Threads;gflags;glog::glog
— Private Dependencies : nnpack;cpuinfo;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/lib/x86_64-linux-gnu/libsnappy.so;/usr/lib/x86_64-linux-gnu/libnuma.so;opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs;opencv_videoio;opencv_video;gloo;onnxifi_loader;gcc_s;gcc;dl
— Configuring done
— Generating done
— Build files have been written to: /home/dexai/exp/pytorch/build
Can we use the instruction on pytorch (https://github.com/pytorch/pytorch#from-source) to install? What is the differences? Thanks!
The instruction in Pytorch uses Anaconda while this article builds directly on the OS.
I was able to build with CPU only, but am having the following error on GPU. (I tested cuda 9.2, 9.1, 9.0, with cudnn7, and cuda 8.0 with cudnn6. All these combination will have the following error information).
Scanning dependencies of target caffe2_observers
Scanning dependencies of target caffe2_pybind11_state
[ 74%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/sgd/caffe2_gpu_generated_yellowfin_op_gpu.cu.o
[ 74%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/utils/caffe2_gpu_generated_math_gpu.cu.o
[ 74%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/core/caffe2_gpu_generated_THCCachingAllocator.cu.o
[ 74%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/core/caffe2_gpu_generated_context_gpu.cu.o
[ 74%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/operators/caffe2_gpu_generated_abs_op.cu.o
[ 75%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/operators/caffe2_gpu_generated_accumulate_op.cu.o
[ 75%] Building CXX object modules/observers/CMakeFiles/caffe2_observers.dir/net_observer_reporter_print.cc.o
[ 76%] Building CXX object caffe2/CMakeFiles/caffe2_pybind11_state.dir/python/pybind_state.cc.o
/usr/lib/gcc/x86_64-linux-gnu/5/include/avx512fintrin.h(9220): error: argument of type “const void *” is incompatible with parameter of type “const float *”
/usr/lib/gcc/x86_64-linux-gnu/5/include/avx512fintrin.h(9231): error: argument of type “const void *” is incompatible with parameter of type “const float *”
/usr/lib/gcc/x86_64-linux-gnu/5/include/avx512fintrin.h(9244): error: argument of type “const void *” is incompatible with parameter of type “const double *”
/usr/lib/gcc/x86_64-linux-gnu/5/include/avx512fintrin.h(9255): error: argument of type “const void *” is incompatible with parameter of type “const double *”
——————————————————————————————————
I was using cmake -DUSE_OPENCV=0 .. (without OPENCV=0 will cause other issues ..).
— Does not need to define long separately.
— std::exception_ptr is supported.
— NUMA is available
— Current compiler supports avx2 extention. Will build perfkernels.
— Building using own protobuf under third_party per request.
— Use custom protobuf build.
— Caffe2 protobuf include directory: $$
— The BLAS backend of choice:Eigen
— Brace yourself, we are building NNPACK
— Found PythonInterp: /usr/bin/python (found version “2.7.12”)
— Caffe2: Cannot find gflags automatically. Using legacy find.
— Caffe2: Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
— Caffe2: Cannot find glog automatically. Using legacy find.
— Caffe2: Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
— Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
— Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
— Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
— Found Numa (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnuma.so)
CMake Warning at cmake/Dependencies.cmake:330 (find_package):
By not providing “FindEigen3.cmake” in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by “Eigen3”, but
CMake did not find one.
Could not find a package configuration file provided by “Eigen3” with any
of the following names:
Eigen3Config.cmake
eigen3-config.cmake
Add the installation prefix of “Eigen3” to CMAKE_PREFIX_PATH or set
“Eigen3_DIR” to a directory containing one of the above files. If “Eigen3”
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
CMakeLists.txt:184 (include)
— Did not find system Eigen. Using third party subdirectory.
Python 2.7.12
— Setting Python’s include dir to /usr/include/python2.7 from distutils.sysconfig
— Setting Python’s library to /usr/lib/python2.7
— Found PythonInterp: /usr/bin/python (found suitable version “2.7.12”, minimum required is “2.7”)
— NumPy ver. 1.15.0 found (include: /home/asl/.local/lib/python2.7/site-packages/numpy/core/include)
— Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)
— MPI support found
— MPI compile flags: -pthread
— MPI include path: /usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include/usr/lib/openmpi/inclu
de/openmpi/usr/lib/openmpi/include
— MPI LINK flags path: -Wl,-rpath -Wl,/usr/lib/openmpi/lib -Wl,–enable-new-dtags -pthread
— MPI libraries: /usr/lib/openmpi/lib/libmpi_cxx.so/usr/lib/openmpi/lib/libmpi.so
CMake Warning at cmake/Dependencies.cmake:434 (message):
OpenMPI found, but it is not built with CUDA support.
Call Stack (most recent call first):
CMakeLists.txt:184 (include)
— Caffe2: CUDA detected: 8.0
— Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
— Caffe2: CUDA toolkit directory: /usr/local/cuda
— Caffe2: Header version is: 8.0
— Found cuDNN: v6.0.21 (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
— Autodetected CUDA architecture(s): 5.2
— Added CUDA NVCC flags for: -gencode;arch=compute_52,code=sm_52
— Determining NCCL version from the header file: /usr/include/nccl.h
— NCCL_MAJOR_VERSION: 2
— Found NCCL (include: /usr/include, library: /usr/lib/libnccl.so)
— Could NOT find CUB (missing: CUB_INCLUDE_DIR)
— Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY)
— MPI include path: /usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include/usr/lib/openmpi/inclu
de/openmpi/usr/lib/openmpi/include
— MPI libraries: /usr/lib/openmpi/lib/libmpi_cxx.so/usr/lib/openmpi/lib/libmpi.so
— CUDA detected: 8.0
— Found libcuda: /usr/local/cuda/lib64/stubs/libcuda.so
— Found libnvrtc: /usr/local/cuda/lib64/libnvrtc.so
— Determining NCCL version from the header file: /usr/include/nccl.h
— NCCL_MAJOR_VERSION: 2
— Found NCCL (include: /usr/include, library: /usr/lib/libnccl.so)
— GCC 5.5.0: Adding gcc and gcc_s libs to link line
— Include NCCL operators
— Excluding ideep operators as we are not using ideep
— Excluding image processing operators due to no opencv
— Excluding video processing operators due to no opencv
— Excluding mkl operators as we are not using mkl
— Include Observer library
— Using lib/python2.7/dist-packages as python relative installation path
— Automatically generating missing __init__.py files.
— A previous caffe2 cmake run already created the __init__.py files.
CMake Warning at CMakeLists.txt:352 (message):
Generated cmake files are only fully tested if one builds with system glog,
gflags, and protobuf. Other settings may generate files that are not well
tested.
[36/1866]
—
— ******** Summary ********
— General:
— CMake version : 3.10.2
— CMake command : /usr/local/bin/cmake
— Git version : v0.1.11-9750-g783f2c6
— System : Linux
— C++ compiler : /usr/bin/c++
— C++ compiler version : 5.5.0
— BLAS : Eigen
— CXX flags : -fvisibility-inlines-hidden -rdynamic -DONNX_NAMESPACE=onnx_c2 -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-arr
ay-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=depr
ecated-declarations -Wno-stringop-overflow
— Build type : Release
— Compile definitions :
— CMAKE_PREFIX_PATH :
— CMAKE_INSTALL_PREFIX : /usr/local
—
— BUILD_CAFFE2 : ON
— BUILD_ATEN : OFF
— BUILD_BINARY : ON
— BUILD_CUSTOM_PROTOBUF : ON
— Link local protobuf : ON
— BUILD_DOCS : OFF
— BUILD_PYTHON : ON
— Python version : 2.7.12
— Python executable : /usr/bin/python
— Pythonlibs version : 2.7.12
— Python library : /usr/lib/python2.7
— Python includes : /usr/include/python2.7
— Python site-packages: lib/python2.7/dist-packages
— BUILD_SHARED_LIBS : ON
— BUILD_TEST : OFF
— USE_ASAN : OFF
— USE_ATEN : OFF
— USE_CUDA : ON
— CUDA static link : OFF
— USE_CUDNN : ON
— CUDA version : 8.0
— cuDNN version : 6.0.21
— CUDA root directory : /usr/local/cuda
— CUDA library : /usr/local/cuda/lib64/stubs/libcuda.so
— cudart library : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
— cublas library : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
— cufft library : /usr/local/cuda/lib64/libcufft.so
— curand library : /usr/local/cuda/lib64/libcurand.so
— cuDNN library : /usr/lib/x86_64-linux-gnu/libcudnn.so
— nvrtc : /usr/local/cuda/lib64/libnvrtc.so
— CUDA include path : /usr/local/cuda/include
— NVCC executable : /usr/local/cuda/bin/nvcc
— CUDA host compiler : /usr/bin/cc
— USE_TENSORRT : OFF
— USE_ROCM : OFF
— USE_EIGEN_FOR_BLAS : ON
— USE_FFMPEG : OFF
— USE_GFLAGS : ON
— USE_GLOG : ON
— USE_GLOO : ON
— USE_GLOO_IBVERBS : OFF
— USE_LEVELDB : ON
— LevelDB version : 1.18
— Snappy version : 1.1.3
— USE_LITE_PROTO : OFF
— USE_LMDB : ON
— LMDB version : 0.9.17
— USE_METAL : OFF
— USE_MKL :
— USE_MOBILE_OPENGL : OFF
— USE_MPI : ON
— USE_NCCL : ON
— USE_SYSTEM_NCCL : OFF
— USE_NERVANA_GPU : OFF
— USE_NNPACK : ON
— USE_OBSERVERS : ON
— USE_OPENCL : OFF
— USE_OPENCV : 0
— USE_OPENMP : OFF
— USE_PROF : OFF
— USE_REDIS : OFF
— USE_ROCKSDB : OFF
— USE_ZMQ : OFF
— Public Dependencies : Threads::Threads;gflags;glog::glog
— Private Dependencies : nnpack;cpuinfo;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/lib/x86_64-linux-gnu/libsnappy.so;/usr/lib/x86_64-linux-gnu/libn
uma.so;/usr/lib/openmpi/lib/libmpi_cxx.so;/usr/lib/openmpi/lib/libmpi.so;gloo;onnxifi_loader;gcc_s;gcc;dl
— Configuring done
— Generating done
— Build files have been written to: /home/asl/work/caffe2-pytorch/build
That’s the problem with GCC not Caffe2 sources. You were compiling with GCC 5.5 and may need to consider downgrading to GCC 5.4 in order to build successfully. This is a known issue that is also faced by other frameworks. You can refer to similar problem for TensorFlow discussed here: https://github.com/tensorflow/tensorflow/issues/10220
Hi Mikael, I’ve been following your steps. Though, during the Step 5, I got the following errors and couldn’t make it successful – any advice or suggestions?
[ 95%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/expand_squeeze_dims_op_gpu.cc.o
[ 95%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/exp_op_gpu.cc.o
[ 95%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_op_cudnn.cc.o
[ 95%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_network_blob_fetcher_op_gpu.cc.o
[ 95%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_network_executor_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/queue/queue_ops_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/sgd/iter_op_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/sgd/learning_rate_op_gpu.cc.o
[ 96%] Linking CXX shared library ../lib/libcaffe2_gpu.so
/usr/local/cuda/lib64/libcudnn.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
caffe2/CMakeFiles/caffe2_gpu.dir/build.make:5977: recipe for target ‘lib/libcaffe2_gpu.so’ failed
make[2]: *** [lib/libcaffe2_gpu.so] Error 1
CMakeFiles/Makefile2:1562: recipe for target ‘caffe2/CMakeFiles/caffe2_gpu.dir/all’ failed
make[1]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/all] Error 2
Makefile:138: recipe for target ‘all’ failed
make: *** [all] Error 2
Hi Bryan,
Could you provide the info about your build environment:
– OS version
– GCC version
– Python version
– CUDA version
– cuDNN version
– the output of the following commands:
$ echo $LD_LIBRARY_PATH $ echo $PYTHONPATH– and the output of cmake for CUDA?
Did you also try to build from the latest Caffe2 source?
Hi Mikael,
Here is the info you requested – let me know if you need anything else
1. OS: Ubuntu 16.04
2. GCC version 5.4.0 20160609
3. Python versions: 2,7 and 3.5
4. CUDA 9.0.176
5. CuDNN 7.1.4
6. echo $LD_LIBRARY_PATH: /usr/local/cuda-9.0/lib64:/usr/local/cuda/extras/CUPTI/lib64
7. echo $PYTHONPATH: (null)
8. Regarding the latest Caffe2 source, can you explain? I’ve just followed the step 4
9. Output of cmake for CUDA
— ******** Summary ********
— CMake version : 3.5.1
— CMake command : /usr/bin/cmake
— System : Linux
— C++ compiler : /usr/bin/c++
— C++ compiler version : 5.4.0
— CXX flags : -fvisibility-inlines-hidden
— Build type : Release
— Compile definitions :
— CMAKE_PREFIX_PATH :
— CMAKE_INSTALL_PREFIX : /usr/local
— CMAKE_MODULE_PATH : /home/bryan/caffe2-pytorch/cmake/Modules;/home/bryan/caffe2-pytorch/cmake/public/../Modules_CUDA_fix
—
— ONNX version : 1.3.0
— ONNX NAMESPACE : onnx_c2
— ONNX_BUILD_TESTS : OFF
— ONNX_BUILD_BENCHMARKS : OFF
— ONNX_USE_LITE_PROTO : OFF
— ONNXIFI_DUMMY_BACKEND : OFF
—
— Protobuf compiler :
— Protobuf includes :
— Protobuf libraries :
— BUILD_ONNX_PYTHON : OFF
— Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
— Removing -DNDEBUG from compile flags
— Compiling with OpenMP support
— MAGMA not found. Compiling without MAGMA support
— Could not find hardware support for NEON on this machine.
— No OMAP3 processor on this machine.
— No OMAP4 processor on this machine.
— Looking for cpuid.h
— Looking for cpuid.h – found
— Performing Test HAVE_GCC_GET_CPUID
— Performing Test HAVE_GCC_GET_CPUID – Success
— Performing Test NO_GCC_EBX_FPIC_BUG
— Performing Test NO_GCC_EBX_FPIC_BUG – Success
— Performing Test C_HAS_AVX_1
— Performing Test C_HAS_AVX_1 – Failed
— Performing Test C_HAS_AVX_2
— Performing Test C_HAS_AVX_2 – Success
— Performing Test C_HAS_AVX2_1
— Performing Test C_HAS_AVX2_1 – Failed
— Performing Test C_HAS_AVX2_2
— Performing Test C_HAS_AVX2_2 – Success
— Performing Test CXX_HAS_AVX_1
— Performing Test CXX_HAS_AVX_1 – Failed
— Performing Test CXX_HAS_AVX_2
— Performing Test CXX_HAS_AVX_2 – Success
— Performing Test CXX_HAS_AVX2_1
— Performing Test CXX_HAS_AVX2_1 – Failed
— Performing Test CXX_HAS_AVX2_2
— Performing Test CXX_HAS_AVX2_2 – Success
— AVX compiler support found
— AVX2 compiler support found
— Performing Test HAS_C11_ATOMICS
— Performing Test HAS_C11_ATOMICS – Failed
— Performing Test HAS_MSC_ATOMICS
— Performing Test HAS_MSC_ATOMICS – Failed
— Performing Test HAS_GCC_ATOMICS
— Performing Test HAS_GCC_ATOMICS – Success
— Atomics: using GCC intrinsics
— Checking for [mkl_intel_lp64 – mkl_gnu_thread – mkl_core – gomp – pthread – m – dl]
— Library mkl_intel_lp64: /home/bryan/anaconda3/lib/libmkl_intel_lp64.so
— Library mkl_gnu_thread: /home/bryan/anaconda3/lib/libmkl_gnu_thread.so
— Library mkl_core: /home/bryan/anaconda3/lib/libmkl_core.so
— Library gomp: -fopenmp
— Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so
— Library m: /usr/lib/x86_64-linux-gnu/libm.so
— Library dl: /usr/lib/x86_64-linux-gnu/libdl.so
— MKL library not found
— Checking for [Accelerate]
— Library Accelerate: BLAS_Accelerate_LIBRARY-NOTFOUND
— Checking for [vecLib]
— Library vecLib: BLAS_vecLib_LIBRARY-NOTFOUND
— Checking for [openblas]
— Library openblas: BLAS_openblas_LIBRARY-NOTFOUND
— Checking for [openblas – pthread]
— Library openblas: BLAS_openblas_LIBRARY-NOTFOUND
— Checking for [goto2 – gfortran]
— Library goto2: BLAS_goto2_LIBRARY-NOTFOUND
— Checking for [goto2 – gfortran – pthread]
— Library goto2: BLAS_goto2_LIBRARY-NOTFOUND
— Checking for [acml – gfortran]
— Library acml: BLAS_acml_LIBRARY-NOTFOUND
— Checking for [ptf77blas – atlas – gfortran]
— Library ptf77blas: BLAS_ptf77blas_LIBRARY-NOTFOUND
— Checking for [blas]
— Library blas: BLAS_blas_LIBRARY-NOTFOUND
— Cannot find a library with BLAS API. Not using BLAS.
— LAPACK requires BLAS
— Cannot find a library with LAPACK API. Not using LAPACK.
— Found CUDA: /usr/local/cuda (found suitable version “9.0”, minimum required is “5.5”)
disabling ROCM because NOT USE_ROCM is set
— MIOpen not found. Compiling without MIOpen support
disabling MKLDNN because USE_MKLDNN is not set
— Looking for clock_gettime in rt
— Looking for clock_gettime in rt – found
— Looking for mmap
— Looking for mmap – found
— Looking for shm_open
— Looking for shm_open – found
— Looking for shm_unlink
— Looking for shm_unlink – found
— Looking for malloc_usable_size
— Looking for malloc_usable_size – found
— Performing Test C_HAS_THREAD
— Performing Test C_HAS_THREAD – Success
— GCC 5.4.0: Adding gcc and gcc_s libs to link line
— Check size of long double
— Check size of long double – done
— Performing Test COMPILER_SUPPORTS_LONG_DOUBLE
— Performing Test COMPILER_SUPPORTS_LONG_DOUBLE – Success
— Performing Test COMPILER_SUPPORTS_FLOAT128
— Performing Test COMPILER_SUPPORTS_FLOAT128 – Success
— Performing Test COMPILER_SUPPORTS_SSE2
— Performing Test COMPILER_SUPPORTS_SSE2 – Success
— Performing Test COMPILER_SUPPORTS_SSE4
— Performing Test COMPILER_SUPPORTS_SSE4 – Success
— Performing Test COMPILER_SUPPORTS_AVX
— Performing Test COMPILER_SUPPORTS_AVX – Success
— Performing Test COMPILER_SUPPORTS_FMA4
— Performing Test COMPILER_SUPPORTS_FMA4 – Success
— Performing Test COMPILER_SUPPORTS_AVX2
— Performing Test COMPILER_SUPPORTS_AVX2 – Success
— Performing Test COMPILER_SUPPORTS_SVE
— Performing Test COMPILER_SUPPORTS_SVE – Failed
— Performing Test COMPILER_SUPPORTS_AVX512F
— Performing Test COMPILER_SUPPORTS_AVX512F – Failed
— Performing Test COMPILER_SUPPORTS_OPENMP
— Performing Test COMPILER_SUPPORTS_OPENMP – Success
— Performing Test COMPILER_SUPPORTS_WEAK_ALIASES
— Performing Test COMPILER_SUPPORTS_WEAK_ALIASES – Success
— Performing Test COMPILER_SUPPORTS_BUILTIN_MATH
— Performing Test COMPILER_SUPPORTS_BUILTIN_MATH – Success
— Configuring build for SLEEF-v3.2
Target system: Linux-4.4.0-91-generic
Target processor: x86_64
Host system: Linux-4.4.0-91-generic
Host processor: x86_64
Detected C compiler: GNU @ /usr/bin/cc
— Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
— Building shared libs : OFF
— MPFR : /home/bryan/anaconda3/lib/libmpfr.so
— MPFR header file in /home/bryan/anaconda3/include
— GMP : /home/bryan/anaconda3/lib/libgmp.so
— RUNNING_ON_TRAVIS : 0
— COMPILER_SUPPORTS_OPENMP : 1
— Include NCCL operators
— Excluding ideep operators as we are not using ideep
— Including image processing operators
— Excluding video processing operators due to no opencv
— Include Observer library
— Using lib/python3.6/site-packages as python relative installation path
— Automatically generating missing __init__.py files.
CMake Warning at CMakeLists.txt:387 (message):
Generated cmake files are only fully tested if one builds with system glog,
gflags, and protobuf. Other settings may generate files that are not well
tested.
—
— ******** Summary ********
— General:
— CMake version : 3.5.1
— CMake command : /usr/bin/cmake
— System : Linux
— C++ compiler : /usr/bin/c++
— C++ compiler version : 5.4.0
— BLAS : MKL
— CXX flags : -fvisibility-inlines-hidden -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
— Build type : Release
— Compile definitions : ONNX_NAMESPACE=onnx_c2;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
— CMAKE_PREFIX_PATH :
— CMAKE_INSTALL_PREFIX : /usr/local
—
— TORCH_VERSION : 1.0.0
— CAFFE2_VERSION : 1.0.0
— BUILD_ATEN_MOBILE : OFF
— BUILD_ATEN_ONLY : OFF
— BUILD_BINARY : OFF
— BUILD_CUSTOM_PROTOBUF : ON
— Link local protobuf : ON
— BUILD_DOCS : OFF
— BUILD_PYTHON : ON
— Python version : 3.6.2
— Python executable : /home/bryan/anaconda3/bin/python
— Pythonlibs version : 3.6.2
— Python library : /home/bryan/anaconda3/lib/python3.6
— Python includes : /home/bryan/anaconda3/include/python3.6m
— Python site-packages: lib/python3.6/site-packages
— BUILD_CAFFE2_OPS : ON
— BUILD_SHARED_LIBS : ON
— BUILD_TEST : OFF
— USE_ASAN : OFF
— USE_CUDA : ON
— CUDA static link : OFF
— USE_CUDNN : ON
— CUDA version : 9.0
— cuDNN version : 7.1.2
— CUDA root directory : /usr/local/cuda
— CUDA library : /usr/local/cuda-9.0/lib64/stubs/libcuda.so
— cudart library : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
— cublas library : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
— cufft library : /usr/local/cuda/lib64/libcufft.so
— curand library : /usr/local/cuda/lib64/libcurand.so
— cuDNN library : /usr/local/cuda/lib64/libcudnn.so
— nvrtc : /usr/local/cuda-9.0/lib64/libnvrtc.so
— CUDA include path : /usr/local/cuda/include
— NVCC executable : /usr/local/cuda/bin/nvcc
— CUDA host compiler : /usr/bin/cc
— USE_TENSORRT : OFF
— USE_ROCM : OFF
— USE_EIGEN_FOR_BLAS : ON
— USE_FFMPEG : OFF
— USE_GFLAGS : ON
— USE_GLOG : ON
— USE_LEVELDB : ON
— LevelDB version : 1.18
— Snappy version : 1.1.3
— USE_LITE_PROTO : OFF
— USE_LMDB : ON
— LMDB version : 0.9.17
— USE_METAL : OFF
— USE_MKL : OFF
— USE_MKLDNN : OFF
— USE_MOBILE_OPENGL : OFF
— USE_NCCL : ON
— USE_SYSTEM_NCCL : OFF
— USE_NNPACK : ON
— USE_NUMPY : ON
— USE_OBSERVERS : ON
— USE_OPENCL : OFF
— USE_OPENCV : ON
— OpenCV version : 2.4.9.1
— USE_OPENMP : OFF
— USE_PROF : OFF
— USE_QNNPACK : ON
— USE_REDIS : OFF
— USE_ROCKSDB : OFF
— USE_ZMQ : OFF
— USE_DISTRIBUTED : ON
— USE_MPI : ON
— USE_GLOO : ON
— USE_GLOO_IBVERBS : OFF
— Public Dependencies : Threads::Threads;glog::glog
— Private Dependencies : qnnpack;nnpack;cpuinfo;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/lib/x86_64-linux-gnu/libsnappy.so;/usr/lib/x86_64-linux-gnu/libnuma.so;opencv_core;opencv_highgui;opencv_imgproc;fp16;/usr/lib/openmpi/lib/libmpi_cxx.so;/usr/lib/openmpi/lib/libmpi.so;gloo;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl
— Configuring done
— Generating done
— Build files have been written to: /home/bryan/caffe2-pytorch/build
I updated the cmake version from 3.5.1 to 3.12.0 and retried but again failed for the same errors (as shown below) – any thoughts?
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/sqr_op_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/sqrt_op_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/stop_gradient_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/tensor_protos_db_input_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/while_op_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/zero_gradient_op_gpu.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_op_cudnn.cc.o
[ 96%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_network_blob_fetcher_op_gpu.cc.o
[ 97%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/operators/rnn/recurrent_network_executor_gpu.cc.o
[ 97%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/queue/queue_ops_gpu.cc.o
[ 97%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/sgd/iter_op_gpu.cc.o
[ 97%] Building CXX object caffe2/CMakeFiles/caffe2_gpu.dir/sgd/learning_rate_op_gpu.cc.o
[ 97%] Linking CXX shared library ../lib/libcaffe2_gpu.so
/usr/local/cuda-9.0/lib64/libcudnn.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
caffe2/CMakeFiles/caffe2_gpu.dir/build.make:4756: recipe for target ‘lib/libcaffe2_gpu.so’ failed
make[2]: *** [lib/libcaffe2_gpu.so] Error 1
CMakeFiles/Makefile2:1567: recipe for target ‘caffe2/CMakeFiles/caffe2_gpu.dir/all’ failed
make[1]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/all] Error 2
Makefile:140: recipe for target ‘all’ failed
make: *** [all] Error 2
see similar issues reported at https://github.com/pytorch/pytorch/issues/9699
Is this a compatibility issue? Would you suggest to change from CUDA 9.0 / cuDNN 7.1 possibly to lower versions (e.g., CUDA 8.0 / cuDNN 6.0)?
Hi Bryan,
Thanks for providing the information about your build environment and also the link to the Github issue. Looking at the log, the problem might be caused by non-matching cuDNN library for building Caffe2. As you built with CUDA 9.0, it’s also necessary to find the correct cuDNN version for CUDA 9.0. There can be incompatibility between recent cuDNN and older version of CUDA.
My suggestion is to upgrade to newer version of CUDA, for e.g. CUDA 9.2 (I haven’t tested CUDA 10.0), and install cuDNN that matches CUDA 9.2. When you login to NVIDIA developer portal, you will be able to find cuDNN deb file for CUDA 9.2.
If you want to stick with CUDA 9.0, you need to install compatible cuDNN for CUDA 9.0.
From this error:
/usr/local/cuda-9.0/lib64/libcudnn.so: error adding symbols: File in wrong formatIt hinted cuDNN incompatibility. You may need to check once again if cuDNN 7.1.4 is compatible with CUDA 9.0.
This issue may also give you additional pointer to resolve the issue: https://github.com/caffe2/caffe2/issues/1887