CAEToR issues related MPI

General Discussions
1

I have compiled and installed CASToR software on my computer (OS: Ubuntu 20.04 LTS) using the provided Makefile according to the relevant instructions in Section III of General Documentation. During compilation, I set the values of environment variables CASTOR_MPI and CASTOR_OMP related to MPI to 1, and MPICH 3.3.2 and openMPI 4.1.4 are also installed on my computer. Although I successfully installed CASToR software, when I used the following commands to reconstruct an PET image, I found that the time consumed by castor-recon (installed by myself) was much longer than the castor-recon_ unix64 (binary executable provided by official)。

>> castor-recon -df XXX_df.Cdh -opti MLEM -it 10:16 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 128,128,64 -vox 3.2,3.2,4.3 -dout recon_img_1

>> castor-recon_unix64 -df XXX_df.Cdh -opti MLEM -it 10:16 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 128,128,64 -vox 3.2,3.2,4.3 -dout recon_img_1

For example, for sensitivity generation, castor-recon takes 13659 seconds, while castor-recon_unix64 takes only 5333 seconds.

I’m very suspicious that MPI is not used when executing castor-recon program. Where should I explicitly specify to enable MPI? Or, did I make any mistakes when compiling?

Here are the commands I entered at compile time:

>> export CASTOR_CONFIG=/home/vgate/Software/CASToR_v3.1.1/config

>> export CASTOR_MPI=1

>> export CASTOR_OMP=1

>> export CASTOR_ROOT=1

>> export CASTOR_SIMD=1

>> export CASTOR_VERBOSE=1

>> export CASTOR_DEBUG=1

>> make -j24

I sincerely look forward to your reply！

2

Hi Joao,
Indeed it looks like MPI does not behave properly. Do you also see a difference for the reconstruction part or is it only for the sensitivity image generation ? Additionally do you have dynamic or static data ?
MPI was implemented a while ago and we don’t test it very often, so I will check next week if I get the same issue with the current version, and come back to you.

3

Hi Tmerlin,

When I compiled CASToR with Makefile, I encountered MPI problem, but later I compiled with CMake, everything was fine. So, I guess this problem may be related to the Makefile.

Best Regard!
Joao Wang

1 Like
4

Hi,

The precompiled binaries do not include CASTOR_VERBOSE, CASTOR_DEBUG, CASTOR_SIMD nor CASTOR_MPI.

When you compiled with the Makefile, you enabled CASTOR_VERBOSE, CASTOR_DEBUG and CASTOR_SIMD. You should not enable these variables unless you really need them.

It is likely that the slow behaviour came from DEBUG and VERBOSE.

When you compiled with CMake, did you enable these features while in the CMake menu ?

The probability that the problem came from MPI is close to zero as there is no difference at all with what should happen when using either Makefile or CMake.

That being said, MPI is not enabled in the precompiled binary because its use is quite specific. Do you really need it ? If no, then you should not enable it.

All the best

Simon