# How to run CASToR in MPI mode?

**URL:** https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603
**Category:** General Discussions
**Created:** [August 8, 2023, 2:25am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603 "2023-08-08T02:25:29Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![JoaoWang](https://avatars.discourse-cdn.com/v4/letter/j/ad7895/32.png) [@JoaoWang](https://castor-project.discourse.group/u/JoaoWang)
#### Post date: [August 8, 2023, 2:25am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/1 "2023-08-08T02:25:29Z")

</div>

Dear,  
I am a newbee of castor.  
I successful build castor-3.1.1 with intel-mpi and intel compiler mpiicpc.  
Also I run the benchmark with the .sh script with single server OpenMP mode.  
My question is: How to run the bencahmark with mpirun command? Is there some one have mpi-based parallel run example or shell sscript can share to me?  
Thanks a lot!

---

<div class="post-metadata">

### Author: ![tmerlin](https://yyz2.discourse-cdn.com/free1/user_avatar/castor-project.discourse.group/tmerlin/32/216_2.png) [@tmerlin](https://castor-project.discourse.group/u/tmerlin)
#### Post date: [August 8, 2023, 3:50pm UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/2 "2023-08-08T15:50:03Z")

</div>

Dear Joao,  
The benchmark script just calls castor-recon and then compares the reconstructed image with the reference one. So you could just add the mpi call commands (mpirun/mpiexec) before castor-recon in order to test it.

Let me know if this works for you.

Best,  
Thibaut

---

<div class="post-metadata">

### Author: ![JoaoWang](https://avatars.discourse-cdn.com/v4/letter/j/ad7895/32.png) [@JoaoWang](https://castor-project.discourse.group/u/JoaoWang)
#### Post date: [August 9, 2023, 4:08am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/3 "2023-08-09T04:08:08Z")

</div>

Hi Tmerlin:

Thanks for your nice reply. I tried add mpirun before castor-recon. The command that I used is :

mpirun -np 4 castor-recon -vb 2 -df benchmark\_pet\_list-mode\_tof.cdh -norm benchmark\_pet\_list-mode\_norm.cdh -fout benchmark\_pet\_list-mode\_challenger -oit -1 -it 2:28 -dim 140,70,63 -fov 560.,280.,252. -off 0.,16.,0. -opti MLEM -proj joseph -conv gaussian,4.,4.,3.5::psf -conv gaussian,6.,6.,3.5::post -th 0 -slice-out 3 -flip-out Y

And I encountered an error as shown below:

→ Image dimensions: [140;70;63] voxels of [Abort(201967372) on node 3 (rank 3 in comm 0): Fatal error in PMPI\_Recv: Invalid argument, error stack:

PMPI\_Recv(173): MPI\_Recv(buf=0x1975638, count=1, MPI\_LONG, src=0, tag=0, MPI\_COMM\_WORLD, status=(nil)) failed

PMPI\_Recv(84).: Null pointer in parameter status

I am very eager for your help!

Thanks!

从 Windows 版[邮件](https://go.microsoft.com/fwlink/?LinkId=550986)发送

 ![33D8A4857EC341E68DE51A1DA9D846C2.png](https://global.discourse-cdn.com/free1/uploads/castor_project/original/1X/bed450d881e998a1f075e7451b3ed2db76895f69.png)

---

<div class="post-metadata">

### Author: ![tmerlin](https://yyz2.discourse-cdn.com/free1/user_avatar/castor-project.discourse.group/tmerlin/32/216_2.png) [@tmerlin](https://castor-project.discourse.group/u/tmerlin)
#### Post date: [August 9, 2023, 10:24am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/4 "2023-08-09T10:24:31Z")

</div>

Hi,  
Looks like primary an Intel MPI error, so I am really not sure what is the origin of the issue. Do you also get an error when setting -np to 1 just for testing ?

Best,  
Thibaut

---

<div class="post-metadata">

### Author: ![JoaoWang](https://avatars.discourse-cdn.com/v4/letter/j/ad7895/32.png) [@JoaoWang](https://castor-project.discourse.group/u/JoaoWang)
#### Post date: [August 11, 2023, 3:24am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/5 "2023-08-11T03:24:04Z")

</div>

Hi Tmerlin:

Thinks for your nice reply!

I’ve tried ‘mpirun -np 1 castor-recon’, it can run normally. But when I changed np to another integer (such as 4), the program reported errors.

I look forward to your further help.

Thanks!

从 Windows 版[邮件](https://go.microsoft.com/fwlink/?LinkId=550986)发送

 ![33D8A4857EC341E68DE51A1DA9D846C2.png](https://global.discourse-cdn.com/free1/uploads/castor_project/original/1X/bed450d881e998a1f075e7451b3ed2db76895f69.png)

---

<div class="post-metadata">

### Author: ![tmerlin](https://yyz2.discourse-cdn.com/free1/user_avatar/castor-project.discourse.group/tmerlin/32/216_2.png) [@tmerlin](https://castor-project.discourse.group/u/tmerlin)
#### Post date: [August 11, 2023, 8:34pm UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/6 "2023-08-11T20:34:57Z")

</div>

Hi,  
The error occurs early in the initialization phases according to your log, and MPI is not used at this point. Does the error message always appear at this stage when using different np values ?  
If not done already I would check first if CASToR has been correctly compiled (if you type _castor-recon -help-comp_, the output should state “Compiled with MPI”).

Best,  
Thibaut

---

<div class="post-metadata">

### Author: ![miguelleaolopes](https://avatars.discourse-cdn.com/v4/letter/m/85e7bf/32.png) [@miguelleaolopes](https://castor-project.discourse.group/u/miguelleaolopes)
#### Post date: [August 12, 2023, 11:27am UTC](https://castor-project.discourse.group/t/how-to-run-castor-in-mpi-mode/603/7 "2023-08-12T11:27:46Z")

</div>

Hi Joao,

Curiously I had a simular problem just a few days ago, with the following error:

```auto
oImageDimensionsAndQuantification::Initialize() -> Initialize image dimensions, basis functions and quantification
  --> Image dimensions: [100;100;50] voxels of [4.0000001e-01;4.0000001e-01;4.0000001e-01] mm3
  --> FOV size: [4.0000000e+01;4.0000000e+01;2.0000000e+01] mm3
  --> Number of parallel threads: 4
sRandomNumberGenerator::Seed for rank 0 is 2213572483
sRandomNumberGenerator::Seed for rank 1 is 2182813001
sRandomNumberGenerator::Seed for rank 2 is 3037910000
sRandomNumberGenerator::Seed for rank 3 is 4079532605

job aborted:
[ranks] message

[0] terminated

[1] fatal error
Fatal error in MPI_Recv: Invalid argument, error stack:
MPI_Recv(buf=0x00000207FBA98778, count=1, MPI_LONG, src=0, tag=0, MPI_COMM_WORLD, status=0x0000000000000000) failed
Null pointer in parameter status

[2] fatal error
Fatal error in MPI_Recv: Invalid argument, error stack:
MPI_Recv(buf=0x000001AC57BE80B8, count=1, MPI_LONG, src=0, tag=0, MPI_COMM_WORLD, status=0x0000000000000000) failed
Null pointer in parameter status

[3] fatal error
Fatal error in MPI_Recv: Invalid argument, error stack:
MPI_Recv(buf=0x000001C9C7608A98, count=1, MPI_LONG, src=0, tag=0, MPI_COMM_WORLD, status=0x0000000000000000) failed
Null pointer in parameter status

```

And I tried to debug with **cout** the _castor-recon.cc_ file, to see where the MPI\_Recv error was coming from.  
It appeared it was from the initialization of the random number generator, code snipet:

```auto
  // ----------------------------------------------------------------------------------------
  // Random Number Generator initialization: (we first require to know the number of threads to use from p_ImageDimensionsAndQuantification)
  // ----------------------------------------------------------------------------------------
  cout << "Test 1: mpi_rank = " << mpi_rank << endl; 
  if (verbose_general>=5) Cout("----- Random number generator initialization ... -----" << endl);
  
  sRandomNumberGenerator* p_RandomNumberGenerator = sRandomNumberGenerator::GetInstance(); 
  p_RandomNumberGenerator->SetVerbose(verbose_general);
  // Use a user-provided seed to initialize the RNG if one has been provided. Use random number otherwise.
  cout << "Test 2: mpi_rank = " << mpi_rank << endl;
  if (random_generator_seed>=0) p_RandomNumberGenerator->Initialize(random_generator_seed, p_ImageDimensionsAndQuantification->GetNbThreadsMax(), nb_extra_random_generators);
  else p_RandomNumberGenerator->Initialize(p_ImageDimensionsAndQuantification->GetNbThreadsMax(), nb_extra_random_generators);
  cout << "Test 3: mpi_rank = " << mpi_rank << endl;

  if (verbose_general >=5) Cout("----- Random number generator initialization OK -----" << endl);

```

Where it only printed the test 1 and 2, thus the error coming from the _Initialize_ function of _p\_RandomNumberGenerator_ (since _random\_generator\_seed = -1_).

In here, i found that the error is coming from this part of the code:

```auto
// if more than one MPI instance, generate seeds for each instance 
  // and dispatch them, otherwise keep the initial seed
  if (mpi_size_temp>1)
  {
    if (mpi_rank_temp==0)
    {
      Engine mpi_generator(m_seed);
      for (int p=0; p<mpi_size_temp; p++) 
      {
        m_seed = mpi_generator();

         Cout("sRandomNumberGenerator::Seed for rank " << p << " is " << m_seed << endl); 

        if (p==0)
          temp_seed = m_seed;
        else
          MPI_Send(&m_seed, 1, MPI_LONG, p, 0, MPI_COMM_WORLD);
      }
      m_seed = temp_seed;
    }
    else
    {
      MPI_Status *status = NULL;
      MPI_Recv(&m_seed, 1, MPI_LONG, 0, 0, MPI_COMM_WORLD, status);
    }
    // wait for all the processes to have their seeds
    MPI_Barrier(MPI_COMM_WORLD);
  }

```

in particular, when there is more than 1 process, the mpi\_rank\_temp is 1,2,3, entering the else:

```
  MPI_Status *status = NULL;
  MPI_Recv(&m_seed, 1, MPI_LONG, 0, 0, MPI_COMM_WORLD, status);

```

where it gives the error of the Null pointer in parameter status. I’ve tried to change to:

```
  MPI_Status status;
  MPI_Recv(&m_seed, 1, MPI_LONG, 0, 0, MPI_COMM_WORLD, &status);

```

and now it works fine!

* * *

Don’t know it your error is in the same place, since is not exctaly the same, but i hope it helps!

Best,  
Miguel