Multi-thread Run Problem

General Discussions
1

Dear all,

I am testing CASToR performance on multi-thread running, but it looks like using the multi-thread function is not very stable.
Before applying multi-thread to recon, every job was good. But recon jobs always dumped with unrecognized problem since I used multi-thread as below:

*** Error in `castor-recon': munmap_chunk(): invalid pointer: 0x0000000002259380 ***

Did anyone ever see this problem before?
Any answer will be highly appreciated! Thank you!

Best,

2

Hi Xinjie,

On which kind of dataset did you get this problem ? Did it occur on every data you tried to reconstruct or just some of them ?

Best,
Thibaut

Xinjie Cao <xinjie.cao@stonybrook.edu> a écrit :

3

Hi,

I already got strange results like that when I switch between different compilation. The reason was that sometimes I forgot to clean the compilation. Did you carefully use “make clean” command between each compilation ?

Simon

4

Hi Thibaut and Simon,

I have compiled two different Castor versions for multi-thread running and single core in separate directories, respectively. The multi-thread is running well now, but single core test had a problem when I move from a lower resolution configuration to a higher one like below,

First recon try:

castor-recon -df Brain_Double_df.Cdh -opti MLEM -it 10:16 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 100,100,16 -vox 2.5,2.5,10. -oit -1 -dout Brain_Double

.........
iIterativeAlgorithm::StepAfterSubsetLoop() -> Save image at iteration 10
vAlgorithm::IterateCPU() -> Total time spent | User: 3699 sec | CPU: 3.6990100e+03 sec
sChronoManager::Display() -> Results from the profiling
.........
--> Custom update step 1: 00 hours 00 mins 00 secs 000 ms

Second recon try:

castor-recon -df Brain_Double_df.Cdh -opti MLEM -it 2:10 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 200,200,16 -vox 1.25,1.25,10. -oit -1 -dout Brain_Double_2

*** Break *** segmentation violation
5

Hi Xinjie,

Does the crash occur directly after you launch the the reconstruction. Could you run it with -vb 3 to have a bit more feedback ?

There is no reason why you couldn’t change reconstruction parameters between two consecutive reconstructions.

Maybe you could send us/upload your dataset & geometry file so we have a closer look at the problem ?

Best regards,

6

Hi tmerlin,

Here is my dataset, geometry and entire error log. Hope you could find something!
Brain_Double_df.Cdf
Brain_Double_df.Cdh

Best,

Error-Log (43.7 KB)

Brain_Double.geom (1.84 KB)

7

Hi Xinjie,

Thanks for your data. I could reproduce your error and it seems it is linked to the Joseph projector. In particular, there is a hard-coded tolerance factor used to compute the planes which are crossed by the lines. Decreasing that factor seems to remove the crash in my tests.

So a quick fix may be to reduce that threshold (you can replace the src/projector/iProjectorJoseph.cc file by the one in attached, then recompile castor).

I am not sure yet whether the crash is limited to that factor. Let me know if it fixes the error for you.

Best,
Thibaut

iProjectorJoseph.cc (62.8 KB)

8

Hi Tmerlin,

It looks like there is no error now, i will let you know if the error appears again. Thanks!

Best,

9

Dear Virtuall,

Could you please share again your dataset (Brain_Double_df.Cdf and Brain_Double_df.Cdh). I want to check out my installed CASToR using Multi-thread.
Also, Could you please describe how to create this dataset. I assume that, you converted a dataset from a specific scanner (interfile) to CASToR interfile format. Thank you very much in advance.