Multi-thread Run Problem

Dear all,

I am testing CASToR performance on multi-thread running, but it looks like using the multi-thread function is not very stable.
Before applying multi-thread to recon, every job was good. But recon jobs always dumped with unrecognized problem since I used multi-thread as below:

*** Error in `castor-recon': munmap_chunk(): invalid pointer: 0x0000000002259380 ***

Did anyone ever see this problem before?
Any answer will be highly appreciated! Thank you!

Best,

Hi Xinjie,

On which kind of dataset did you get this problem ? Did it occur on every data you tried to reconstruct or just some of them ?

Best,
Thibaut

Xinjie Cao <xinjie.cao@stonybrook.edu> a écrit :

Hi,

I already got strange results like that when I switch between different compilation. The reason was that sometimes I forgot to clean the compilation. Did you carefully use “make clean” command between each compilation ?

Simon

Hi Thibaut and Simon,

I have compiled two different Castor versions for multi-thread running and single core in separate directories, respectively. The multi-thread is running well now, but single core test had a problem when I move from a lower resolution configuration to a higher one like below,

First recon try:

castor-recon -df Brain_Double_df.Cdh -opti MLEM -it 10:16 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 100,100,16 -vox 2.5,2.5,10. -oit -1 -dout Brain_Double

.........
iIterativeAlgorithm::StepAfterSubsetLoop() -> Save image at iteration 10
vAlgorithm::IterateCPU() -> Total time spent | User: 3699 sec | CPU: 3.6990100e+03 sec
sChronoManager::Display() -> Results from the profiling
.........
--> Custom update step 1: 00 hours 00 mins 00 secs 000 ms

Second recon try:

castor-recon -df Brain_Double_df.Cdh -opti MLEM -it 2:10 -proj joseph -conv gaussian,4.,4.5,3.5::psf -dim 200,200,16 -vox 1.25,1.25,10. -oit -1 -dout Brain_Double_2

*** Break *** segmentation violation

Hi Xinjie,

Does the crash occur directly after you launch the the reconstruction. Could you run it with -vb 3 to have a bit more feedback ?

There is no reason why you couldn’t change reconstruction parameters between two consecutive reconstructions.

Maybe you could send us/upload your dataset & geometry file so we have a closer look at the problem ?

Best regards,

Hi tmerlin,

Here is my dataset, geometry and entire error log. Hope you could find something!
Brain_Double_df.Cdf
Brain_Double_df.Cdh

Best,

Error-Log (43.7 KB)

Brain_Double.geom (1.84 KB)

Hi Xinjie,

Thanks for your data. I could reproduce your error and it seems it is linked to the Joseph projector. In particular, there is a hard-coded tolerance factor used to compute the planes which are crossed by the lines. Decreasing that factor seems to remove the crash in my tests.

So a quick fix may be to reduce that threshold (you can replace the src/projector/iProjectorJoseph.cc file by the one in attached, then recompile castor).

I am not sure yet whether the crash is limited to that factor. Let me know if it fixes the error for you.

Best,
Thibaut

iProjectorJoseph.cc (62.8 KB)

Hi Tmerlin,

It looks like there is no error now, i will let you know if the error appears again. Thanks!

Best,