using mpi - merging rate monitor outputs from ranks

[asinha]

Hi,

I'm using MPI to speed up my simulations. Now, I use the RateMonitor to get the firing rates of neurons from a group into files for analysis like this:

Code: Select all

RateMonitor * rmon_e = new RateMonitor( neurons_e , strbuf.c_str(), rate_mon_sampling_interval );

This gives me individual files from each rank, as is expected with fields:

Code: Select all

time nrn[i] nrn[i+1] .... nrn[i+n]

Until now, I've just been merging the ratemonitor files from different ranks column wise after removing the time stamp, assuming that rank 1 gets neurons [0,n], rank 2 gets [n+1,2n] and so on. Then, I pick the time I need, "wrap" the line into the required rows and cols and plot it.

This doesn't make a difference when you're working with uniform configurations, such as vogels fig 4A and B. However, now that I'm working on 4C, I'm beginning to wonder if my assumption is correct. I used pattern files as you'd suggested in the other post, but the final figure that I come up with doesn't match. I ran a quick test and indeed, my assumption is wrong.

Is there a way to find what neurons a rank is dealing with, or alternatively, is there an easier way to generate the snapshots?

Thanks,
Ankur

[zenke]

In Auryn, neurons are generally distributed in Round Robin fashion over all or a subset of nodes. This is done to avoid "hot spots" (e.g. all neurons of an active pattern being simulated on one node and slowing down the simulation. This makes the PopulationRateMonitor only useful to monitor the rate of the entire population (btw to merge the rates from different files correctly you should compute the mean over all the merged columns).

For what you would like to do, have a look at the PatternMonitor (http://www.fzenke.net/auryn/doku.php?id ... ernmonitor). For instance

Code: Select all

sprintf(strbuf, "%s/%s.%d.pact", dir.c_str(), file_prefix.c_str(), world.rank() );
PatternMonitor * patmon = new PatternMonitor( neurons_e, string(strbuf) , "patterns.pat");

will take your "pattern.pat" file and treat these patterns as subpopulations of which it will output the population firing rates. First column of the output file contains the time stamp. The second column contains the subpopulation firing rate of the first pattern (what you want for the graphical output), the third contains the fraction of active units from that pattern during the sampling interval (again for the first pattern). The fourth column contains the subpopulation firing rate of the second pattern and so forth. The last column counts the number of total units active normalized by the number of neurons.

This approach outlined above will give you firing rates averaged over all neurons in a pattern. However, if you want to create an activity plot similar to the one shown in Fig 4 (Vogels et al. 2011), you will need to dump all the spikes using a SpikeMonitor and analyze the spiking data directly which is what we did in the paper.

[asinha]

In Auryn, neurons are generally distributed in Round Robin fashion over all or a subset of nodes. This is done to avoid "hot spots" (e.g. all neurons of an active pattern being simulated on one node and slowing down the simulation. This makes the PopulationRateMonitor only useful to monitor the rate of the entire population (btw to merge the rates from different files correctly you should compute the mean over all the merged columns).

Ah, hum, so there's no real way of knowing what set of neurons go to which rank. I guess this also means that the distribution of neurons on the ranks will vary with each simulation run too. I'd figured out the mean part though - I used this for the WeightSumMonitors that I used.

For what you would like to do, have a look at the PatternMonitor (http://www.fzenke.net/auryn/doku.php?id ... ernmonitor). For instance

Code: Select all

sprintf(strbuf, "%s/%s.%d.pact", dir.c_str(), file_prefix.c_str(), world.rank() );
PatternMonitor * patmon = new PatternMonitor( neurons_e, string(strbuf) , "patterns.pat");

will take your "pattern.pat" file and treat these patterns as subpopulations of which it will output the population firing rates. First column of the output file contains the time stamp. The second column contains the subpopulation firing rate of the first pattern (what you want for the graphical output), the third contains the fraction of active units from that pattern during the sampling interval (again for the first pattern). The fourth column contains the subpopulation firing rate of the second pattern and so forth. The last column counts the number of total units active normalized by the number of neurons.

This approach outlined above will give you firing rates averaged over all neurons in a pattern.

Ah. I did tinker with PatternMonitor, but not enough to use them for my graphs yet. I'll run some more simulations tomorrow and work on this.

However, if you want to create an activity plot similar to the one shown in Fig 4 (Vogels et al. 2011), you will need to dump all the spikes using a SpikeMonitor and analyze the spiking data directly which is what we did in the paper.

OK. I was hoping I could just use the firing rates provided by RateMonitor. For example, this script was enough until now. It generated an overall graph with average firing rates and the figures 4A and B. It didn't work for C, though. http://fpaste.org/155995/ I'll go look into SpikeMonitor.

Thanks again.
Ankur

[zenke]

Ah, hum, so there's no real way of knowing what set of neurons go to which rank. I guess this also means that the distribution of neurons on the ranks will vary with each simulation run too. I'd figured out the mean part though - I used this for the WeightSumMonitors that I used.

Well mostly it's ROUNDROBIN over all ranks. You can see this in the log files when your SpikingGroups are initialized. The distribution changes whenever you change the number of ranks you run your simulation, but otherwise rank 0 - neuron 0, rank 1 - neuron 1. When you reach the end of your ranks you start over again.

A final word on the issue. You should try to avoid running all the analysis "online" unless you can decrease IO sustantially. For all my puposes it turned out to be optimal to record spikes (because there are fairly few of them) and to analyze these data after the simulation offline. Here you can use a cluster again, but this approach will avoid bloating up your simulation code, will allow it to run nicely load-balanced and if you made a mistake in the analysis you do not need to re-run the simulation.

[asinha]

A final word on the issue. You should try to avoid running all the analysis "online" unless you can decrease IO sustantially. For all my puposes it turned out to be optimal to record spikes (because there are fairly few of them) and to analyze these data after the simulation offline. Here you can use a cluster again, but this approach will avoid bloating up your simulation code, will allow it to run nicely load-balanced and if you made a mistake in the analysis you do not need to re-run the simulation.

Ah, yes - of course. I only run the script on the output files after my simulation has finished running. The script takes arguments so that I can control exactly what graphs I want generated. The script won't work with Spike time output, though - it only uses firing rate information, so I'll have use something else. It isn't too complicated, though, I should be able to have something up in a bit.

Thanks again,