How do you process ras data from auryn simulations?

[asinha]

I'm using a double, which is what AurynDouble comes to. That paste was probably from an older commit

[zenke]

Just wanted to look up asinha solution again and saw the link has changed. It's now here
http://ankursinha.in/blog/2015/02/20/ex ... files.html

[zenke]

Hi there,

I am in the process of implementing a few changes to the BinarySpikeMonitor and wanted to give you a heads up, because I know you are using it. Nothing dramatic, but thinking a bit ahead for new formats etc ... First, I will change the format of SpikeEvent_type. It will probably the following:

Code: Select all

struct SpikeEvent_type
{
    AurynTime time;
    NeuronID neuronID;
};

The reason for using an int type instead of double is mostly performance and uniqueness (no rounding problems on different architectures at the output level). Moreover, since the time step can be changed in a simulation, I am also including a header field now where it is specified. Specifically, the first frame (struct) of the file will encode the dt and some header information such as the current software version and file type information etc. These information will fit in the first 16bytes (so essentially the first frame). The first field (the time field) contains 1.0/dt (i.e. the number of timesteps in one second). The second field contains a numeric constant or tag in the following format

Code: Select all

const NeuronID tag_binary_spike_monitor = 287960000+100*AURYNVERSION+10*AURYNSUBVERSION+1*AURYNREVISION;

Checking for these values will allow it to distinguish a binary ras file easily from different files. I am currently testing this, but you can expect it to be merged soonish into a feature or the develop branch. If you have some new ideas from using the setup I am all ears. I haven't implemented the merging part yet, so if you have anything very efficient there I'd be happy to hear about it.

[zenke]

Me again. Added merging. Can be neater, but for now I am reasonably happy with this:
https://github.com/fzenke/auryn/commit/ ... 72af2524f0

[asinha]

Hiya,

Thanks for the update. I haven't had a look at the new code yet, but I'll do so sometime this week, since I'll need to update my postprocessing programs to handle the header and things. I've put up the stuff I use here:
https://github.com/sanjayankur31/herts- ... master/cpp

I don't think it can be optimised much - especially the merge. The only other optimisation is to use memory mapped files instead of file streams, like I do. Using them with threading makes it incredibly quick, especially when your simulations are long. For example, my spike ras files were sometimes 100gigs, and if I don't use memory mapped files, I'll have to read 16 x 100gig files into RAM which itself will take a long long time, even in systems with large RAMs. Memory mapped files speed this up a lot since only the exact locations are read to memory, which depending on how much data you want to extract will generally be quite small compared to the entire simulation.

[asinha]

Just wanted to look up asinha solution again and saw the link has changed. It's now here
http://ankursinha.in/blog/2015/02/20/ex ... files.html

Yikes, sorry about that - my shared hosting couldn't handle wordpress and all that so I combined my blogs and threw everything into a static site generator

[zenke]

Hey,

I guess you are right, the memory mapped file solution might be a bit faster. However, ifstream does not load the entire file into memory and is pretty fast as well. Multi-threading however would be a nice addition to the code.

Out of interest and for comparison, I just tested my code. On a single core (Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz) and using a single 387MB file with spikes from 250 Poisson neurons (@5Hz) for 28800s I get the following timings. Most of the time is actually spent formatting the output spikes. If you do statistics directly in your program (e.g. compute firing rates) this will get faster, but I am adhering to the a modular design strategy for flexibility. I use command line scripts like the ones here to process the ras data (https://github.com/fzenke/malleable/tree/master/bin).

Results:
all spikes: 38s
last 3600s: 4.5s
7200s < t < 10800s: 4.5s
10800s < t < 10810s: <0.015s

and when merging 4x387MB files:

Results:
all spikes: 1m51s
last 3600s: 14.7s
7200s < t < 10800s: 13.7s
10800s < t < 10810s: <0.04s

Code:
I used sim_poisson from the examples in which I replaced the SpikeMonitor by a BinarySpikeMonitor and changed the output file extension accordingly.
single file:

Code: Select all

time aube -i poisson.0.bras --last 3600 > /dev/null
time aube -i poisson.0.bras --last 3600 > /dev/null
time aube -i poisson.0.bras --from 7200 --to 10800 > /dev/null
time aube -i poisson.0.bras --from 10800 --to 10810 > /dev/null

merging four files:

Code: Select all

time aube -i poisson.?.bras --last 3600 > /dev/null
time aube -i poisson.?.bras --last 3600 > /dev/null
time aube -i poisson.?.bras --from 7200 --to 10800 > /dev/null
time aube -i poisson.?.bras --from 10800 --to 10810 > /dev/null

[asinha]

Aha, yea, I double checked. ifstream with seek doesn't read the entire file to memory. In most of our cases, the speed limitation is caused by I/O - either reading or writing - generally writing in my case. The processing part is generally quick enough. Whenever I get down to upgrading my postprocessing part, I'll try and update the aube code to use memory mapped files with threads as well. It probably won't be this week, though. Hopefully next week.

I have a bunch of scripts too, but they're not well commented or very up to date at the moment. They're all here: https://github.com/sanjayankur31/herts- ... er/scripts

These include scripts to generate graphs, automate simulations with different parameters and so on. Pretty simple stuff, but saves lots of time. Maybe when we have time we could put up a few helper scripts on the wiki and things.

[zenke]

Cool. I also put up some example scripts in the malleable repository. Just to demonstrate that the analysis tool chain is quite "malleable". I would not worry about the memory mapped files for aube now unless you know for sure that the performance increase will be substantial. However, if you get around to making your memory mapped files into a neat code example to compute firing rates we could add this as an alternative tool "arate" or whatever to illustrates their use and add it to the next release version. As for your code, just make sure you note the change of the data frame struct before you pull from develop. It which now uses the first frame to encode which dt to use and all subsequent frames for the actual spikes