R buckling under stress

[atreides]

I found a .csv dataset that is fairly clean, and had some trouble loading it into R. The dataset has about half a million observations and 10 variables.

Initially it took about 5 minutes to get the data in R, after which I could run some stats on the data... On a second try I cleared my environment went back and changed a few options in R and it took about 40 seconds to get the data in.

Since R is widely used in finance/quant circles, I thought half a million lines / 60MB of data would be a cakewalk.

Maybe practitioners who use R professionally can comment here... I would assume you play around with tons of data.

 

[alain]

one drawback of R at the moment is that it loads all the data into memory. It means you need to have a lot of memory at your disposal. There are solutions to deal with this issue.

What do you mean by a "cakewalk"? loading the data? manipulating the data? I'm sure after the data is loaded, everything will be fairly fast.