Cleaning data

[kean]

Hi Folks,

I am hoping some of the quant gurus can advise their data cleaning process....really want to know whether only using SQL to rearrange the data...or using SAS, SPlus, R or do it directly from the database management system.
I am using Splus to clean data. Please advise.

Thanks,
K

 

[Uncle Max]

I'm not a quant guru just yet but will reply anyway

For my last project I used SAS to clean the data. There is no particular reason that I've chosen SAS. However, it worked great for me. I had a database of about 200,000 loans. There were a lot of missing data points and I had to fill them with something for the further analysis. SAS worked extremely fast with this large data set.

Cleaning data is rather creative process. SAS helped me to minimize technical complications and concentrate more on data itself.

 

[kean]

Thanks

I find it R and SPlus are quite good. R is free. I used SAS before at university but I don't have the license at home therefore I choose R.

Also, R allows interface with C++ which is quite ok. I tested.

Cheers,
K

I'm not a quant guru just yet but will reply anyway

For my last project I used SAS to clean the data. There is no particular reason that I've chosen SAS. However, it worked great for me. I had a database of about 200,000 loans. There were a lot of missing data points and I had to fill them with something for the further analysis. SAS worked extremely fast with this large data set.

Cleaning data is rather creative process. SAS helped me to minimize technical complications and concentrate more on data itself.

 

[John]

I find it R and SPlus are quite good. R is free. I used SAS before at university but I don't have the license at home therefore I choose R.

Also, R allows interface with C++ which is quite ok. I tested.

Cheers,
K

I also love the R/S-Plus platform, although it's been a while since I have worked with it. One simple way of working with data is convert the data into a data matrix, and use apply() operator in combination with functions such mean, min, max, as is.na, etc. to generate summary statistics. Once this is done you can proceed to convert the N/A, missing data into whatever you need. Send me a PM.