Neural Network on Bond Rating,Database needed

[YongLim]

Hi everyone,

There are a lot of financial related works had been done by neural network such as time series prediction and credit rating classification.
Currently,I am interesting in develop a neural network which can predict the company bond rating.By doing this, I need a huge database which contained numbers of companies financial statement,such as annual income,return equity,net asset,debt load,sales and etc.After having these data as my training set,I am able to train my network and predict the bond rating of the companies.
Unfortunately I am struggling in obtaining these database now,I hope that someone can point me to the right direction or give me some suggestion regarding finding the database.

Thanks!

 

[alain]

how much are you willing to pay?

 

[YongLim]

how much are you willing to pay?

Hi alain, this is an academic study and non-profitable, so I am looking for some free database for this project.

 

[daleholborow]

Does the academic institution you are studying under have any data systems available? I would guess that most finance schools would subscribe to some data (Bloomberg not as likely as it's expensive, possibly Thomson Datastream, Computstat, stuff like that).

I've found it very hard (read: more or less impossible) to find a good free source of information on the fundamentals of companies. How good are you at programming, in particular web programming? I have a link saved at home to a website that has very comprehensive fundamental data for lots of US companies, and some other internationals. You could write some html-scraping code to pluck the fundamental data and store it in a format that you prefer... thats what I did

 

[alain]

Fixed Income data is closed to impossible to find for free. It might be out there but I haven't seen it. There are couple of threads in QN saying the same thing. Hence, unless you are willing to pay or find somebody who is willing to pay, you are going to be in a rough spot.

If this is for a personal project, good luck. Hopefully you can come back here and post where you get the data.

 

[YongLim]

Does the academic institution you are studying under have any data systems available? I would guess that most finance schools would subscribe to some data (Bloomberg not as likely as it's expensive, possibly Thomson Datastream, Computstat, stuff like that).

I've found it very hard (read: more or less impossible) to find a good free source of information on the fundamentals of companies. How good are you at programming, in particular web programming? I have a link saved at home to a website that has very comprehensive fundamental data for lots of US companies, and some other internationals. You could write some html-scraping code to pluck the fundamental data and store it in a format that you prefer... thats what I did

Hi daleholborow,

Actually,I am not a finance school student but a electrical engineering student. Usually we apply neural network on signal processing or pattern recognition and this is the first time I try to apply on financial data analysis. I am not familiar with web programing but I will be appreciate if you can share your web link to me.

 

[YongLim]

Fixed Income data is closed to impossible to find for free. It might be out there but I haven't seen it. There are couple of threads in QN saying the same thing. Hence, unless you are willing to pay or find somebody who is willing to pay, you are going to be in a rough spot.

If this is for a personal project, good luck. Hopefully you can come back here and post where you get the data.

Hi alain,

Thanks for your information, I had been spent few days on google some free database but I am not able to get some.Maybe this is the reason you mentioned about. Hopefully I am not in the dead end yet, I will try to share if I get some data.

 

[Andy Nguyen]

@daleholborow
Looks like a how-to tutorial to scrap the data and store them in xml would be useful. Up for it?

@YongLim
If you can specify the list of company, type of data column, I may get someone to help you with the data. No promise but you need to make it less work for people who can help.

Did you try this? http://www.quantnet.com/forum/threads/master-list-of-free-financial-data.1036/

 

[daleholborow]

Yeah, I can probably upload some demo code to show people a quick and dirty hack on how to scrape some web-based data. I'll look for my project when I get home. I've been meaning to start up a bit of a blog to post entries for stuff like this, but lack of time and a black belt in procrastination have delayed it

 

[YongLim]

@daleholborow
Looks like a how-to tutorial to scrap the data and store them in xml would be useful. Up for it?

@YongLim
If you can specify the list of company, type of data column, I may get someone to help you with the data. No promise but you need to make it less work for people who can help.

Did you try this? http://www.quantnet.com/forum/threads/master-list-of-free-financial-data.1036/

Hi Andy,

I did try your link,many of them on the list are dead,but some of them are still alive and I am looking on it.
I really appreciate and more than happy if you can help. Actually,I don't have some specific list of company and data type right now.
Generally, the larger of database and representative parameters will lead to a more accurate model.
I will try to have a prepared list,thanks Andy!

 

[alain]

Good luck with the NN thing. It has being tried sometimes in finance and it has failed miserably. Hopefully you can come up with something better.

What kind of neural network are you planning to use?

 

[elektor]

YongLim,
You could try to query the database of the exchange on which the company is listed on. They may have some downloadable data sets of fund. info of a comp.

 

[daleholborow]

Alrighty... with ABSOLUTELY NO WARRANTY, and provided SOLELY AS AN EXAMPLE of how to use C# and HtmlAgilityPack to try to expedite the retrieval of data from a website, I offer the following.

Basically, its some code I hacked together to parse either a sample html file, or a URL, look for a particular table of data based on its attributes (you can retrieve HTML elements by ID, but in this case, the table we wanted didnt HAVE an ID so we get it based on known html properties instead), use the HtmlAgilityPack core functionality to quickly parse the tabulated data, assign all the field elements to our simple plain old container object, and then use .Net 3.5 LINQ-to-XML functionality to quickly save the results in XML format.

Obviously, you can - patience and ability permitting - change this simple demo to target any HTML file / url.... obviously the poco assignment would change, etc etc blah blah. If you don't already know what I mean by now, then this code isn't going to help you anyway.

The attached file is a C# solution. Commenting is minimal, things aren't particularly tidy, but hey... its a quick hack demo, not production-ready

 

[elektor]

dale,

have you experimented with HttpWebRequest to pull data from web pages? I was trying to open the following page using HttpWebRequest but it flunks out.
http://www.nseindia.com/marketinfo/...APR2011&symbol=USDINR&flag=1&exp_date= 270411

 

[daleholborow]

At various times, I've done various web post/retrieve calls for different projects. But nothing recently, and without more info than "flunks out", I couldnt make any useful suggestions. Any details on what the specific error messages are?

 

[Tsotne]

I've currently been given an answer to a similar question. Fixed income securities' quotes are very hard to obtain. So the right direction is to concentrate on one particular type -bonds for exmaple (as you are doing). However it's not guaranteed that you'll get al the desired data. As Bob told me, institutions place a hard work on obtaining and distributing those data on a daily bases so they charge a big amount. Lower maturity securities are a bit easier to search.

 

[YongLim]

Hi everyone,sorry for the inactive for few days. I had been tied for few days on my coursework.

Good luck with the NN thing. It has being tried sometimes in finance and it has failed miserably. Hopefully you can come up with something better.

What kind of neural network are you planning to use?

Hi alain, my plan is to use SOM for the feature extraction and further using RBF. This is my first try using NN on finance,hopefully I can come up something exciting.

 

[YongLim]

YongLim,
You could try to query the database of the exchange on which the company is listed on. They may have some downloadable data sets of fund. info of a comp.

Hi elektor,I know that I can download some data from a particular company. But the NN approach need a huge dataset,collecting the data from one company to another will be very tedious. I am looking for a "package" of numerous company data,for example. 500 companies from S&P 500.

 

[YongLim]

Alrighty... with ABSOLUTELY NO WARRANTY, and provided SOLELY AS AN EXAMPLE of how to use C# and HtmlAgilityPack to try to expedite the retrieval of data from a website, I offer the following.

Basically, its some code I hacked together to parse either a sample html file, or a URL, look for a particular table of data based on its attributes (you can retrieve HTML elements by ID, but in this case, the table we wanted didnt HAVE an ID so we get it based on known html properties instead), use the HtmlAgilityPack core functionality to quickly parse the tabulated data, assign all the field elements to our simple plain old container object, and then use .Net 3.5 LINQ-to-XML functionality to quickly save the results in XML format.

Obviously, you can - patience and ability permitting - change this simple demo to target any HTML file / url.... obviously the poco assignment would change, etc etc blah blah. If you don't already know what I mean by now, then this code isn't going to help you anyway.

The attached file is a C# solution. Commenting is minimal, things aren't particularly tidy, but hey... its a quick hack demo, not production-ready

Hi daleholborow,

Thanks for your sharing,I will look at it and figure out this solution.

 

[YongLim]

I've currently been given an answer to a similar question. Fixed income securities' quotes are very hard to obtain. So the right direction is to concentrate on one particular type -bonds for exmaple (as you are doing). However it's not guaranteed that you'll get al the desired data. As Bob told me, institutions place a hard work on obtaining and distributing those data on a daily bases so they charge a big amount. Lower maturity securities are a bit easier to search.

Hi Tsotne,
You are right,since this is an academic project.
I would like to start from a less maturity securities data first.