I looked it before, it is pretty cool. If you just want to write your own stuff and use it to solve math problem, it is a good choice. It is free!
However, I doubt about it will get popular. Matlab is sort of a standard skill for science and engineering students nowadays. Everyone is so used to it, and there are tones of packages written in it and available free online. I don't think those will work quite well in Octave. Not sure ppl will be willing to switch. Also,the most important part about Matlab is its tool boxes. That's the killer and I didn't see Octave have enough toolbox that can match up.