That's the problem.
That's too a problem to some folks. Some people are lured to think that a system works when it doesn't.
And the systems have a nasty tendency to evaporate.
So what? :-) This is not about pop music.
I would like to be more serious about this.
Seriously it is often difficult to prove if a system works on historical data or not. If a system is obviously a random walk based loser, that's good news for the tester. If it is not a loser, there remains the tough question, is that an illusion, a temporary phenomen or reality?
Fundamentally, you *can't* prove that a system works using historical data. Stock market data is extremely noisy, and on any past data set you can guarantee to find patterns which give you excellent results on that data set, but which are in fact random.
You might think that you could get round that by splitting your historical data into two sets; you use one to generate your algorithms and the other just to test them. However, you then have to ask what you would actually do with the tests which failed the second part. Suppose your first sample generates 20 possible systems, but when you try them on the second set 17 prove not to be useful. At that point you will no doubt throw those 17 away and try to sell the other three or use them yourself (how many people have you seen selling systems which don't work on their test data?!) But this means that you have in fact trained your algorithms on the full data set, not just half of it.
Potentially you could have a situation where you had more confidence. For example, you could split the past data into ten sets, train one single algorithm on one of them, and then find that it worked on all nine others. However, I've never seen anything like that - perhaps not surprisingly, since anyone who did find something that robust would probably not want to advertise it.
If you want to go to detail, layman's term is 'to prove', a statistician would phrase that something like 'have a confidence'.
Usually the problem is not the noise. On the contrary, the problem is usually some nonrecurring fundamental phenomen which has it's finger print on the data. In the worst case the phenomen is buried inside the data but its origin was never public information.
Actually split data sets work fine in practise. And usually the data sets do not run out. If there is no confidence to the system after supplementary testing, more data can be acquired. I advice against starting with too large data set in the first place.
Of course there are multiple ways to check the system results. The distribution of the results tells much.
It helps to realize that the assumptions of math and statistics are art.
I advice not to overtest. The goal is not to have a confidence that there exists one timeseries which makes the system fail.
Having a working system and making money with it are unfortunately different things.
"Prove" was your choice of word.
I'm not sure what you mean by "the problem", what problem? Of course there are non-recurring phenomena, all kinds of things are non-recurring, the last few years are full of major events which are pretty much one-offs. Pretty public too, but that doesn't tell you how to deal with them.
So you keep looking for more data until you find some which shows that your system works?
Right, if you knew for sure it didn't work you might get depressed ...
This must be some new definition of "working" that I'm not familiar with.
The obstacle preventing the discovery of the system. By the way, the concepts of natural language are bayesian by nature.
Yes, that is the part of the art, trying to deal with them.
The other way around. I look for new data until I find some data sets, which reject the system. Then I consider if I reject that data set :-) All the data is not created equal. This is in the area of art too.
As I stated earlier, failures with spurious data sets have very little significance in real trading. Just avoid those areas of finance.
To be successful in this field it is good to have somebody creative and intuitive and another one logical, like a mathematician, because it is very hard to find those qualities in one package; mathematicians are notorously incapable in thinking, they do not comprehend what is essential.
BeanSmart website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.