  There are many practitioners that create software to buy and sell financial
assets in an autonomous way. There are some digital platforms that allow the
development, test and deployment of trading agents (or robots) in simulated or
real markets. Some of these work focus on very short horizons of investment,
while others deal with longer periods. The spectrum of used AI techniques in
finance field is wide. There are many cases, where the developers are
successful in creating robots with great performance in historical price series
(so called backtesting). Furthermore, some platforms make available thousands
of robots that [allegedly] are able to be profitable in real markets. These
strategies may be created with some simple idea or using complex machine
learning schemes. Nevertheless, when they are used in real markets or with data
not used in their training or evaluation frequently they present very poor
performance. In this paper, we propose a method for testing Foreign Exchange
(FX) trading strategies that can provide realistic expectations about
strategy's performance. This method addresses many pitfalls that can fool even
experience practitioners and researchers. We present the results of applying
such method in several famous autonomous strategies in many different financial
assets. Analyzing these results, we can realize that it is very hard to build a
reliable strategy and many published strategies are far from being reliable
vehicles of investment. These facts can be maliciously used by those who try to
sell such robots, by advertising such great (and non repetitive) results, while
hiding the bad but meaningful results. The proposed method can be used to
select among potential robots, establishes minimal periods and requirements for
the test executions. In this way, the method helps to tell if you really have a
great trading strategy or you are just fooling yourself.
