DM: Hallberg Rassy's problem

[Warren Sarle]

Hallberg Rassy writes:
> 4.Using SPSS neural connection I made up a neural network based on 
>a set 
> of 4000  active+4000 churned
> 5.The data was: calling patterns of july, agoust and september the 
> target was:  churn/no churn situation in december
> 6.The results was promising: 90% of real churn anticipated, with a 
> cut-off  probability of 80%
> 7.The same network was used on october, nov, dec. data to 
>anticipate 
> march churn  the results dropped to a terrific 11% with the same 
>cut-off 
> of 80%: totally  useless

There is far too little information to render an accurate diagnosis, 
but
when the success rate on the test set drops so far below chance, the
first thing to consider is user error. Note that the success rate on 
the
test data is approximately 100% minus the success rate on the training
data. Perhaps the target variable was coded backwards in the test set.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not 
necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.