DM:Introduction

[Murray Jorgensen]

I am a statistician with an interest in the analysis of large data 
sets
'lurking' on this list. I am interested the extent to which the 
insights
statisticians have built up from the design and analysis of smaller
collections of data can be applied to larger data sets. I am also 
interested
in how these insights might need to be modified for larger sets of 
data.
Recently I have supervised a PhD developing a model-based alternative 
to
cluster analysis.

Earlier in my career I worked in a Government department and was 
frustrated
by how difficult it was to get access to information in computer 
systems and
databases in a form suitable for analytical work. It is very pleasing 
to see
this situation changing.

The tradition in statistics is to work as closely as possible with the
'client' who has obtained the data. We are very conscious of how easy 
it is
to draw misleading conclusions from the data, especially when a large 
number
of variables are being studied at the same time. We are suspicious of 
'black
box', off-the-shelf systems.

My feelings of caution for how the technologies might be misused are
balanced by my feelings of excitement at the sheer size of the new 
areas of
data exploration opened up. We are entering new worlds here and I 
suspect
that it will not be long before companies discover that it will be 
more the
quality of the *people* who are the analysts than the quality of the
particular software packages used.

Dr Murray Jorgensen        maj@waikato.ac.nz       Phone +64-7 838 
4773
Department of Statistics         home phone 856 6705;      Fax 838 
4666
University of Waikato  
http://www.cs.waikato.ac.nz/stats/Staff/maj.html
Hamilton, New Zealand        **** Editor: New Zealand Statistician 
****