The quantitative predictions of a model are determined by using any mimimization routine such as the program STEPIT (Massaro, 1998). In STEPIT, a model is represented to the program in terms of a set of prediction equations and a set of unknown parameters. By iteratively adjusting the parameters of the model, the program maximizes the accuracy of the predictions by minimizing the RMSD. One potential worry is that minimization programs can settle into “local minima” and therefore not output the best possible fit. This problem can be easily overcome by beginning the parameter search at a variety of starting values for the parameters. In addition, “local minima” do not usually occur given the fairly simple data sets and models in this domain. The successful outcome of minimization is a set of parameter values which, when put into the model, come closest to predicting the observed results. The RMSD is used to evaluate the goodness-of-fit of a model both in absolute terms and in comparison to other models. Two potential limitations of RMSD are that it does not take into account the number of free parameters of a model nor its flexibility in predicting a wide range of outcomes. Given these limitations, other techniques have been developed to evaluate models.
The data to be fit can be average response probabilities or ratings but the values have to be between 0 and 1 because the currency of the model is in the form of truth values between 0 and 1. The response probabilities or ratings are averaged across a series of observations in order to give fairly reliable measures for each experimental condition.
We begin with a simple 3 by 3 factorial design with results from 2 subjects. Here are the data to be fit so that there are two subjects with nine observations each for a total of 18 scores. Scores are always entered in terms of standard factorial input with fastest to slowest moving variables. Subjects are always the slowest moving varable.
NOTE: [notes in square brackets added to outputs]
.59 .90 .96 .26 .90 .96 .08 .37 .87 .25 65 .92 .15 .50 .85 .05 .23 .66
c trace data model #1
c
COMMON /FUN/ pr(9),ob(9)
c number of predicted and observed data points for each subject
real d(9,2)
c num of observations per subject, number of data sets (subjects)
external funk,pp
call sas('trace1.dat','tdm1.lst','tdm1.fit')
c data and model names
write(7,"(' *** tracedat model 1: FLMP ***')")
read(9,"(f4.2)")d
call runstp(funk,6,9,2,d,ob,pr,pp)
c number of parameters, number of data points, number of data sets
stop
end
SUBROUTINE FUNK
COMMON /STEP/ NV,NTRACE,MATRIX,CHISQ,MASK(100),X(100)
COMMON /FUN/ pr(9),ob(9)
c g w
real d(3,3)
c labeling factorial design
equivalence (pr,d)
integer g,w
real xg(3),xw(3)
c 3 levels of g and 3 levels of w
equivalence (x(1),xg),(x(4),xw)
c this instruction defines the free parameters: 1-3 for g and 4-6 for w
c
c what follows is model
do 60 g=1,3
do 60 w=1,3
t=xg(g)*xw(w)
tn=(1.-xg(g))*(1.-xw(w))
60 d(g,w)=t/(t+tn)
c compute sum of squared deviations between observed and predicted values (9 of them here)
call cchi(ob,pr,9)
RETURN
entry pp(is,rmsd)
c write out rmsd and parameters
write(7,117)is,rmsd,xg,xw
117 format(/' Sub 'i2' rmsd= 'f7.4,/' g = '3f7.4/,
& ' w = '3f7.4/)
return
end
tdm1.lst gives the RMSDs and the parameter values
*** tracedat model 1: FLMP *** [name of data and model] [ data set RMSD and parameter values & correlations of predicted and observed ] Sub 1 rmsd= 0.0317 g = 0.1945 0.7960 0.9763 w = 0.8510 0.6169 0.1351 obs~ pre: r= 0.996 r2= 0.993 t= 31.15 df= 7 Sub 2 rmsd= 0.0041 g = 0.1203 0.4332 0.8253 w = 0.7087 0.5628 0.2870 obs~ pre: r= 1.000 r2= 1.000 t= 198.63 df= 7 Sub 3 rmsd= 0.0246 [ computes the average data and gives the fit for this averaged data ] g = 0.1932 0.6058 0.9114 w = 0.7319 0.5651 0.2274 obs~ pre: r= 0.997 r2= 0.994 t= 34.16 df= 7 Sub ** rmsd= 0.0179 [ gives the mean of the individual subject fits & parameter values ] g = 0.1574 0.6146 0.9008 w = 0.7799 0.5898 0.2111
0.59000 0.90000 0.96000 0.26000 0.90000 0.96000
0.08000 0.37000 0.87000 0.57972 0.95706 0.99577
0.28000 0.86270 0.98515 0.03636 0.37873 0.86554
0.25000 0.65000 0.92000 0.15000 0.50000 0.85000
0.05000 0.23000 0.66000 0.24965 0.65031 0.91994
0.14967 0.49593 0.85874 0.05217 0.23528 0.65530
0.42000 0.77500 0.94000 0.20500 0.70000 0.90500
0.06500 0.30000 0.76500 0.39532 0.80749 0.96560
0.23734 0.66629 0.93038 0.06584 0.31139 0.75165
say r modelname & e.g. r tdm1 & to compile and run modelname.f
fx $1.f; mv a.out v$1; rm $1.o; v$1; rm v$1
fx file (vax vers) compiles fortran program and links it with stepit libraries f77 $1 /d/i/psych36/lib/s.oo /d/i/psych36/lib/steps.oo