## Rubrics as Data – Part III

It has been a long time since I have addressed this topic.  I was going to add, “for good reason.”  Is utter discouragement and temporarily quitting a good reason?  Anyway here is the saga.

I first worked my way steadily through Statistical Rethinking by Richard McElreath entering all the examples in R and learning about Bayesian modeling.  The examples seemed a little arcane and the golem metaphor a little off-putting but by the end (nearly the end)  I felt that I could build and explore simple Bayesian models.   I was empowered.   And what better way to use my newly-developed skills than to work on a rubric as data model.  I set about building the model as described in Section 11.1 Ordered Categorical Outcomes.  What could go wrong?

I made MAP (maximum a posteriori) and a stan (probabilistic programming modeling language) models using my example from the last posts – 3 1 scores, 13 2 scores, 15 3 scores, and 8 4 scores.  When I did the sampling from the models, I got the occasional nonsense which I attributed to having my cut points (the points separating the scores (logit) ) (technical details will be omitted) getting out of order.  Now what?  I quote Dr. McElreath’s text, “As always, in small sample contexts, you’ll have to think harder about priors.  Consider for example that we know alpha 1 < alpha 2, before we even see the data” (page 335).  So that was my problem.

How to fix them.  Before that, I need to know that I had “fixed” it. Here is how I graphed the sampled data from my models.

I sorted the samples by goodness-of-fit to the original data.  The distribution histogram is in the upper left corner. I moved down the sorted list so that I had 50 evenly spaced samples and plotted them on the large bar graph to give a sense of the variability.   I plotted the distribution for each score below the central bar graph as proof I could calculate prediction intervals.

I could now see if anything went wrong, like in this graphic for a small sample.

My object became to remove the odd looking distributions in the middle of the picture.  I began by making reasonable changes to the model.  Reasonable to me but, I eventually figured out, incomprehensible to the the shell that Dr. McElreath built to simplify his exposition.  This took a while.  I then plunged into the R literature.  This was difficult.  I ended up tracking through threads, reading non-answers and references manuals with few and poorly remarked examples.  In the end I decided to go with a purely stan model.  At this point I got discouraged.  I was fighting picky syntax and the only help (stackflow) always seemed to avoid direct answers.  The responders seemed more interested in criticizing the asker’s model or suggesting a better model.  Please just help us build the model no matter how wrong it is.  Also have a easily located place where we can ask about models not the code.  I got discouraged and quit.

A month ago, under the influence of excess coffee, I found the optimism to start again on the stan model.  It worked.  The use of the ordered data type did the trick and away I went.  Here is the result.

Now I can use posterior sampling to get prediction intervals.  For instance model in the graph above, The 95% prediction intervals are

My next step is to explore stratified sampling (say sample results from each major) using partial pooling.