It has been a long time since I have addressed this topic. I was going to add, “for good reason.” Is utter discouragement and temporarily quitting a good reason? Anyway here is the saga.
I first worked my way steadily through Statistical Rethinking by Richard McElreath entering all the examples in R and learning about Bayesian modeling. The examples seemed a little arcane and the golem metaphor a little off-putting but by the end (nearly the end) I felt that I could build and explore simple Bayesian models. I was empowered. And what better way to use my newly-developed skills than to work on a rubric as data model. I set about building the model as described in Section 11.1 Ordered Categorical Outcomes. What could go wrong?
I made MAP (maximum a posteriori) and a stan (probabilistic programming modeling language) models using my example from the last posts – 3 1 scores, 13 2 scores, 15 3 scores, and 8 4 scores. When I did the sampling from the models, I got the occasional nonsense which I attributed to having my cut points (the points separating the scores (logit) ) (technical details will be omitted) getting out of order. Now what? I quote Dr. McElreath’s text, “As always, in small sample contexts, you’ll have to think harder about priors. Consider for example that we know alpha 1 < alpha 2, before we even see the data” (page 335). So that was my problem.
How to fix them. Before that, I need to know that I had “fixed” it. Here is how I graphed the sampled data from my models.
I sorted the samples by goodness-of-fit to the original data. The distribution histogram is in the upper left corner. I moved down the sorted list so that I had 50 evenly spaced samples and plotted them on the large bar graph to give a sense of the variability. I plotted the distribution for each score below the central bar graph as proof I could calculate prediction intervals.
I could now see if anything went wrong, like in this graphic for a small sample.
My object became to remove the odd looking distributions in the middle of the picture. I began by making reasonable changes to the model. Reasonable to me but, I eventually figured out, incomprehensible to the the shell that Dr. McElreath built to simplify his exposition. This took a while. I then plunged into the R literature. This was difficult. I ended up tracking through threads, reading non-answers and references manuals with few and poorly remarked examples. In the end I decided to go with a purely stan model. At this point I got discouraged. I was fighting picky syntax and the only help (stackflow) always seemed to avoid direct answers. The responders seemed more interested in criticizing the asker’s model or suggesting a better model. Please just help us build the model no matter how wrong it is. Also have a easily located place where we can ask about models not the code. I got discouraged and quit.
A month ago, under the influence of excess coffee, I found the optimism to start again on the stan model. It worked. The use of the ordered data type did the trick and away I went. Here is the result.
Now I can use posterior sampling to get prediction intervals. For instance model in the graph above, The 95% prediction intervals are
My next step is to explore stratified sampling (say sample results from each major) using partial pooling.