Calling self-experimentation N=1 is incorrect
The table below shows a dataset that is discussed in this YouTube video on HealthCorrelator for Excel (HCE). It refers to one single individual. Nearly all health-related datasets will look somewhat like this, with columns referring to health variables and rows referring to multiple measurements for the health variables. (This actually applies to datasets in general, including datasets about non-health-related phenomena.)
Often each individual measurement, or row, will be associated with a particular point in time, such as a date. This will characterize the measurement approach used as longitudinal, as opposed to cross-sectional. One example of the latter would be a dataset where each row referred to a different individual, with the data on all rows collected at the same point in time. Longitudinal health-related measurement is frequently considered superior to cross-sectional measurement in terms of the insights that it can provide.
As you can see, the dataset has 10 rows, with the top row containing the names of the variables. So this dataset contains nine rows of data, which means that in this dataset “N=9”, even though the data is for one single individual. To call this an “N=1” experiment is incorrect.
As a side note, an empty cell, like that on the top row for HDL cholesterol, essentially means that a measurement for that variable was not taken on that date, or that it was left out because of obvious measurement error (e.g., the value received from the lab was “-10”, which would be a mistake since nobody has a negative HDL cholesterol level). The N of the dataset as a whole would still be technically 9 in a situation like this, with only one missing cell on the row in question. But the software would typically calculate associations for that variable (HDL cholesterol) based on a sample of 8.
Calling self-experimentation N=1 is misleading
Calling self-experimentation “N=1”, meaning that the results of self-experimentation are not a good basis for generalization, is very misleading. But there is a twist. Those results may indeed not be a good basis for generalization to other people, but they provide a particularly good basis for generalization for you. It is often much safer to generalize based on self-experimentation, even with small samples (e.g., N=9).
The reason, as I pointed out in this interview with Jimmy Moore, is that data about oneself only tends to be much more uniform than data about a sample of individuals. When multiple individuals are included in an analysis, the number of sources of error (e.g., confounding variables, measurement problems) is much higher than when the analysis is based on one single individual. Thus analyses based on data from one single individual yield results that are more uniform and stable across the sample.
Moreover, analyses of data about a sample of individuals are typically summarized through averages, and those averages tend to be biased by outliers. There are always outliers in any dataset; you might possibly be one of them if you were part of a dataset, which would render the average results at best misleading, and at worst meaningless, to you. This is a point that has also been made by Richard Nikoley, who has been discussing self-experimentation for quite some time, in this very interesting video.
Another person who has been talking about self-experimentation, and showing how it can be useful in personal health management, is Seth Roberts. He and the idea of self-experimentation were prominently portrayed in this article on the New York Times. Check this video where Dr. Roberts talks about how he found out through self-experimentation that, among other things, consuming butter reduced his arterial plaque deposits. Plaque reduction is something that only rarely happens, at least in folks who follow the traditional American diet.
HCE generates coefficients of association and graphs at the click of a button, making it relatively easy for anybody to understand how his or her health variables are associated with one another, and thus what modifiable health factors (e.g., consumption of certain foods) could be causing health effects (e.g., body fact accumulation). It may also help you identify other, more counter-intuitive, links; such as between certain thought and behavior patterns (e.g., wealth accumulation thoughts, looking at the mirror multiple times a day) and undesirable mental states (e.g., depression, panic attacks).
Just keep in mind that you need to have at least some variation in all the variables involved. Without variation there is no correlation, and thus causation may remain hidden from view.
25 comments:
You do know that the "N" of a study refers to the number of subjects, don't you? Not the number of variables.
Hi Anon. The N refers neither to the number of variables nor to the number of subjects of a study. It refers to the number of measurements drawn from instances of the study’s main unit of analysis.
In a cross-sectional study, if a study’s main unit of analysis is a group of individuals, then each measurement (row) refers to a separate group of individuals. Not a single “subject”; which, btw, is a term that is becoming dated in reference to people, giving way to the more modern term “participant”.
In a longitudinal study, if a study’s main unit of analysis is the individual, then each measurement refers to an individual’s attribute (e.g., HDL cholesterol) at a different point in time, as in the example used in this post.
Within the field of applied and experimental behavioral psychology (technically, experimental analysis of behavior/applied behavior analysis) experiments with one person or organism are common. These are referred to as "single subject experimental designs." I believe that description is appropriate in this context as well.
Single subject experimental methods are described in great detail in Murray Sidman's classic book "Tactics of Scientific Research."
N usually refers to the size of the population that is being studied. Researchers take n samples from population of N "potential participants" and then try to make inferences and apply it to the entire N. So, I have a bit of a squirm when I see N=1 instead of n=1. I still use N=1 all the time, and don't see it as a timid statement, as you do. It's more like, "My N=1 can beat up your honor student," or said another way, the inferences you make based on your limited sample not predictive for me, so here is an additional piece of data.
As I mentioned in comments on Masterjohn's blog, I don't believe that repeated self-experimentation always leads to a better signal. I disagree with you when you say that you'll get this nice, low variability when you test only on yourself. Repeated testing can just lead to more variability, and not that "good" kind you are looking for if you want to get a correlation indication.
so, I looked at your bio and I'll bet you've run into "The Beer Game" here and there. A good example of how repeated tweaks with lagging information can lead to disaster. That's how I feel about repeated self-testing. Maybe that isn't how it works for you, but for my N=1, many tests aren't repeatable. (Sorry, just couldn't resist mis-using it once again. Would you prefer YMMV?)
Aha, but here is what makes the human body very different from simulation games like the Beer Distribution Game and People Express – the human body is an awesome homeostatic system that tries to preserve the status quo at all costs.
This is what makes self-experimentation so useful, and related results so uniform compared with population results.
Btw, both of those simulations/games are from the MIT, and they are great! I used to teach management principles in New Zealand using People Express.
Well, maybe, you have an "awesome homeostatic system", but let's not make the assumption that everyone's system settles down to a single point. My N=1 says it does not. For some supposedly simple experiments I have done on myself, I have gotten out of whack for months. If I were to be included in a small study, it would be an outlier for sure, but my social network anecdotes tell me I am not alone, even though people like me are unstudied.
People playing the beer game also try to do their best, of course at the end they end up laughing at how bad things can get when there is insufficient communication. But I also think that people everywhere with hormonal miscommunication issues aren't laughing yet.
I also used to use the beer game in some of my classes.
I will agree that throwing stuff together for some kind of big average hasn't served anyone well.
Hi EB. Actually that is not at all the point of my comment. The simulation games have “states” that vary easily in all kinds of directions, depending on the user inputs. There is no homeostasis built in most simulation games, as far as I can tell. Otherwise they would not be as fun, I guess.
Human bodies have homeostatic mechanisms, but that doesn’t mean that acute responses will not be seen. If one’s blood glucose levels go down too much, because of the reactive hypoglycemia, several acute body responses will ensue.
Still, one’s individual blood glucose levels will vary much more uniformly in response to inputs (e.g., white bread), and somewhat predictably for that individual, than the blood glucose levels of a group of individuals:
http://bit.ly/nUTNPD
Perhaps you can give some examples of what happened with you in your self-experimentation, to illustrate your point.
You are being overly pedantic. When people use the "n" of the study, they are referring to the number of participants. I work in the field of clinical studies and everyone understands "n-1" to mean "anecdotal evidence".
And, as they say, the plural of "anecdote" is "data."
The post is not as much about being a pedant, worrying about semantics, as it is about highlighting the power of self-experimentation.
Off topic, I realize, but have you read Volek and Phinney's "Art and Science of Low-Carbohydrate Living?"
I feared it would be just another "diet" book, but iin fact it does a nice job of tying together some interesting aspects of the science--and mentioned some things that were entirely new to me.
Haven’t read the book David, but I’ve read many of Dr. Volek’s articles. He is one of the most prolific and widely cited researchers in the world in exercise physiology, and somewhat unique in his LC approach within that sphere.
Btw, I’m now reading the Perfect Health Diet.
I've finished the new Volek/Phinney and highly recommend it. Now I'm reading "Power, Sex, and Suicide" which, althugh it sounds like it might be about Las Vegas, is about mitochondria.
Let us know what you think of Perfect Health Diet--I haven't read it.
Many dieters are quite successful when they first try a low carb diet. If they veer off for a bit, and then restart, it is usually not as successful as the first time. There is plenty of debate as to why, but whatever the reason, clearly doing the "treatment" multiple times produces different results.
I have read several times from people on low carb, something like, "I was doing fine until too much stuffing and pie at Thanksgiving, and now I haven't been able to get back into gear." We tend to write that off as lack of motivation, but what has happened is probably a hormonal shift, and until what has changed gets back to where it was, any such "treatments" will be confounded by that most giant lurking variable.
Now for myself and others I know, we may go off for a weekend and a couple of buffet lunches, come back, restart the perfect diet, and struggle for MONTHS to get back to where we were. Yes, MONTHS! I know young thin males don't get this part, as they truly think we're lying about what they eat. Now, here is a data point for everyone here.
Just had the greatest time recently reading all about Atlatl Bob, the guy who probably taught Robb Wolf how to take down an elk. Seems experts had been saying all along that the atlatl wasn't that useful. Then someone came along and figured out what was really going on. Well, that's how I feel about this homeostasis theory.
Oops, that s/b "Lying about what WE eat,"
Does blogger have a setting that would prevent 'certain people' from leaving multiple sequential comments. I don't want to read the stream of consciousness of an individual commenter who can't be bothered to think first and then write concisely.
Sorry, wishful thinking, it was just time for a rant.
n=the number of study participants. Period
Post a Comment