Why tobacco use modeling need economic mechanisms - FDA workshop slides

I have not been doing a great job of creating updates about my THR modeling work, so here is one easy very-partial remedy for that.  The FDA Center for Tobacco Products is holding a workshop about modeling and other methods that relate to their regulations and I am presenting a talk using these slides.

As those of you following my work on this subject know, I am rather critical of the usefulness of the existing "models" of tobacco use.  (For those not familiar, follow the tag on this post for previous and future posts on the subject.)  I use the scare-quotes because I argue that most of what are called models in this space fail because they are not actually a simplified version of the real system, but are really just complicated calculation tools.  They completely omit the underlying mechanisms of the system -- the consumer economics -- and thus just translate high-level statistics (e.g., "assume 2% of the smokers transition to e-cigarettes each year") into high-level outcomes (e.g., "the smoking rate over time follows this path").  This does not mean they are not useful, of course.  There is value in that.  But the value is calculating the answer to high-level hypothetical questions, not actually representing the system.

My argument in this talk is about how the lack of that real representation of the system means, most obviously, that it is impossible to make predictions about previously unobserved phenomena.  (If the only use of data is to say "we have observed that when X happened then Y resulted", you cannot say much about situations that have not happened before.)  But it also means that even the high-level predictions are likely to be wrong because they are based on a misuse of the data (which I call superstition rather than science).

I made the tactical error of offering to present on any of several aspects of my research agenda, but fortunately the organizers shared my opinion that this bit is the most crucial for people to understand at this point.  (Note to self: Don't count on that in the future.)  The talk is likely to come as a rather unwelcome coda (it is scheduled very late in the workshop) to a series of presentations about "models" that fail to do what I am saying must be done.  Of course, I might be pleasantly surprised and discover that my message has already been covered.  Such good news for the science would be bad news for my talk, of course, making it awkward with a lot of phrases like "this has already been discussed, but to reiterate the point".  But I am not optimistic/pessimistic that there is much chance this will occur.

Models v. Mechanisms at FDA Center for Tobacco Products

Observed coincidences occur far more often than chance would suggest because we look for them and define our list of what would constitute an interesting coincidence based on what actually occurred (we have no intuition for just how huge the denominator is).  I know that.  Still, I find it pretty remarkable that for the last couple of days I have been trying to nail down exactly how to explain what is missing from the behavioral modeling by FDA CTP, and then discovered in my morning econoblogosphere reading, the answer I needed seems to be the topic of the hour.

Yesterday I posted some advice to FDA CTP about the need to understand social science (mainly economics) in their modeling of behavior. There is relatively little economics that needs to be considered in traditional FDA missions, and that which is needed is relatively simple.  But regulations that are intended to affect preferences about a freely-chosen consumer product where preferences vary across the population (i.e., like tobacco products, in contrast with medicines or food safety) are all about economics.  The failure to include explicit economic analysis in the recent report on the possibility of banning menthol cigarettes illustrated the problems, both scientific and ethical.

But if you were to suggest to the people working in the FDA orbit that they do not really have a model of people's choices about tobacco products (as I have argued), they would probably reply that they do have models.  Several of them.  (For those who are familiar with this field I am, of course, talking about Levy, Mendez, Environ.  For those not familiar, that should present no obstacle to understanding this post -- just know that I am talking about a handful of well-known specifics.)

Ok, there are models.  But there is something fundamentally wrong with them.  They do not offer us any reason to believe in what they say will occur at the micro level (that is, why each individual person whose actions, collectively, result in the outcome, will do what they suggest they will do).  What I mean is that they tell us things like "if X% fewer people start smoking each year", say, due to a menthol ban, "then this graph shows the number of smokers in the future, which is lower than current trends by Y" (if they fill in other information and make a bunch of other assumptions about what is happening, of course).  But as for why X% fewer people would start smoking, there is nothing at all.  There is just the number.

I have criticized these as being more like calculators than models.  If a population starts at 1 and doubles, every period then the number after n periods is 2^n.  But it is hard to call the equation "2^n" a model.  Similarly, I argued, the preferred "models" used by the tobacco policy inner circle are just more complicated equations.

I learned, however, that this point was not widely convincing based on sociological empiricism -- i.e., I tried to make the point to people and did not have much success.  I realized perhaps why this was based on my blog reading from this morning:  I was using the wrong words to make my point.  It is not that these are not models; any calculation, no matter how simple, can be called a model if it is representing a worldly phenomenon in some useful way (even that lowly 2^n).  The problem with these models, and the reason they failed as legitimate models, is the lack of mechanism.

A model uses numbers and equations to show how one variable/construct/point-in-time/etc. affects others.  But the model may not capture why a particular effect occurs (the mechanism), as with that X% reduction in smoking initiation.  In such cases, it is really just answering a hypothetical question ("if X were true, then Y") rather than making real predictions ("X appears to be true, therefore Y").  But the models that would be useful for FDA purposes are ones that tell us "therefore" not just "if...then".  Of course, every model is going to have some simplified or hypothetical elements (if there are no simplifications it is not a model, it is reality) and, once again, there are no bright lines since the mechanisms generally are abstractions (i.e., models) in themselves.  But for a model to offer predictions that do not just result from hypothetical inputs, there has to be some "why" built into it.

It seems that the problem is that these models have been developed in a world where the only familiar social science is epidemiology.  Epidemiology usually fails as a social science, and as a science more generally, because there is very little attention paid to mechanisms.  That it fails as a social science is fairly easy to explain: most people doing it do not realize they are engaging in social science, and most people teaching it have no background in social science.  They think they are just doing medical trials.  Sometimes this is literally true, of course, and sometimes the observational epidemiology is legitimately an attempt to substitute for medical trials.  But as soon as what is being studied is not purely biological, and involves people as people, not just as organisms, it is social science.  Medical trials are easy because either the mechanism is obvious or it does not matter -- e.g., this drug makes cancer go away, and we probably have a guess about why, but that guess does not matter because the mechanism does not matter to the epidemiology (though obviously it does for the drug development process).

The failure to be good science at all is less easy to explain or defend.  For almost 15 years, mechanism-oriented methods have been developed and taught (in the few good epidemiology departments).  These tend to be pretty simple, just boxes and arrows that show what is causing what, but that is most of what you need.  Unfortunately these are (a) seldom used at all and (b) almost exclusively used just for identifying confounders.  The latter is useful, of course, and doing it is far better than not doing it.  But what is missing is use of these mechanistic models to address questions like "if X is really causing Y by affecting Z, then I should be able to observe not just an association between X and Y, but also..."  Such scientific hypothesis testing is close to completely absent from epidemiology.  Instead, mechanisms in epidemiology exist entirely in the untested conclusion statements.  You have seen it: An association is observed and there is a discussion of how X must be to be causing Y as a result of Z, or whatever, but whether that really seems to be true is never addressed scientifically.  It is worth reiterating:  In epidemiology, mechanisms live almost entirely in the conclusions and not in the science.

So circling back to the question of tobacco behavior modeling, when the models are developed in the tradition of epidemiology, it is little wonder that there is no mechanism.  The "why" of what happens when a variable changes is not part of epidemiology, and so not part of the models.  It is just assumed that if the effects of a particular variable changing were observed in the past -- or more likely, merely if there was just some association observed in the past, with no effects of changes observed -- then that same association will still occur if an intervention is imposed (e.g., menthol is banned).  But there is usually no reason to believe that, and indeed, often a lot of reason to not believe it.  To take an extreme case, one of the popular models assumes that without menthol, the rate of smoking initiation would drop by the rate at which smoking is initiated with menthol cigarettes.  Put a little more simply, this basically is the assumption is that everyone who would have initiated with menthol will therefore never smoke (it is even a bit worse than that because it is based on past associations which might themselves change).  I suspect I do not need to explain why the implicit mechanism about people choosing to initiate smoking menthol cigarettes is rather absurd.  The absurdity of that seems unfathomable unless you recognize the mechanism-free mindset: "all we know [the mindset goes] is the association we observed before, so we just have to assume that association will always exist".

FDA on menthol cigarettes, some suggestions for research methods

The U.S. FDA is probably the most respected and influential medical research organization in the world.  Sure they have their hiccups and there are criticisms (many legitimate, many not) about the science and the choices about what risks to take (historically entirely in the direction of them being too quick to keep a potentially useful drug off of the market, more recently in both directions).  But all in all, it is hard to imagine engineering an institution that does much better.

But tobacco controllers (including a former head of FDA during his tenure) and their pet congressmen got the bright idea of adding a category of non-medical consumer goods to FDA's scientific purview.  To these non-scientists, it probably seemed that there was no contradiction here.  Science is science, right?  Clinical trials of medicines, monitoring food safety, consumer preferences, social forces, climate change research, isolating the Higgs boson -- if they are good at the first two, and experienced in the related ethical questions, then they must be able to do the rest, right?

Um, no.

It is clear that the FDA scientists who have been asked to look at tobacco products are trying, and it is equally clear they are frustrated.  Their latest report on menthol in cigarettes and the predicted effects of banning it [I got it here but that link seems to not work now], offers an opportunity for some unsolicited advice.  After I collect comments and my thoughts, I might include some of it as a public comment, [UPDATE: link fixed] which they are currently soliciting.

Their evaluation concludes that there is no measurable difference in the biological risk between menthol and non-menthol cigarettes (keeping the quantity of exposure constant).  This is based on research that FDA is pretty good at; it is not quite the same as their core competencies, medical trials and safety testing, but in the neighborhood certainly.

But the crux of the decision about whether to ban menthol is presented in terms of economics.  Unfortunately they do not say this.  This is presumably because economics -- the relevant science when you are looking at consumer choices  -- has never been part of what FDA does.  Drugs and medical devices are evaluated based on people who "need" them and are generally assigned by gatekeeper rather than chosen, so there is limited need to consider economics.  To the extent that economics is considered, it is the rump economics of "cost-effectiveness" and "quality-of-life" measures, which are definitely useful and nontrivial, but only a bit of the picture.  The food side is rather closer to normal consumer goods, but the focus is still on what everyone "needs" (are not willing to give up at any plausible cost), like non-infectious food.  When legitimate economic questions do come up (e.g., some people prefer to be able to consume raw dairy products, even though they are judged not safe enough by the simple bright-line standards), the system has no mechanism for balancing competing preferences, a hint of the challenge in regulating tobacco products.

It is natural that there is no historical capacity to do much economics within FDA.  I would argue that this is the biggest problem the normative side (i.e., ethics -- assessing what is the right thing to do to make people better off), though selling that message is a tough fight.  But it is also a problem on the positive side (assessing how the world works when individual free choices are involved), which ought not to be too tough to sell.  Understanding consumer choices, and being open and explicit about the science, really should be part of the Center for Tobacco Products.

The specific economic question at hand is the effect of menthol on the number of people who choose to smoke, the quantity they choose to smoke, and whether they choose to stop smoking.  These are all economic questions, and when someone tries to address them ad hoc, using epidemiology rather than welfare economics, it does not work out so well.  Indeed, even the description of the question at hand, with the key word "choose" in it, defies the standard narrow medicalized method of looking at products (and does not seem to appear at all in the FDA report, despite how crucial the concept is).

The questions being asked are in the form of "whether" -- e.g., is there likely to be more smoking if menthol is available -- rather than "how much".  But a tiny bit of economics reveals that the "whether" question is like asking "does this object have a weight" rather than "how much does it weigh".  Of course the availability of a flavor that some people like leads to more product use, and therefore its elimination would reduce how much people like the product.  Somewhere out there is someone who is barely on the positive side of indifferent between smoking and not, and very much likes menthol.  Remove the availability of menthol and he would not smoke.

(Notice that I am avoiding the question of implementation here, and simply positing the "removal" outcome.  Merely banning menthol cigarettes rather than magically removing them from the world creates all kinds of interesting complications about black markets and do-it-yourself mentholation, which is quite easy.  I will come back to that in a later post.)

The "how much" question is quite a bit more difficult to answer.  To have any hope of making a useful prediction, it is critical to understand what is going on:  people are using a product because they like it better than its close substitute (nonmenthol cigarettes), most of them probably prefer the substitute to abstinence, some of them like their product enough that they would defy the law, etc.  Without these economic points, it is difficult to imagine making a useful prediction.  Indeed, if you look at the models that have been used for prediction, they are clearly based on premises that are indefensible but probably the only premises that someone can come up with if they ignore economics.

Some consist of assuming that any additional use or initiation of cigarettes that is associated with choosing menthol (i.e., smoking rates are higher in subpopulations that use menthol more often) is causal, and thus without menthol the rate would drop to the average.  The economics shows that while this outcome is in the plausible range, it would be mere luck if it really turned out to be right because the basis for the claim does not actually support the claim.  Which is to say, the prediction has no validity because the premise of it is clearly wrong.  It actually gets worse than that, with some of the modeling going so far as to predict that all consumers of menthol cigarettes would be abstinent if menthol were not available, which is not even in the plausible range of values.  (No, I am not kidding -- one of the most cited predictions about the effect of banning menthol is based on this premise.)

Some of the most dramatic errors in the history of science, to say nothing of incorrect claims you see in the news today, result from confusing statistics with mechanism.  That is, researchers who do not know why something is happening (e.g., atomic theory has not yet been discovered so they have no idea why samples of pure elements weigh what they do, to take a classic historical example; or they seem to not realize that people make choices based on preference, to take the example of tobacco control industry researchers) sometimes go to great lengths to make measurements.  But when they try to interpret the observations without understanding the underlying phenomenon, and basically just assume that the measurements are the phenomenon (an example of which is assuming that all observed association is causal), then whether they are right becomes just a matter of luck.

As I mentioned, the other problem with not understanding the underlying mechanism when dealing with worldly questions is that wrong (in the sense of accuracy) can also be wrong (in the sense of unethical).  When tobacco control activists hide the phenomenon of people's preferences, choices, and happiness behind naive statistics, they avoid having to admit that they are a special interest group trying to impose a narrow "moral" view.  Our nation's government is not a special interest group and generally does a pretty good job of resisting imposing narrow moral views on the citizens (thank you, James Madison et al.!).  But if FDA research ignores the economics, it tends to prevent decision-makers from realizing they are making ethical, not technical, decisions.  (And it allows those who know they are imposing narrow "moral" views to pretend they are not doing so.)

Banning menthol would serve only one purpose: intentionally lowering the welfare of people who currently choose a particular product.  As soon as you express the economic situation in economic terms, this becomes immediately apparent.  So, is that justified by the (legitimately predicted) benefits it would produce?  Is such an action by a for-the-people government ever justified?  Do smokers deserve to have their welfare lowered?  None of these questions are answered by economics or any other science, but economics has the advantage of forcing a recognition that those are the questions that need to be addressed.  Anyone who suggests that the question "should we ban menthol cigarettes" can be answered scientifically, rather than ethically, is doing the wrong science.

Finally, as a comparatively minor aside about how to do social science, I note that the first paragraph of the FDA report makes a claim about the portion of the US cigarette market that is menthol, citing it to a 2004 paper.  2004??!  Folks, social science does not work that way.  People who are used to dealing with biology and other sciences that study phenomena that do not change much over time get into the habit of ignoring when a measurement was made.  This is a mistake even then, but it is a fatal error when dealing with social science -- just think about how much has changed in the tobacco product markets in the last decade.  An economist wanting to make such a summary claim would either find a more recent estimate, make a rough general claim without citation (making clear that the exact number is not known to the author but that it does not matter much), or look at the most recent statistics themselves and do the calculation.  Part of the problem is that people who are used to dealing with only one area of science, medical research, get the mistaken impression that all or most useful information is contained in journal articles.  That is clearly not the case in social science, where constantly updated statistics, working papers, and the blogosphere generally contain much better current information and thinking.  Indeed, as with many serious sciences, when an article appears in a good journal it is more like an archiving and awarding of a trophy, and is not really the publication, since anything worth reading has already been circulating long before the final version is etched in stone.  These are just a few of the things that FDA researchers need to understand now that they have entered the world of studying people as people, and not just as biological agents.

[More on this theme in the next post.]

Monte Carlo uncertainty as a method for hiding uncertainty (wonkish)

I am writing this mostly as a placeholder for some thoughts emerging from modeling work I am doing right now.  I thought that some of my more technical readers would find it interesting and maybe some of you (or at least one -- talking to you, Prof. I.B.) could help me think this through or maybe even identify where others have made the same observations.

The work that really put me on the map (presented to much acclaim in 1999 though I could not get it published in final form until 2003) was presenting a way to properly report the uncertainty around epidemiologic estimates.  To explain, the only uncertainty around point-estimates results in epidemiology that was (and still almost always is) the confidence intervals.  These provide a heuristic measure of about how much random sampling error there is in a result.  But the reporting of CIs tends to obscure all the other non-random errors in the result for most readers (including most people who claim to be experts in the field).  People see this error bar around the estimate and assume that it really represents how uncertain the estimate is, which it most certainly does not do.  Thus, in some sense, the dutiful reporting of one measure of uncertainty serves as much to hide uncertainty as it does to report it.

What I did was propose a way to report an estimate of the impact of other types of errors (measurement error, sampling bias, etc.) in addition to the random sampling error.  The method that I used to do the calculation for this model was Monte Carlo simulation.  This was purely a calculation technique -- MC is the easiest way to do complex calculations when you are working with uncertain inputs.

(For those who do not know, the method consists of taking a random draw from each uncertain input and calculating one model result, and then repeating that many thousands of times with different random draws to show the distribution of possible results based on the distribution of inputs.  It is theoretically possible to calculate the same result directly using equations, but that is mind bogglingly difficult, whereas MC is easy.  It is basically equivalent to doing a calculation using a computer, or digging a hole with a backhoe, rather than doing it by hand -- the MC simulation, computer, or digger is just a tool to make the job easier, not the essence of what is being done.)

Much to my annoyance, almost everyone (I can think of only one exception) who took these ideas and ran with them did two things that were utterly contrary to the spirit and goals of what I was presenting:  1. They treated the MC tool as if it were the important essence in itself, rather than properly treating it as just the method to get to a goal.  2. They started using the approach to replace one misleadingly precise claim (the epidemiologic point estimate that ignores the errors) with a more complicated misleadingly precise claim (that the rough distribution that can be calculated is a precise estimate of the results of uncertainty).

Fast forward to today, when computers are quick and cheap (it took my best computer 2.5 weeks to run the simulation that was the core of what I produced in 1999), and we see MC error calculations of various sorts in many calculations.  But these seem all to serve mainly to impress naive readers with the fancy tools, but also to pretend to account for the uncertainty and thereby hide the real uncertainty.

I have started thinking of it as "Monte Carlo porn".

So, for example, a model might ask what will happen to smoking rates over time when a predicted 6.3 percent reduction in smoking initiation caused by some anti-smoking policy filters through the population over time.  The modelers then report "the uncertainty" by allowing the reduction to differ by +/-10% of the predicted value, run a MC simulation using random draws from that range, and report a simple summary of the distribution of results.  This adds nothing of genuine scientific value.  Anyone who is capable of understanding the modeling in the first place can figure out that if the predicted reduction is high by 10% then the difference in the medium-run impact between the reduction scenario and the baseline scenario is going to be about 10%.  Maybe it will be a bit more and maybe a bit less, but that really does not matter.

But an unsophisticated reader (i.e., most everyone to whom the results are touted) is going to interpret that reported uncertainty as being a genuine measure of total uncertainty (just as the same people misinterpret the bounds of CIs as representing the range of possible values that could result from random error).  Never mind that a perfectly plausible estimate of the effect of the policy is a 1% or even 0% reduction in smoking initiation.  When the typical reader sees the reported overly-narrow range of uncertainty, they are tricked into believing that it is the real uncertainty (just as they are usually tricked, by the reporting of CIs, into believing that the only possible source of error is random sampling).

So, basically, the current practice -- some unknown portion of which actually traces back to my work that was about trying to fix the problem of failing to quantify uncertainty -- serves to hide genuine uncertainty by making a mock presentation of uncertainty.  So much for progress.

Agent-based model of THR adoption (and basic case for THR from City Health 2012)

I recently presented a talk on tobacco harm reduction at the City Health 2012 conference in London.  I believe that a video of the actual presentation and ensuing discussion will appear on their website eventually (and I will update this post to link to it).  In the meantime I recorded a voiceover version of the slideshow:

[I will suggest/request that anyone who wants to link to the video please link to this post instead.  I would like to encourage comments and discussion here, and will probably not monitor the comments on the youtube page itself.  Also, there is more background that might be useful.]

The heart of the presentation is a social dynamics model of how THR (e.g., switching from smoking to e-cigarettes) occurs in a community thanks to the education and communication of social norms that come from social interaction.  It starts out with a general overview of THR since many in the audience were not familiar with that.  If you are not interested in the overview, you might want to skip to about 9:30 and just see the presentation of the new model.  (On the other hand, I have been told that it is one of the better existing presentations about the core concepts and justifications for THR.  Not as good as what I presented at the Beirut IHRA conference, unfortunately, but I do not have a recording of that.  So you might want to view that part even if you already are familiar.)

The presentation speaks for itself so I will not try to summarize it here.  But to provide a bit more background on the modeling (and if this is confusing, just watch the video -- it is less technical than what follows, but still explains what you need to know):  There is an interest in predicting THR behavior, in part for obvious reasons, and in part because of a make-work exercise that the US FDA is imposing on anyone trying to promote THR.  As with any modeling of population dynamics, there are various methods available.

The simplest is to just project a trend by extending past numbers.  This is largely useless for anything that involves conscious choices by people, and utterly useless when there are emerging technologies involved.  Despite this, these are the models that are used when people make simplistic predictions about how many smokers there will be 20 years in the future, which others then report as fact.  Such projections about tobacco/nicotine use are perhaps slightly better than trying to project a trend about how many people will be using 11-inch tablet computers 20 years from now, but not much better.

Next simplest is Markov modeling, which basically divides people into different bins (smoker, e-cigarette user, non-user, etc.) and assumes that knowing how many people are in each bin is all you need to know to know about them to determine what happens in the next period (i.e., the next day or year).  This allows for much more robust modeling of some interacting influences, but under the hood, it is still based on projections of population level trends (e.g., what portion of current smokers will adopt THR as a function of how many have already done so).  Allowing for subpopulation-based trends is an improvement over just projecting graphs into the future, as it were, but at its core it is still just a version of that, with all its limitations.

Agent-based models are based on the recognition that the behavior of a population, when considering a decision-based process like THR, is really the aggregation of a lot of individual decisions.  Thus, such models are based on individual actors rather than just population percentages, and the population statistics are emergent properties of the actions of individuals.  The individual decisions are based on economic motives (i.e., considerations of costs and benefits) which are affected by various global factors as well as social interactions.  Individuals can be realistically modeled as having different preferences and other characteristics rather than being all the same.  The agent-based models also allows for social interactions at an individual level -- i.e., people can affect their neighbors and those they encounter, and the results of this may not be the same as treating everyone as if they just have the "average" experience ever period.

The model that we have created is about the simplest model possible that still captures the social dynamics, individual variability, and economic decision making that affects a population's adoption of THR.  It allows for THR adoption to be a social contagion, with someone's chance of adopting it being a function of how much of it they encounter, as well as global forces.  People learn (and their level of learning persists through time) and decide (based on individual motives).  This contrasts with a simple projection or subpopulation-based model, where the future is basically determined by the choice of a single function -- e.g., "P% of the population smokes and that is trending down at a rate of R, so next year the number of smokers will be...." or "if X people have adopted THR in period t, then D% of the rest will adopt it in period t+1, for a total of X+D".  As shown in the video, this produces population outcomes that are not just the obvious immediate result of the choice of those functions.

Update: I discuss some of the implications of this model in the context of anti-THR claims at the antiTHRlies blog.

Update (13 Nov 12): The "live" version of this (the presentation I actually gave in London) has been posted by the conference.  As is usually the case, it is a bit rougher than the studio version, but for those who are are completists (are there any Phillips completists? I doubt it -- I am not even one :-), there it is.  I think there are also some bootlegs, but I don't have them. Unlike most live versions, this one is a bit shorter (the studio version includes a bit more information).