Negative Feedback

Tag: bias

  • What does “it works” mean?

    Response to my rants about the hologram bracelets and articles like this one about magical thinking on SciAm have forced me to re-evaluate my language. I would like to say that the hologram bracelets “don’t work” because they have no physical effect on the phenomena that they claim to affect. However, people who use the bracelet will probably have a measurable improvement in their performance as per the SciAm article. So what to do?

    Ever since reading Ben Goldacre‘s Bad Science, I’ve been fascinated by the term “no better than placebo”. The placebo effect is one of the reasons for using blind testing and shows what kind of rigour needs to be employed to avoid being fooled. Now, the idea is that a medicine is only really effective if it works better than placebo. Is there a corollary that says that if a medicine will not be particularly effective, it is better to attempt placebo treatment? I suppose this makes a kind of sense if the medical solution has physical side effects but the placebo doesn’t. I just can’t bring myself to think that I should start believing in crystals and holograms so that I can have cheap placebo improvements to my performance. Is this why superstition persists? I suppose it should be no surprise that it would have been selected away unless it conferred some advantage.

    Unfortunately this brings us to an uncomfortable point. If the superstitions really do confer advantages (even if only mental) what is the sceptic’s method of achieving similar benefits for similar (low) cost? It costs nothing to wear gold thongs to important interviews – what should you do if you honestly don’t believe the thong will work? In fact, what does “work” even mean in this context? I hate semantics.

  • Productivity: perception vs reality

    I am haunted by the thought that somewhere someone is doing what I am doing more efficiently. I pride myself on knowing efficient ways to do things. Note that this is somewhat different from “being efficient”, which in many cases involves not learning new ways of doing things you will probably just do once. I have slowly learned that I am fooling myself when I figure out ways in which to do things better next time, as there probably won’t be a next time and my time is largely wasted.

    That said, the feelings aren’t going away, and in fact I have learned things that make it even more difficult to figure out what’s going on. Witness this discussion – the meat of which is that keyboard shortcuts feel faster but mousing is actually faster as measured by a stopwatch. Oh no! Now I need to time myself doing common tasks instead of relying how productive I feel? This brings up an even more interesting question: What is more important to me day to day, being productive or feeling productive?
    The sad truth is that I have come to the conclusion that much of my investment into learning lots of different ways of doing things and even the environments that I enjoy the most have been focussed on making me feel productive rather than improving my actual productivity. Unfortunately I am now experiencing the sunk cost dilemma. I have spent almost 9 years of my life learning (and I stress that I have aimed to become proficient rather than just getting by) Linux, grep, sed, awk and all the unix tools and gnu plotutils so that I can do tons of stuff from the terminal. I have learned Emacs, memorising its arcane and non-transferrable shortcut keys and how to interface it with the shell commands. In Emacs I have learned AUCTeX for latex editing, where I often write code for pictures. I also use org-mode a lot for task management. I suppose my investment into learning Python was the least contentious of these decisions. On the way I have also shed Matlab and ignored powerful but expensive tools like Mathematica in favour of SciPy and Maxima, and lately Sage. I have replaced Simulink with Modelica.
    I am now beset by doubts and I wonder if I shouldn’t have taken the blue pill after all and when I started getting into Open Source when I graduated, just continued with my Windows box, installed Visual Studio and rekindled my original love for C++, used MS Word for my Masters and Matlab or even Excel for all my data processing, or perhaps laid my hands on Mathematica or MathCad. Would I not be more capable now and would I have not gotten more things done in the end by not going against the flow?
    Unfortunately there is no easy answer. I am typing this on a MacBook Pro, which I love because Mac OS X has a nice unix layer beneath it but also because there are less hardware issues than I have endured using Linux. I have not yet been successful in using XCode to do anything meaningful because I don’t want to write an app that I can’t use on my Linux box at work. I have found myself developing code that I am forced to maintain and run simply because I am the only guy on my floor that has Linux on his machine and I have not wrapped the stuff in a GUI so that our secretary can run it. So I experience the need post facto of being able to create GUI apps and have other people run them, and of having a common environment that people can help me with.
    However, every time I look into the “more productive” way I feel like gouging my eyes out. I hate word processors of pretty much every description, as the whole philosophy of “when it looks right it is right” galls me, and figures don’t float – what’s up with that? When I try to use Eclipse for development, I am up against the problems of using such environments for dynamic languages, in addition to the fact that all my current projects’ directory structures don’t lend themselves to Pydev’s idea of what a Python project should look like. In fact, when I try to use statically typed languages, I feel totally put out by having to think of the type information in the first place. I cannot find an easy way to get from where I am now into the mainstream, and it annoys me.
    I have vowed to stop looking around for now, until I have finished my PhD, but once that is done, should I become a good little MS drone? Have I wasted my time learning these skills or can I justify my rationalisation that I am more productive now? Does that even matter when I feel more productive (and happier) when I am using my current environment?

  • Don’t trust science, look for evidence.

    My wife has the flu (along with several other people I know), so it’s time for the usual recommendations from her mother to use a lot of vitamin C. Bad Science has fanned the flames of my natural combative streak, so I went out and tried to find a good study for the Vitamin C thing. I found this, and sent it on to the mother-in-law (who hasn’t replied yet). I also posted it in reply to a friend on Facebook’s status about dosing up on vit. C. Not long after that, a friend of hers replies with this link — implying, of course, that anyone can prove anything by linking to stuff on the Internet.

    So, of course, I reply, mentioning the number of people involved in the trials I had quoted, the significance of the results etc, which are all in the article and their links, while the other link has some hand-wavey trials with very little real evidence. And while I was posting this reply (which I’ll quote, because I’m lazy):
    “Your point was quite clear, which is why I had to reply: It’s very popular at the moment to roll one’s eyes and point to all the contradictory evidence on the internet, typically ending in “so in the end we don’t know as much as we think” equivocation of the options. The reality is that in many cases we do know quite well what is happening (as in the case with Vit. C), and the underlying meme that science is always to be taken with a pinch of salt is damaging to proper understanding of scientific method and results.

    This story repeats itself so many times that people become less likely to investigate the real results themselves (note that I’m not saying we should just trust scientists, we should be capable of understanding and evaluating the results). This goes for antiretrovirals, vaccines and many other treatments that have been shown to work, but are being used less than they should be because of general public distrust.”

    Of course, I can think of many other examples: Cell phones and/or their towers causing cancer (which is silly even on the face of it considering the power of the transmitters, but has also been pretty solidly disproved by long studies of cancer incidence), stuff in plastic water bottles causing cancer, overhead power lines, induction stoves causing sterility (same argument as with the cell phones). I think there is a general distrust of science, perhaps because people trusted science so much in the past and have become disillusioned.
    So here’s the thing: science isn’t really a trust thing. The whole point of science is that the evidence should be so overwhelming when you make a positive statement that it cannot be avoided. In fact, many scientists would say that no hypothesis can ever be proved, we can just fail to disprove it after trying very hard. Reading popular media summaries of scientific work makes this difficult because headlines like “Scientists do many tests and find insufficient evidence to disprove this hypothesis” don’t really sell papers. It also doesn’t gel well with the way people talk and think in general. The problem is that most of our decisions every day are made in extremely unscientific ways, including a wide array of effects that mean we can’t even reason accurately about our own experience. Trust is a proxy for research. If someone I trust a lot (or a source I trust a lot) says something, I am unlikely to repeat their experiments and likely to take them at their word. If I know something about the field in question, I can also interrogate my understanding of the situation to see if the results are consistent with other things I know to be true. In an ideal world, we would all be repeating all the experiments to make sure they work. In reality, we lack the time, skills and funds to do so, so trust is necessary.
    Where does this leave us? We need to base our trust on good, objective measures. In science, this means being able to look at the results and weigh the evidence, not the authors or publishers. It means checking consistency of theories and avoiding repeating claims that have poor statistical significance. It means doing real science, and not cargo cult science. More on that later.