Mr. Heisenbug posted recently about some differences between Tina Saey’s results from American Gut and uBiome. In Tina’s results, where she submitted contemporaneous samples (two swabs from the same toilet paper) to both uBiome and American Gut, she found the ratio of Firmicutes to Bacteroidetes to be inverted.
We found this quite interesting because we’ve wanted to compare our sample results to American Gut for a long time since we use slightly different methods and would like to see how comparable our results are. It’s known that different extraction techniques (Guo and Zhang 2012 and Henderson et al 2013), primers, collection techniques, and other factors can cause huge differences in results.
We’re still investigating, but we have several ideas about why the differences in Tina’s samples might have occurred:
1. Extraction technique. There are numerous steps in the extraction process, and many of them could potentially lead to variation in the final result. While we have automated extensively (robots!), we use a very similar pipeline to American Gut, based on the Human Microbiome Project protocols. We do, however, use a different extraction kit. (This is super technical, but basically, we extract the DNA using a lab kit manufactured by a different manufacturer than American Gut. There are several industry standard extraction kits that are used in a variety of studies.)
Different sample collection and DNA extraction techniques can lead to significant variation. The kit we are using to extract DNA has been used in numerous studies of the human gut microbiome (Eloe-Fadrosh et al, 2013; Flores et al. 2012; Michail et al 2012; Rigsbee et al 2012). At the phylum level, the results of these studies using the same extraction kits have been generally consistent with uBiome results.
To test this idea, we examined Tina’s samples from the American Gut Project and corrected it using estimated extraction bias as observed in Guo and Zhang 2012 between the two kits. As you can see, the results are much more similar. We will be running some comparison studies that will help isolate this difference and perhaps replicate these results with a larger sample set.
Tina Saey’s Sample – Bias-Corrected American Gut and uBiome Numbers
2. Reproducibility. Reproducibility is a huge issue in science and a significant proportion of research has been found not to be replicable. See, for example, the efforts of the Reproducibility Initiative in cancer biology, the Reproducibility Project in psychology, and a fantastic special in Nature, Challenges in Irreproducible Research.
Our data, as shown below, are highly reproducible. That is, if you put the same sample through the pipeline multiple times, you will get the same result. Following standard validation methods, we computed some statistics about the reproducibility of our results. In a preliminary comparison with 10 samples (each with about 75 taxa), results are shown in the graph below. Re-running this on a larger data set, with 61 samples, we saw a coefficient of variation of 5.76% with a standard deviation of 3.79%. What this means is our results are fairly internally consistent, which means that uBiome samples can be compared to each other successfully (even if they differ from the results from other methods).
3. Sample mix-up. The tubes we use are pre-barcoded and associated with their registered kit in our dataset. The whole system is automated, making sample mixups extremely unlikely. Still, it is possible that there was a simple mix-up of samples.
4. Overgrowth. Our sample collection techniques are based on the Human Microbiome Project, and we have also added special buffers and stabilization solutions that lyse (break open) the bacteria, which prevents them from growing as they travel to our lab. It also makes them shelf-stable for months — so mailing delays are unlikely to be the problem. However, differences in sample transport and collection technique could have caused an issue.
In any case, we look forward to doing some follow-up research to learn more about why these differences are occurring. As we learn and explore more about the microbiome, it is our goal to create more valuable and comparable data that will help us all to unlock the potential of the microbiome for humanity.