Potential Problems in Surveys: If you can, you should watch for a number of potential problems that might arise in a survey:
• The definition of the sampling method, if not mentioned, can be a cause for suspicion.
• Often people sampled cannot or will not participate in the survey. The more people have refused, the more suspect the results.
• Often the survey taker will arbitrarily determine who is to be sampled and what is to be learned about the sample.
• The nature and order of questions can bias a sample, so have a look at the actual survey questions and their order.
• If a survey is undertaken over weeks or months, views can change, creating an inconsistent sample.
• If methods of data collection change (e.g., telephone vs. personal interview), results could be inconsistent.
• A common error in many surveys is that findings are extrapolated from a sample rather than obtained directly from the entire population.
• Often the overall sample may be large enough to be significant, but stratified sub-samples will be insignificant.