Creating Effective Survey Projects - By Jim Houran and Alan Keith

Why You Should Care about Creating Good Survey Projects

Bad Surveys:
• Can be long and difficult to complete and can alienate respondents.

• Waste your valuable time and money.

• Can fail to identify the most important issues, which then continue to plague the organization.

• Can misidentify unimportant issues as important, leading to resources wasted on ineffective changes.

• Can fail to identify opportunities, leading to difficulty or failure to thrive.

• Can lead organizations can make incorrect and costly business decisions.

• Provide a false sense of objectivity and security that may not be valid, defensible or scientific.

Good Surveys:
• Can be designed to eliminate irrelevant questions, be more valid with fewer questions, and be easier and faster for respondents to complete.

• Ensure your time and money are spent on valid tools that gain expected results.

• Can identify opportunities that make a real difference in the success of your organization.

• Can identify and address your most important issues, allowing you to hire great staff, perform effective employee assessments, gain customer feedback, gain market information and/or accurately predict future behavior.

• Can support your organization in making correct business decisions and allocating time and money where they will make the most difference.

• Give you peace of mind because they are valid, defensible, based on proven scientific theories.

Four Key Elements of a Good Survey Project

1. Conduct a pilot study

Do not assume you already know what people think and why they think so. If you really did, then conducting a survey would be unnecessary, correct? So, instead, try to really listen and keep an open mind. What the question writer thinks is important should be completely secondary to what the intended respondents think. You should always do a pilot study before doing the main study. No matter how well you think you know people or a particular marker, you will be surprised!

2. Design your survey based on modern scientific survey theory

Do-it-yourself survey questions are often selected based on how interesting or appealing they sound - or they are based on the insights of a particular manager in an organization. Amateur survey creators tend to rely on anecdotal evidence or mistaken media reports. A survey constructed based on 'instincts' or experience is rarely able to accurately predict employee or customer behavior.

Survey and questionnaire design is a science. Modern scientific statistical analysis allows behaviors to be predicted quite reliably from the right indicators. Years of research have shown that a systematic, explicit and correct theory about people's behaviors, including factors like respondents' beliefs, attitudes and intentions, is a necessary foundation to create a good survey that actually predicts behavior. Even the simplest and shortest questionnaire needs to take into account all these issues so the right questions can be included or omitted.

For example, you might think that using more answer categories is always better. In reality, most people cannot handle more than six pieces of information at a time. For the best results, do not provide more than six response categories. In fact, to be on the safe side, four categories are probably fine (e.g., 'Strongly Disagree, Disagree, Agree, Strongly Agree'). Do-it-yourself and casual business surveys created by non-statisticians frequently impose an unnecessary burden on respondents and provide invalid answers as a result.

In another example, you may assume that some 'neutral' category (e.g., 'I don't know') is needed to allow people to be non-committal. Studies have shown that neutral categories are usually counterproductive and rarely give you the information you want. The selection of a neutral category often does not reflect uncertainty or indecision, but instead it is used to hide socially undesirable answers. The answer 'I do not want to say' or 'does not apply' may have a completely different - and possibly very important - meaning than 'I agree somewhat.'

There are many types of approaches to survey design, and some are better for business use than others. The 20|20(TM) team often follows Fishbein and Ajzen's Theory of Reasoned Action, because it is flexible enough to be useful for a variety of business and HR applications, while still accurately predicting behavior. Contact us to learn more.

3. Use a valid data analysis method

You might be tempted to analyze questionnaire and survey data with traditional approaches like raw-score sums, percentile rankings or percentages. In fact, many 'professional' survey vendors make this mistake, too! Why not use these standard approaches? Standard analysis and reporting are often incomplete, severely limited and results can even be misleading.

A good example is again neutral categories and the challenges they present at analysis time. Our method of analysis should be able to tell us whether middle categories are used inconsistently and whether someone is giving valid data in the first place. Using the right methodology, middle categories can often be analyzed to provide just such information - information that would be unavailable or misleading when analyzed using standard statistical procedures.

For these reasons, the 20|20(TM) team relies almost exclusively on the use of Rasch scaling (a form of Item Response Theory), a complex but extremely flexible approach to statistical analysis. This approach is unique because:

• Missing data are not a problem, indeed, they are inherently acceptable and provide additional insight into behaviors.

• We can clearly judge the quality of the data and the questionnaire from the responses.

• We obtain truly accurate measurements of the strength of the traits, opinions or other customer variables we wish to study and understand.

• We can determine the extent to which the data are biased by factors such as age, gender or other demographics of survey respondents.

Surprisingly, professional survey companies almost never use Rasch scaling, or other Item Response Theory approaches, due to its complexity and the expertise needed to provide accurate analysis. This means that the business advantage is substantial for organizations who apply these methods in their analytics. Such information allows you to make more targeted and valid business decisions, confident that the survey, the data and the analysis are accurate, predictive and are as relevant as possible to your audience or market.

4. Be ready and willing to take action based on the survey results

Even the best survey is worthless if you do not plan to use it to take action. Although you should keep an open mind about survey results, you should also plan to allocate resources to address the business opportunities identified by a well-designed and executed survey.

For example, the 20|20(TM) team creates 'Action Plans' based on an organization's data. We build a mathematical model of the data that allows us to identify statistical outliers. We then feed this into our software to generate an interpretation for the observed misfit. For personnel assessment, the result is a person-specific and tailor-made diagnostic profile that can be used in a variety of ways.

Action Plans form the basis for the highly successful 20|20 Skills(TM) selection and training assessment. We have also applied it to educational testing where experts now tout it as a form of 'curriculum sensitive' testing. Our team has recently completed the theoretical work to apply this approach to whole groups, too. We can now segment markets and organizations and provide mathematically correct profiles for entire groups and subgroups. The potential for market intelligence and targeted advertising applications is tremendous and wide open. For more information, see our article entitled 'Reading Consumers' Minds...' It is available free at: http://www.hvs.com/Jump/?aid=3030.

Your Professional Survey Team

How should you evaluate a survey vendor to help with your survey projects? An Internet search results in thousands of hits - everything from 'do-it-yourself' survey software to huge multinational companies. How can you be sure the survey team you choose will help you reach your goals and be more successful? Here are some important considerations:

• A trained psychometrician should be an integral part of the survey team.

• The survey team should recommend the survey project should begin with a pilot study and end with implementation of a targeted action plan.

• The survey team has years of real-world experience creating surveys, analyzing results and advising organizations to take action.

• The team uses a valid method to analyze results, preferably Rasch scaling (Item Response Theory).

• The team is small enough to provide customized consulting to identify your unique business challenges and opportunities.

The Bottom Line

Keep these points in mind to help ensure the success of your survey initiatives. Doing so will transform your questionnaires from uncertain exercises to highly effective business tools.

1. Do not assume you already know what your intended audience thinks - maintain an open mind and have the patience to make evidence-based, not belief-based, business decisions.

2. Writing good surveys requires special expertise to provide valid questions and data. If you do not have this expertise, consider investing in professional assistance.

3. Proper questionnaire analysis is as important as good questionnaire design. The most specific and valid findings derive from modern test theory methods, like Rasch scaling. Raw-score sums, percentages and percentile rankings are severely limited and can motivate wrong business decisions.

4. If you are not going to take action, do not bother doing a survey.

5. Choose an experienced survey team as your partner to be sure you get the best value for your time and money.

Million-dollar decisions affecting the long-term success of your organization should be based on solid questionnaire design, a valid and flexible analysis process and a targeted action plan. Doing so can save time and money, but most importantly, allow you to identify critical issues and important opportunities for your organization - and act on them! Finally, you can rest assured that you have made the right decisions for your organization - in the near term for the survey project- and in the long term for the future success of your organization.

Source of Article: Restaurant News Resource