A scientific method of research requires a study to be based on evidence and to lack any possible bias. Consistency and precision are the essential values that suggest that evidence generated with a tool that possesses such qualities genuinely reflects reality (Amirrudin et al., 2020). The Values and Motives Questionnaire (VMQ) provides a suitable example of a test that requires further tuning due to the methods utilized by its authors when they tried to prove its validity. This paper will analyze the VMQ and the basis on which its authors claim reliability for their appropriateness.
Types of Reliability and Validity
Creating a viable test requires researchers to ensure that it is applicable to any situation in which such a tool can be employed. The reliability of measurement instruments can be assessed through test-retest probes and by proving their internal consistency (Values and motives questionnaire: The technical manual, n.d.). It is also crucial to provide mathematical proof of a study’s appropriateness. Relating tests’ values with descriptions of what they represent is called criterion validity while interpreting a questionnaire’s relevance to its specific scope (Values and motives questionnaire: The technical manual, n.d.). The VMQ presents a questionable application of these notions and requires further investigation.
The manual does establish how its objectivity can be assessed, yet does not follow its recommendations as required. While there are several types of reliability and validity proofs, the questionnaire utilizes only a portion of the provided tools (Values and motives questionnaire: The technical manual, n.d.). The authors attempt to show a correlation between an external criterion and the test’s measures. The outcomes are used to prove that the manual is applicable by confirming that the internal consistency coefficient is satisfactory.
Cronbach’s Alpha Coefficients
This questionnaire utilizes internal consistency reliability, making it rely on Cronbach’s Alpha. Misapplications of this coefficient are common in modern scientific communities, causing numerous studies to be non-generalizable (Amirrudin et al., 2020). Many values in this manual revolve around a 0.7 coefficient, which is acceptable to be assumed as an average result for each measurement except for achievements and inconsistencies. The low category should be considered below 0.65, and the high is above 0.90. The two exceptions must be viewed at a lower point, although a good test has at least a 0.65 reliability score, making 0.05 the maximum allowed deviation for these scales’ ranges (“What makes a good test?,” n.d.). However, the VMQ contains five scales below average, putting their reliability close to unacceptable. Even within its flawed take on the population, the VMQ’s Cronbach Alpha is far from an ideal point on several value scales (Values and motives questionnaire: The technical manual, n.d.). Eight remaining coefficients are within the acceptable reliability category, with two of them close to high.
Sample Size and Nature of the Population
There are issues with the validity of this questionnaire due to the population sample utilized for its assessment. It is critical to understand that there are generational specifics that affect the average scores in such a test. For the VMQ, a small sample of students from a non-randomized population is selected (Values and motives questionnaire: The technical manual, n.d.). This factor causes a high level of bias, which is further fueled by the fact that no population data is released. The reliability coefficient requires a more variable sample to be viewed than the one provided by the authors, as the impact of individual factors is decreased with the rise of a more diverse population (Amirrudin et al., 2020). Using only a single generation and further limiting it to a single course in a university does not lead to creating generalizable test material. At the same time, other tests included as examples in the VMQ manual are also visualized through small populations of Psychology undergraduates (Values and motives questionnaire: The technical manual, n.d.). I believe that such an approach causes severe problems with the accuracy of the results.
Furthermore, with no data on the cultural, socioeconomic, and ethical backgrounds of the population that was used to prove the validity of the questionnaire, it is impossible to draw conclusions. Moreover, the gender percentage does not represent the general public due to an imbalanced scale (Values and motives questionnaire: The technical manual, n.d.). Therefore, the validity of the VMQ is put into question due to its described flaws in testing.
Conclusion and Personal Opinion
In my opinion, the VMQ has the potential to become a helpful tool on par with other psychometric options. However, it must undergo additional reviews for fine-tuning its values and judgments derived from one’s results. I do not believe that the authors have proven their paper’s soundness. There are appropriate methods for verifying the reliability and validity of a study, which the paper outlines yet fails to incorporate adequately into its processes. Not only did the VMQ use only a single measure for its validity and reliability, but the method that was employed to perform this action was also flawed due to the improper sampling technique. When a population is limited to such a point, it is not representative of the actual average values that the questionnaire tries to use in its foundation.
References
Amirrudin, M., Nasution, K., & Supahar, S. (2020). Effect of variability on Cronbach Alpha reliability in research practice. Jurnal Matematika, Statistika dan Komputasi, 17(2), 223-230.
Values and motives questionnaire: The technical manual. (n.d.). Psytech. Web.
What makes a good test? (n.d.). Psychological Testing. Web.