Search

ELT Times Search
  • Preparing mainstream teachers for English-language learners: is being a good teacher good enough? October 8, 2010
    Introduction More and more teachers find themselves teaching students from increasingly diverse linguistic and cultural backgrounds. In a recent report (National Center for Education Statistics, 2002), 42% of the teachers surveyed indicated that they had English Language Learners (ELLs) in their classroom, but only 12.5% of these teachers had received more […]
  • Teaching English to prostitutes in China October 6, 2010
    By Robert Vance “Quite a few of your English students are prostitutes,” a friend told me today as she recounted a conversation that she had with her hair stylist recently. “The guy who cut my hair told me that many of your training center’s female students come to him two or three times a week to [...] […]
  • Without 1, where would we begin? Small sample research in educational settings September 29, 2010
    I study preservice teachers and the ways they attempt to make sense of method course instruction (theory) and real classroom applications (practice). Given the complexity of completing this task my chosen sample size has always been quite small. Coming out of graduate school, I actually thought that what I learned about qualitative research made sense. [...] […]
  • Writing research papers… alternative options September 29, 2010
    Students are turning to alternative methods to solve their time management dilemmas. Employing the services of online research paper sites is becoming an ever more popular solution. Students have a heavy load of work to complete for which they have to study day-and-night, in addition to attending supplementary studies in order to achieve high grades. A [...] […]
  • TEFL training courses – accreditation and certification September 15, 2010
    Every TEFL teacher training course should be accredited with “accredited” meaning that an outside institution has reviewed the course, course content and the trainers delivering the course. Accreditation is important for teachers looking to enroll because it is a way of telling that the TEFL course meets a minimum of standards with regards to [...] […]
  • International TESOL training and EFL contexts: the cultural disillusionment factor. September 14, 2010
    Md. Raqibuddin Chowdhury’s article reports on a study examining the implementation of communicative language teaching (CLT) in Bangladesh in general and at the University of Dhaka in particular. When CLT was first introduced across Europe, the English as a foreign language (EFL) context in which it would inevitably be applied was not considered. Here univers […]
  • Exploring a new pedagogy: Teaching for Intellectual and Emotional Learning (TIEL) August 31, 2010
    The role of teacher educators is to develop the capacity in pre-service teachers for complex teaching that will prepare them to create and teach in “learning communities [that are] humane, intellectually challenging, and pluralistic” (Darling-Hammond, 1997, p. 33). To establish and maintain such learning communities, however, requires knowledge of intellectu […]
  • Teaching factual writing: purpose and structure August 26, 2010
    David Wray and Maureen Lewis remind us of the need to focus on the teaching of factual texts in primary classrooms. They offer one particular teaching strategy, ‘writing frames’, trialed by teachers in the EXEL (Exeter Extending Literacy) Project, as a useful strategy in assisting young writers learn to write factual texts. Introduction As members [...] […]
  • Who qualifies to monitor an ESP course: a content teacher or a language teacher? August 24, 2010
    As it is known, ESP materials are developed in order to respond to the specific needs of English learners. ESP is a branch of applied linguistics in which investigators attempt to put their fingers on the specific needs of individuals or groups of individuals in English in order to design materials related to their specific [...] […]
  • Defining whole language in a postmodern age August 22, 2010
    Can whole language be ‘defined’ in the true sense of the word? Lorraine Wilson believes that while whole language can never be ‘defined’ in the sense suggested by the word’s Latin root (definire = to finish, finalise), certain core principles and assumptions can be made explicit. In this article she describes how a group of [...] […]

Recent Comments

Without 1, where would we begin? Small sample research in educational settings

I study preservice teachers and the ways they attempt to make sense of method course instruction (theory) and real classroom applications (practice). Given the complexity of completing this task my chosen sample size has always been quite small. Coming out of graduate school, I actually thought that what I learned about qualitative research made sense. I still do. However, after five years in academe it appears to me that some researchers still hold onto quantitative ideologies when critiquing qualitative research manuscripts. I have had two manuscripts turned down for publication not because my findings were in question, but because I used an N of one. The reviewers’ comments all pointed out that because I used an N of one my findings were suspect. In other words, (quantitatively speaking) they were not valid, reliable, and generalizable.

What exactly is a valid and reliable study? Is it one that is capable of producing generalizations applicable to certain populations based on sampling methods? What would this study look like? How are the key terms–validity and reliability–addressed? And what are the limits of the generalizations produced? Or, are these questions more appropriately applied to the uniqueness of different methodologies that are relative to the study itself?

ResearchIn the following article I address the previous questions and provide literary evidence for the non-universality of the terms validity, reliability, and generalizability. Instead of concrete rules applied in a rigorous and rote method; better, would be a choice of theoretical implications that dictate how these three terms should be applied relative to the unique research question at hand.

The issues surrounding research methodology are complex and diverse. The term research itself denotes an applied procedure capable of producing knowledge: White coats; lab equipment, and disheveled scientists deeply involved in manipulating variables, recording and observing outcomes. For the researcher in educational environments, the previous illustration may seem slightly out of context. But is it? The seen itself may be, but the theoretical methodology being applied is quite appropriate. As Borg and Gall state, “educational research has been built largely on the research traditions and methods that were initially developed in the physical and biological sciences” (1989, 379). And it is these methods that still influence arguments concerning the appropriate research methodology for use in educational settings. In Maxwell’s critique of experimental methods in causation studies, he reports that these methods are still considered the “gold standard” (2004, 3). Borg critiques the use of this research paradigm in social science because, “human beings, the usual subjects in educational research, are much more complex organisms than the subjects studied in other sciences. . .” (1987, 154). Because of this inconsistency, the possibility of using different research approaches to understand the complexity of issues encountered in social science was called for and answered in the form of a more interpretive approach to research–qualitative methodology (Krathwohl, 1993). “Researchers need all the help they can get, and qualitative methods have an important and useful place among the research methods” (Krathwohl 1993, 314).

Qualitative Research

Qualitative research has a multitude of inquisitive methods with understanding as the keystone of construction. As Peshkin put it,

To qualitative researchers, what is to be learned does not invariably necessitate a particular study design involving theory, hypotheses, or generalization, though it may. It necessitates a judgment that leads them to decide what research designs they should frame to produce one or more of many imagined and yet unimagined outcomes … (1993, 23).

This idea emphasizes that there is much we do not know, and without the ability to “imagine” other possibilities, educational researchers are condemned to a life of walking through tunnels rather than exploring caves.

Many researchers dismissed qualitative methods on the grounds that they would not yield lawful generalizations. This objection softened when it was understood that “not even quantitative methods … yield valid and reliable … generalizations” (Fenstermacher 1986, 42). In fact, qualitative inquiry “legitimates a concern with understanding particular situations … rather than” viewing only surface variables of the most general contexts (Maxwell 2004, 9).

Sample Size

The theoretical argument most often presented in opposition of small samples sounds something like this: You need a large enough sample pulled from a representative population so that the generalizations offered can be valid. This may be true of quantitative methods based on theoretical sampling maxims, but sampling in qualitative research can be quite different. Qualitative researchers using small sample N’s use a different approach in selecting samples than researchers concerned with extending generalizations to other populations (LeCompte & Preissle 1993). “In educational research it is almost never possible to study an entire population” (Borg 1987, 139). In defense of small sample size in educational research, Borg and Gall state that,

In many educational research projects, small samples are more appropriate than large samples. This is often true of studies in which role-playing, in-depth interviews, projective measures, and other such time consuming techniques are employed…. A study that probes deeply into the characteristics of a small sample often provides more knowledge than a study that attacks the same problem by collecting only shallow information on a large sample. (1989, 236-237)

So it is not only possible to use a small sample, but sometimes it is preferable. This idea should not seem so controversial. Even in medicine the idea of a single case is seen as a valuable investigation that can offer in-depth insight into the treatment of illness that is often overlooked in large-scale studies.

Wolcott, a bit more critical of “larger is better,” offered this insight,

The preference for larger N’s is a legacy from quantitative research, where a small number of cases can seriously undermine the press for generalization. How fortunate some researchers are to be able to establish sample size by means of formula! (p. 181)

The use of a small sample may benefit qualitative inquiry because it permits the researcher access to deep caverns that call for detailed exploration and mapping. Let me illustrate: An exploring spelunker is sent to map out a cave. Preliminary exploration may yield a multitude of tunnels, all of them offering an exit. Should she map all of the tunnels in one attempt? Or is each tunnel explored separately? The task for the spelunker now becomes a choice. Does she rashly map out the tunnels, therefore neglecting to detail any dangers or wondrous sights of nature all in the order of finding the fastest way out? Or does she take each tunnel as an individual means of exit, describing in detail any and all characteristics–so that spelunkers who follow will be aware of the intricacies involved in exiting? Wouldn’t it be dangerous to just tell future spelunkers that tunnel (A) is the fastest exit, neglecting the pertinent details. One such detail the mapping spelunker left out was to point out that the future explorers need to be able to hold their breath for three sustained minutes so they can swim the underwater cavern that permits access to the exit. Therefore the choice of sample selection can be purposely chosen, so that all the important details can be highlighted. Knowing this to be the case using a small sample purposely chosen so as to present detailed descriptions of phenomena is appropriate.

Validity

Arguments concerning validity tend to preoccupy most critics of small sample research. I would like to start off with an extreme response from Harry Wolcott concerning validity and then ease his position by addressing validity more positively in qualitative inquiry. Wolcott (1994) states that, “validity does not seem a useful criterion for guiding or assessing qualitative research” (337). This stance appears quite spiteful of validity, but Wolcott does offer a loosely arranged method to be applied by the inquirer using small sample research that can preserve validity.

(1). Talk little, listen a lot; (2). Record accurately; (3). Begin writing early; (4). Let readers “see” for themselves; (5). Report fully;(6). Be candid; (7). Seek feedback; (8). Try to achieve balance; (9). Write accurately. (1994, 348-354)

I present this first because of its more human appeal. The plain English is an applicable way of establishing validity in qualitative inquiry using a small sample.

If this is a method for achieving validity what do other researchers have to offer the discussion? One procedure is adherence to a select philosophy of context so that any validity concerns can be overlaid with the philosophical framework (Cronbach 1982; Krathwohl 1993; LeCompte & Preissle 1993). In the context of small samples Peshkin states that, “verification” (27) of study results by participants can bolster the validity of the study. In other words, allow those being studied access to interpretations so they can confirm the accuracy of the researcher’s portrayal.

What about internal and external validity? Borg states,

All the researcher can do is weigh the advantages of rigorous control of extraneous variables against the advantage of doing research in natural educational environments and come to a compromise that provides acceptable levels of internal and external validity. (1987, 229)

So in order to achieve internal validity, verification with participants can be used. External validity can be achieved by extending findings to philosophical foundations and by helping to further develop existing theories. Also, it should be noted that, “understanding is a more fundamental concept for qualitative research than validity” (Maxwell 1992, 281). Maxwell (1992) goes on to quote Brinberg and McGrath (1985) as stating that ‘validity is not a commodity that can be purchased with techniques…. Rather, validity is like integrity, character and quality, to be assessed relative to purposes and circumstances (280-281). While validity can be explained or achieved by diverse methods–How does or do these answers apply to generalizations in small sample research?

Generalizability

Again the term generalizability in the context of quantitative methods refers to a study’s potential to affix its findings outside of the study to a target population. The structural rules that apply to generalizability based on sampling techniques may not be a useful guide when conducting small sample research. The single generalization sought after by most quantitative methods is not a goal of small sample research that encourages, “multiple meanings” (Krathwohl 1993, 323). However their are ways to achieve generalizability with small samples, but the term creates a varied methodology. Generalization in small sample research can occur within the study that is based on a theory. This form of generalization as Maxwell declares, “is normally based on the assumption that this theory may be useful in making sense of similar persons or situations, rather than on an explicit sampling process and the drawing of conclusions about a specified population through statistical inference” (1992, 293). The researcher using small samples relies on interpretation which is the crucial aspect in generalizability (Firestone 1993; Peshkin 1993; Wolcott 1994). Interpretation provides readers with a unique insight of research settings Cronbach (1982) and Firestone (1993) point out one other aspect concerning both conventional arguments for generalizing–they are of limited utility for making decisions in specific cases.

Also, it may be that small sample research does not try to present definitive reality but uses purposive samples to demonstrate the plausibility of multiple realities (Erikson 1986). It should not be assumed, however, that generalizability in small sample research is a purely negotiable trait without any understandable technique to help assure the different forms of generalizability discussed above. Triangulation in purposive sampling helps preserve the validity of any generalizations and can be documented with three forms of triangulation; Data triangulation, using multiple data sources across time and space; investigator triangulation, using multiple investigators; and method triangulation, using multiple methods (Krathwohl 1993).

So it is possible by use of triangulation that generalizability can be achieved in small sample research, but it should also be noted that in small sample research the purpose, extent, and technique or theory all create different interpretations of generalizability.

Reliability

Although I lumped validity and generalizability together without mentioning reliability, that does not mean reliability is a mute point in small sample research. But again it should be warned that any reference to reliability should be disassociated with quantitative terminology. Wolcott notes that reliability is “almost exclusively associated with testing” (1994, 343), this association is not appropriate with small sample research. The concept that research methodology applied over time and settings should yield extremely similar results to meet the criterion of reliability is not always true in qualitative inquiry. As LeCompte and Priessle echo, reliability poses some dilemma’s when applied to naturalistic observation or participant interviews because of the near impossibility of replicating results in a setting that is a unique setting in its own right (1993).

In order to apply reliability, Bodgan and Biklen (1992) see small sample reliability more as a fit between recorded data and the reality of situations under study. Because of the problem of reliability in small sample research a disassociation with quantitative reliability should be adhered to with the variability associated with purposive sampling dictating a flexible attachment to reliability. What about the spelunker? I make reference back to this analogy to bolster my assertions. There are two positions that can be supported by this analogy. Choosing one tunnel may be successfully explored by using different tactics or searched for specific characteristics. Spelunker (A) and Spelunker (B) use the same tunnel in exiting the cave, however, Spelunker (A) reminisces the vivid colors produced on the tunnel walls from the filtered sunlight close to the exit area. Spelunker (B) tells of the jagged crags and the skill needed for the successful negotiation over and around the razor sharp edges. Which spelunkers’ interpretation is a reliable construction? Or are they both reliable relative to the individual’s lens of perspective?

Summary

So are there advantages to using small samples in research studies? Are there limitations? Yes, as Borg puts it, “the fact that most studies have limitations does not mean that they are without value” (1987, 159). Validity, reliability, and generalizability can be achieved in small sample studies, but a reconceptualization of the use of these terms are needed. There is ample evidence that from observations and other preliminary data that tunnel (A) is the shortest and should therefore permit a speedy return to the surface. Translated to research methodology: The researcher would be testing a hypothesis. But what if the cave and all of its tunnels present a virgin opportunity for exploration? The spelunker has the opportunity to construct a detailed survey of the inner workings of the cave. Translated to research methodology: The researcher would be hypothesis generating. If we want to understand educational contexts, should we not understand the uniqueness and the complexity of those contexts? The question is not one of hypothesis testing, but one of hypothesis generating. “Conceptualizing the boundaries of whatever the researcher wants to study is the … first task” (LeCompte & Preissle 1993, 59). “We are able to select a purposive sample that includes individuals, … situations, … and other aspects that can further develop the target area of information” (Krathwohl 1993, 324).

Researchers have to keep in mind that “every method of data collection is only an approximation to knowledge. Each provides a different and usually valid glimpse of reality …” (Warwick 1973, 190). In the current oppressive era of No Child Left Behind and rigid accountability in public education, educational researchers need to value each other’s contribution to understanding teaching and learning. We need to unite to help policy makers and the public understand the harmful effects of believing in a system that only allows for a single vision of reality. In closing I point to Peshkin to offer a final defense of small sample research,

I think we would benefit from ’stories’ of the several types of persons who participate in the educational enterprise. I visualize biographies, life histories, or case studies addressing, … teachers … who succeeded and fail, with success and failure variably defined. (1993, 25)

References

Bogdan, R. C. & Biklen, S. K. (1992). Qualitative research for education: An Introduction to theory and methods. Needham Heights, MA: Allyn & Bacon.

Borg, W. R. & Gall, M. D. (1989). Educational research: An introduction. 5th edition. White Plains, NY: Longman.

Borg, W. R. (1987). Educational research: A practical guide for teachers. 2nd Edition. White Plains, NY: Longman.

Cronbach, L. J. (1982). Designing and evaluations of educational and social programs. San Francisco: Jossey-Bass.

Erikson, F. (1992). Why the clinical trial doesn’t work as a metaphor for educational research: A response to Schrag. Educational Researcher, Volume 21, Number 5.

Erikson, F. (1986). Qualitative methods in research on teaching. In J. P. Shaver (Ed.), Handbook of research on teaching, New York: Macmillan.

Fenstermacher, G. D. (1986). Philosophy of research on teaching. In J. P. Shaver (Ed.), Handbook of research on teaching (pp. 37-49). New York: Macmillan.

Peshkin, A. (1993). The goodness of qualitative research. Educational Researcher, Volume 22, Number 2.

Firestone, W. A. (1993). Alternative arguments for generalizing from data as applied to qualitative research. Educational Researcher, Volume 22, Number 4.

Krathwohl, D. R. (1993). Methods of educational and social science research: An integrated approach. White Plains, NY: Longman.

LeCompte, M. D. & Preissle, J. (1993). Ethnography and qualitative design in educational research. San Diego, CA: Academic Press.

Maxwell, J. A. (2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, Volume 33, Number 2.

Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, Volume 62, Number 3.

Warwick, D. P. (1973). Survey research and participant observation: A benefit-cost analysis. In D. P. Warwick & S. Osherson (Eds.), Comparative research methods (pp. 189-203). Englewood Cliffs, NJ: Prentice Hall.

Wolcott, H. F. (1994). Transforming qualitative data: Description, analysis, and interpretation. Thousand Oaks, CA: Sage..

Timothy D. Slekar

Penn State Altoona

VN:F [1.9.5_1105]
Rating: 10.0/10 (2 votes cast)
VN:F [1.9.5_1105]
Rating: +2 (from 2 votes)
Without 1, where would we begin? Small sample research in educational settings, 10.0 out of 10 based on 2 ratings
Sphere: Related Content

Share

3 comments to Without 1, where would we begin? Small sample research in educational settings

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Get Adobe Flash playerPlugin by wpburn.com wordpress themes