Saturday, August 24, 2013

Webster & Watson

Webster and Watson. 2002. Analyzing the past to prepare for the future: Writing a literature review


  1. 1. My first question is about prospective authors and topics. According to this article, there are two types of reviews exist: 1) a mature topic where an accumulated body of research exists 2) an emerging issue that would benefit from exposure to potential theoretical foundation. Both of the two types seems to have something to do with experienced researchers. But for us, to be more general, students who are just going into this field, which of the two would be better for us to choose as literature review topic? What factors lead you to make the decision of the choice?

    2. My second question is about identifying relevant literature. Basically there are three ways: 1) major contributions are addressed in leading journals. 2) review the citations for prior articles. 3) Search for articles those cite the key articles identified in the previous steps. But when you are researching on a topic, sometimes you will find yourself overwhelmed by the large amount of articles. For example, if you identify one article is valuable for your review, then you would like to check all the citations. Then for the papers in the citations, each of them has several more citations you need to check. For this process on and on, you seem to be trapped into an infinite loop. So my question is how do we decide where to start to ensure the quality of literature review and where to stop to ensure not losing valuable information left unchecked?

    3. My third question is about the reviewing and revision process. In most cases, the reviewers’ response is valuable for you to revise your paper. However, sometimes, the concerns that the reviewers insist on may be unrelated to your research field, or the reviewers misunderstand your paper to some degree. In that case, how would you response to the reviewers’ concerns? Or one step further, how could you revise your paper to alleviate such kind of concern?

  2. 1) In the structured approach to determine the source material for the review, the author states that “Major contributions are likely to be in leading journals”. Although this statement seems true, it would also help us to evaluate the amount of assumption in the statement. How does a reviewer rate the content of a research work without a journal bias? How likely is it for a not-so-famous journal to come up with a path breaking research concept? If we follow the structured approach as stated in the paper above, don’t we increase the risk of missing a relevant research work, just because it has not found its way to a leading journal?
    2) The author also points out the importance of synthesizing the literature by dividing the literature into concept matrices. But in reality, it is more likely to find overlapping concepts i.e, a particular work might be a composition of different concepts at work and it might not always be possible for the reviewer to divide the work into different concepts. How can a literature be synthesized into concepts if the proposition contains several intertwined concepts at work?
    3) Under the topic “Evaluating Your Theory “ (page 8), the author discusses about the need for a theory to be interesting, parsimonious, falsifiable, useful and be built from multiple paradigms. Should a theory necessarily contain one or many of these characteristics? Do these characteristics essentially differentiate between a good theory and a bad one? What is the necessity for a theory to be falsifiable?

  3. 1.Do you agree with Weick (pg. xx) that good theories must "provide answers to why"? I feel that good research contributes to theoretical progress by tickling the boundaries of its discipline(s) and providing avenues of further research and learning. With that in mind, don't you think that research which includes a systematic and (almost) all-encompassing literature review, and instead goes on to pose new questions or present previously unseen findings for future deliberation should be considered good research? The latter also seems to align with what MIS Quarterly Reviewers are looking for (pg. xxi).

    2.An MISQ reviewer noted (pg. xvi) the importance of “searching by topic across all relevant journals.” IS is an evolving field that references / relies on many other fields - like Computer Science, Psychology, Art, etc. - and most of the time IS research taps into these realms. Given constraints such as time, resources, and the ever-changing boundaries of this field, do you think it is (completely) inexcusable for a literature review to focus on the leading and more recognized journals in the field?

    3.I feel MISQ’s definitions of concept-centric and author-centric research may limit the research in the field. It is definitely important for research to deconstruct the underlying concepts and use them as a stepping-stone for further investigation. However, taking into account the “lack of theoretical progress” (pg. xiii), evaluating and summarizing relevant articles is also crucial. Don’t you think the concept-centric should encompass author-centric? Wouldn’t a combined approach be more beneficial?

  4. 1) In the Prospective Authors and Topics section, Jane and Richard state that prospective authors should seek senior editors for "advice whether another author is currently working on the topic and to [receive] broad guidance on the direction of the work". This suggestion gives me the impression that the authors favor sequential releases when doing literature reviews. However, wouldn't independent overlapping literature reviews allow for stronger statements to be made when multiple authors reach similar conclusions or help outline gaps and differences in the research when different authors propose different models or justifications?

    2) Throughout the paper, Jane and Richard state that the proposed theory should be interesting, delightful, and creative. However, wouldn't this be impossible to achieve since the entire paper was constructed in such a way that the proposed theory seems feasible and acceptable and thus either the entire paper will be interesting, delightful, and creative or not? Moreover, when the proposed theory is justified with past empirical studies, wouldn't it be harder for the theory to be any of the latter?

    3) When writing a literature review, Jane and Richard advise the use of a Concept Matrix with units of analysis. However, when the matrix shows a clear division among the concepts and the author can not develop a theory for such phenomenon, what would be the best course of action for the author? Would it be better for the author to focus in one set and encourage other scholars to research the other (depth) or would it be better to write a literature review exposing both sets and the difference without showing a preference for one or the other (breath)?

  5. 1. In writing a literature review, how do you bring in and consider the authors' biases when discussing the works together? If comparing data, how do you account for the differences in experimental conditions?

    2. In the article, Webster and Watson stated that they frequently received articles focusing on solely on the “top” journals (p. xvi) and proposed a system for finding influential articles. Do you agree with the steps they outlined? How else could you find influential journals? Should you use citation rates and impact factors in choosing the journals from which to base your literature review?

    3. How should you account for flaws in previous studies when writing a literature review? Webster and Watson suggest that sometimes these works should be downplayed or not included (p. xviii). Do you agree? Would it sometimes be helpful to discuss flawed studies so that the flaws are not repeated?

  6. This comment has been removed by the author.

  7. 1. The authors suggest that a person who is putting together a review should draw material from a range of subjects since IS is a highly interdisciplinary field. They later list a few specific areas such as computer science and organization theory, but they all seem closely related- maybe with the exception of biology. I am wondering how broad this range of disciplines can be, especially since I (and most of the others in our class, I’m sure) are interested in focusing our work at the iSchool in a specific direction, in my case towards Archives and Preservation.

    2. In the section “Identifying Relevant Literature” it is suggested that one should examine “selected conference proceedings” as a strategy to delve beyond leading journals. Do you have any advice for finding relevant conference proceedings, or is there a search engine that is particularly good at helping to conduct research in this area?


3. “The Web of Science” is listed as a good resource “to identify the key articles identified in the previous steps”(p. 4) of the initial article-compiling process. Would this also potentially be a good resource for determining how original one’s own work is shaping up to be, since it lists other recent analyses of the sources which have been chosen to examine?

  8. In the introductory paragraph, we are told that an effective review article “facilitates theory development, closes areas where a plethora of research exists, and uncovers areas where research is needed”. What does closing areas where a plethora of research exists mean? Doesn’t this hinder the process of advanced research in the field, if that area is downplayed and labelled as ‘a field that is closed’? Were there any previous reviews which upheld the ‘closing’ of a field?

    In theoretical development of article, it is mentioned “Extending current theories or developing new theories will create directions for future research” – I feel that until the extensions to theories are backed by justifications, i.e. by practical or empirical results or from experience, they should not be considered as potential directions for future research. Moreover, for research in emerging fields it might be better to verify the current theory before extending it hypothetically. This is because there were cases where theories that were previously thought to be correct were refuted at a later stage.

    Though the MISQ’s suggestion of using the concept-centric approach than the author-centric approach may provide for a pleasant reading, it might limit the progress in IS. There could be situations where the author needs to compare several logically-structured works (as mentioned about Cuban missiles in ‘Theoretical Development of your Article’ section). For example, if two different algorithms that achieve a same task need to be compared, then it is more sensible to compare them sequentially (Author-centric) than to compare them in parallel (Concept-centric). The algorithms might be huge and involve different techniques and data structures - each for solving different parts of the problem. We still prefer studying them wholly so as to identify the merits and demerits than comparing their elements without thorough understanding of their usage in the complete algorithm.

  9. 1. As for identifying the relevant literature for the topic, the paper gives excellent suggestions for steps of finding relevant literature for given topics, however, due to the limitation of space, few detail is discussed on which to choose in those relevant literatures. For specific topics there’ll be a lot of relevant literature focus on areas from theories, methodologies and applications. How can we determine which is most relevant or suitable for our topic.

    2. As it’s said in the paper, it’s an important but difficult task to extend current or develop new theories in the review, and the author suggests two approaches, variance theories and process theories. What are the advantages and disadvantages for these theories and are there topics specifically suitable for one of them?

    3. The paper provides a detailed process of how to develop a review paper. One thing that is not emphasized and might be important is the timeline of the development of theories in given field. Whether it’ll make a better review paper if we arrange the topic in the way that it talks about how the theories originate, and what’re the flaws and how following up theories modify and fix them?

  10. 1. In most research papers I have read, the author provides two important sections: related works and future works. For the related works section, the author compares and contrasts the contributions of his work with that of others working on a similar topic. Later in the paper, the author lists branches of research that could stem from his work for future explorations. With these in mind, what makes a literature review more impactful? A regular research paper is able to compare and contrast the past and lay a foundation for the future. Is the main benefit of writing a separate literature review that a scholar can analyze research and help guide the field without having to, in conjunction, produce research worthy of a publication in his field?

    2. The authors mention that the most important part of the review is proposing new theories in the field, but it is consequently the most challenging section. The paper goes on to list a couple different methods of tackling this problem. Is it possible to have a literature review that summarizes the past and pinpoints where the gaps or discrepancies are in the current research standing? Technically, the literature review is not explicitly stating new theories but I can see where this type of summary viewpoint could be very useful. In addition, being aware of the gaps is outlining the subtopic areas where new theories can be developed.

    3. Once the author of a literature review has outlined new theories, he is supposed to provide justification for his ideas. What if an author is proposing a truly novel idea? For instance, sometimes in research, an author will draw inspiration from a seemingly unrelated philosophy in a different field and attempt to apply it to his discipline. Without doing preliminary research of his own to justify the idea, is a literature review even the correct forum to express such an idea? The authors place a lot of emphasis on the justification section, even going so far as to say “the why or logical reasoning is the most important component of the explanation.” Therefore it would seem that more than just an authors reasoning is needed for a justification of new theories.

  11. 1. If systematic approach of search is to be employed to identify the related articles and journals with respect to a research field then the search becomes exhaustive. What strategies would help in pointing out that the most important ones have been referenced in our paper without missing critical ones? The paper states that colleagues’ suggestions would be helpful in identifying this but would it not be more user-centric in that case? What is important to one may not be so for others.So how to device a strategy that would both be topic relevant and critical for review?

    2.The article states that one needs to appreciate the past work rather than criticizing the work of the future scholars. The tone of the review should not be one-sided detailing the faults of the existing work but at the same time would it not be ideal to use an aggressive tone to state the failure models with their limitations and go one step ahead and express how the new approach proposed can make a difference to it?

    3.Extending or developing theories is expressed as a weakest part of a review but should it not be the strongest part of any review? This is contradicting to the author’s initial view which stated that a literature should kindle its readers towards building strong research interests in developing new theories and enrich the field with more papers. So in my opinion reviews' not only interesting but also strong part is the new idea incorporated in the research.

  12. 1. My experience with reading academic papers is in computer science, and I
    spend most of my time just understanding the topic that is presented. It seems
    difficult and possibly intractable to develop a good command of all of the
    articles in a research area in just one semester. Honestly it seems like the
    kind of thing that one develops over years of research in a field. So a field
    like information retrieval seems like an overly broad research area to take on
    in a literature review. How would we scale this down to one semester's work?

    2. The paper mentions that the tone of literature reviews should focus on
    identifying patterns that emerge, and an "overly negative approach to the
    previous literature" indicates "amateurism". While I've found this to be true
    in the areas where I am familiar, how does one go about providing constructive
    negative feedback?

    3. This paper suggests that one of the primary functions of literature reviews
    is exposing gaps in research. Being relatively unfamiliar with this field, how
    would one differentiate between a research gap and simply a lack of familiarity
    with the existing research?

  13. In the part of identifying the relevant literature, the author considers that a high-quality review is prone to focus on concepts. Here, what do the concepts mean? Are they just confined to the real concepts, or do they also involve some other aspects of researches on information technology, like research methods?

    About the tense used in academic writing, I have a question here, which tense is prone to be used when we describe and explain our researches in papers?

    In this paper, the author says that it’s an important way to evaluate our theories by asking for the helps from our colleagues to read and comment on our work. So, is there any other ways to evaluate theories; for instance, need we conduct some experiments or researches to test theories?

  14. Under the “Prospective Authors and Topics” section, Webster & Watson present the idea that two types of literature reviews exist – one focused on consolidating past research and the other dealing with current research. Given the nature of IS would not a third type of literature review acting as a fusion of the two be beneficial for the field as it continues to grow?

    Webster & Watson make it a point to present the idea under the “Tone” section that being critical of a work should not take place and instead a writer of a literature review should merely state how the work contributed to future work or how it applies to the topic of the review. How does one go about presenting a work in such a way as to show the contribution it has brought to the IS field while pinpointing potential shortfalls in relation to current or future research and ideas?

    Because IS encompasses many different fields of study, how does the list of reviewer concerns apply to each of these disciplines. What might seem like an impact or contribution to one area of study might not be the case to another area and vice versa. How then might we approach writing a review if it does not happen to intertwine nicely with one of the subjects closely associated with IS such as computer science?

  15. As a visual person, I love the idea of the grid/table that is used, is this something that is ever included in a paper? Perhaps in the appendix? I don't recall ever seeing one, but I think this would be great to help readers to reference documents and see if they agree with the writers assessment/categorization of a given paper.

    The author also points out that "highlighting the discrepancy between what we know and what we need to know alerts other scholars to opportunities for key contributions". Often, in most journal articles the authors will state what further research needs to be done, or what opportunities for further research exist. Would it be considered inappropriate/stealing of another scholars ideas to take those holes that others have identified and do research on them?

    What/(Is there) a standard protocol for following up on those identified missing pieces of research?

  16. The authors recommend a structured approach to determine the source material for the review. While it becomes a common practice for researchers to look into leading journals and top conference proceedings to get a fair understanding of the state of the art in the research domain, I reckon it is a good suggestion to also review the citations for the articles identified in the previous step and even look into 'future' articles citing those articles. But there is one question remaining that how we can efficiently locate most related source materials for our research topics from this approach? A pertinent example is a few months ago we gathered about three hundred papers from a similar approach for a research topic but in the end we randomly picked up half of them as we simply did not have time to go through each paper(not even skimming through by reading only abstract, introduction and conclusion). The potential risk is that we might miss a very important related work which overshadows the entire research.

    The authors present a concept matrix where we can assign groups to each article. I find it is a good practice and very useful in writing section of 'related works'. But the key step of how to develop a logical group can not be ignored as over-categorization is as risky as under-categorization. I am wondering whether there are any guidelines for doing this.

    The authors identify a main source for propositions reasoning as practice or experience. While I believe it shall help to support a proposition, the approach has the risk of being biased as it is very subjective and in the current literature, I am wondering what the percentage of acceptance is in the top conferences if papers mainly use practice or experience as main supporting evidence.

  17. 1. The concept-centric approach seems intuitive with the concept matrix, I wonder if a multi-dimensional analysis exploring concept overlap or potential overlap is an interesting extension to guide future-work/progress.

    2. The paper does a good job at identifying various stages of writing a literature review, what is not clear is an approximate time share between stages and where should focus of the review lie, or if changing focus results in literature reviews of comparable quality.

    3. Is it fair to conclude that the quality of a literature review largely depends on the quality and diversity of the review panel.

  18. 1. The authors write about extending a current theory or developing a new theory as being part of the review part. However, shouldn't this be the domain of a 'theory' paper? I would have thought a review paper was simply meant to synthesize the material in a consistent and readable manner, so that others who were interested in providing a theoretical underpinning could look to the review for a starting point.

    2. The units of analysis provided as an example in the paper seem more closely pertinent to the social sciences (group, organization and individual). Are there analogous units in computer science (e.g. would 'parallel' and 'sequential' qualify?)

    3. Some of the concepts mentioned in the section on theoretical development could do with an illustrative example to clearly indicate their meanings. For example, can we concretize a variance model and compare it against a process model?

  19. 1. A review is “nearing completion when you are not finding new concepts in your article set”. (p. xvi) How can you guarantee in practice that there is no new concept?

    2. The authors suggested using the Web of Science (the electronic version of the Social Science Citation Index) to identify articles. (p.xvi) What sources should researchers use if they are not in social science?

    3. “A literature review is concept-centric.” (p. xvi) Is there a conflict between "concept-centric" and "complete"? I mean, complete refers to extend and cover all related concepts, but concept-centric, in practice, may result in some dropping work which may impact the "complete", how to trade-off?

  20. 1. One aspect that is not mentioned in the article is writing to the specific journal format. How much of which paper sections should suffer for short conference publications as opposed to full journal articles?

    2. The Webster article mentions that a diverse article set should make up the literature review. In the ideal situation the literature review would be made up of just the necessary sources, but notions of credibility and paper requirements sometimes put pressure to add more tangentially related sources. What is the line between a citation that is necessary and a citation that is superfluous?

    3. One strategy the paper recommends is for researchers to create a ‘Concept Matrix’ of reviewed articles. What are some other strategies that people have used while doing a literature review that they think are helpful?

  21. The authors point out the importance of going beyond just the most well-known journals in the field when researching, but also advise researchers to use the citations and references in the articles to find to broaden their search. If the top journals have the "best" articles in the field and those articles refer to other works in the field, wouldn't that by nature make the researcher broaden out to those other articles in the process of research?

    The authors stress the need to respect previous research in your review. If a specific article has flaws and gaps in their processes and you feel it necessary to include that as part of your review, how would you be respectful while emphasizing their mistakes?

    Webster and Watson state that developing new theories is the most important part of a literature review. They also point out that a review create directions for future research. If a review is well written and points out gaps in the current knowledge of a topic, but fails to propose a serious theory as to why those gaps exist, hasn't it still given direction for future research and in that sense been helpful to the field?

  22. 1.The authors state that a high-quality review is complete. A complete review covers relevant literature on the topic (p. xv). The question is to what extent can we say a review covers relevant literature on a certain topic?
    2. Because IS is an interdisciplinary field, we have to look outside the IS field when reviewing and developing theory.(p. xvi) Is this a too wide scope for IS reseachers? How to do that?
    3. The authors of this article suggest that researchers ask colleagues to read and comment on the literature review before submitting it. Is it better if the evaluation starts before the literature review is written?

  23. 1. In the section on 'Structuring Review' the author recommends the creation of a Concept Matrix encouraging us to synthesize the literature specific to each identified concept. Since, this matrix (even post compilation) is restricted to commutative non-weighted relationships - I wonder if we could work towards a multi-dimensional weighted analysis which will additionally support representation of the transitive relationships that we've already ascertained through the backward and forward searches. Further, for me as a Machine Learning enthusiast it's the array of applications that see implementation through a core concept and similarly the several concepts which are embedded together in a single application that is stimulating and I believe it is vital to map both this disparity and similarity with equal fervour. What could be a potential comprehensive solution to this issue?

    2. Although the Variance factors and the Process Theories form the basis from which most Conceptual Models are derived - I'm curious about Framework Models which look at systems holistically as opposed to in entities or parts which is a characteristic that both Variance and Process share. For instance, if we look at the Biological process of Homeostasis - the purpose of this system is to work towards constancy as opposed to variation and is triggered on the basis of a positive/ a negative feedback while always considering the human body as a whole. This Model wouldn't make it as a fit in the hybrid model category or into Variance or Process individually and so is there another model we need to consider?

    3. While elucidating on the identification of critical knowledge gaps, the author encourages the reviewer to weigh 'What we know' with ' What we need to know'. Doesn't this overemphasize the position of the reviewer enormously? The solution suggested in the paper involves 'developing a roadmap with supporting propositions'. Isn't incorporating our information so far on propositions likely to lead us to an infinite regress as the entire roadmap is on the basis of conjecture of what the reviewer 'feels' and so lacks an empirical justification?

  24. 1. A successful literature review constructively informs the reader about what has been learned. Writing a review not only requires an examination of past research, but means making a chart for future research. The above two are both point of view of the author, however, they may be conflicting themselves?

    2. Is it necessary to focus not only on North American or a small set of "top" publication but also some relative minority? The author thought the sours journals are too small and limited. However, if a topic is really interesting ,promising and important, it will surly appear on some "top" publication someday. A review is to present the most important content of some topic or fields, so "top" publication may just be enough.

    3. The author stresses that the author of the review should create their own discussion and conclusions in their review, which may be the most important part of the article. However, the review's accelerate the accumulations of field knowledges, how could him guarantee the reviewer's own point of view may not mislead or confuse the readers?

  25. In this article the subject of the interdisciplinary nature of the Information Science field is discussed as something that lends to the small amount of theoretical articles in the field. This seems counter-intuitive however as it seems that this would allow authors from several fields contribute to the literature of the IS field thus expanding it. Do the authors of this field merely have a narrow view of what constitutes IS literature or are they right in their assumption that the multi-disciplinary nature of the field hinder theoretical articles being published?

    In this article the authors state that the majority of articles in literature reviews come from North America. Is this just due to bias or is most IS research being done in North America? Do cultural differences across nations affect IS research in a way that strategies and ideas developed overseas would not be applicable in North America?

    In this article the authors seem to make a big point of stating that it is important to not be too critical of previous research as the point of the review is not to critique research that has been done but rather to build on that research. However, they do not give very good examples of how critical is too critical. If there are flawed conclusions in an article that could lead future research in the wrong direction then is it not important to be critical of that paper as to inform the authors of that paper and others?

  26. This comment has been removed by the author.

  27. 1) Why is it critical that literature reviews be conducted in the field of IR now and what can people newly entering the filed contribute in particular?

    2) What special challenges arise when conducting literature reviews in this field?

    3) According to the article a ‘good’ theory was needed to be memorable, parsimonious, falsifiable, useful, relevant and to explain, predict, delight, among lots of other things. Since in our case a ‘peer review’ would come from within the class can we develop a class definition of what we believe a ‘good‘ theory will be and what we find valuable in literature reviews in general?

  28. 1) The author suggests that one of the primary components of a successful literature review is proposing a theory, including a future avenue of research. Given this sentiment, what can a potential author, who is a novice in the field, do to add utility to their review, since they will probably fall short in terms of the quality of the theory?

    2) The author discusses the issue of bias in literature reviews which focus solely on a small set of “top” publications. While I understand the concern of disregarding journals that might offer potentially new concepts, increasing the scope to include “mediocre” journals might leave the review open to the reviewer’s internal biases regarding what are “important” concepts. The reviewer might present worked from a diversified set of journals, but glance over concepts popular in top journals, under the reasoning that just because they are in top journals they are not as important concepts. Does being biased towards top journals actually inject a different flavor of objectivity into a review, since “top” is agreed upon by the community?

    3) The article emphasizes “developing a conceptual model with supporting propositions.” This requirement seems to imply that the writer of the review would most likely need to have significant expertise/experience in the field. Would it be useful to have a literature review from someone outside the field looking in? Since the best research often threads the line between two disciplines, such a reviewer might be able to offer unique insight / perspective.

  29. I have the following comments/queries on this reading material:

    1) Why does the article not encompass comparison/performance evaluation of models of information systems within the structure of literature review?

    2) Webston et al. urge perspective review paper authors to propose conceptual models but how can radically different solutions be generalized into a single conceptual model? Won't this enforce a constraint on the pool of papers surveyed? And does this fact not emphasize the need for comparison and evaluation between different models?

    3)The article does not consider the categories of target audience who might look forward to review papers. They could range from a naive researcher to an interdisciplinary expert. Isn't this a major factor that should be taken into account while writing a review paper? Why the authors don't discuss it?

  30. 1. The author has talked about the creating a structured table to help the organization of thoughts. I would agree that it might help in some cases. But in most of the situations it is the inter-dependencies between various lines of thought, which need to be argued and discussed. How can the review be structured in such a situation?

    2. Evaluation from various sources of a literature review from numerous sources might result in a huge no of inputs, which might and might benefit a given review. What criteria need to be defined to decide weather a given advice actually benefits the said paper or makes it weaker?

    3. The author has suggested that the limitations and the assumptions of the topic being discussed need to be organized in the beginning section of the paper. But wont placing the assumptions as and when they are needed while trying to make a point be more beneficial for a person trying to comprehend the concept as a whole?