research paper sample short

A. Title of the Research Paper
B. Author's Name
C. Institutional Affiliation
D. Course Information
E. Date

A. Purpose of the Study
B. Research Questions/Objectives
C. Methodology
D. Key Findings
E. Conclusion

A. Background Information
B. Statement of the Problem
C. Significance of the Study
D. Research Objectives/Hypothesis
E. Structure of the Paper

A. Overview of Relevant Literature
B. Key Theories or Concepts
C. Discussion of Previous Studies
D. Gaps in the Existing Literature
E. Theoretical Framework

A. Research Design
B. Participants or Sample
C. Data Collection Methods
D. Data Analysis Techniques
E. Limitations

A. Presentation of Findings
B. Data Analysis
C. Tables, Graphs, or Figures (if applicable)
D. Interpretation of Results

A. Summary of Findings
B. Comparison with Literature
C. Implications of the Results
D. Limitations and Future Research
E. Conclusion

A. Summary of the Study
B. Contribution to the Field
C. Recommendations
D. Concluding Remarks

A. Citations in APA/MLA/Chicago style
B. Books, Articles, Journals, and Other Sources Cited

(Paper concludes with a bibliography)

In a similar vein, resist the urge to call the Great Depression the “saddest chapter in American history,” or T.S. Eliot “the most famous modern poet.”

If your paper does not actually examine all chapters in American history, or all famous modern poets, such a vague claim adds nothing to your argument.

15 thoughts on “ Short Research Papers: How to Write Academic Essays ”

Hi, I was searching for some information on how to write quality academic paper when I came across your awesome article on Short Research Papers: How to Writer Academic Essays ( https://jerz.setonhill.edu/writing/academic1/short-research-papers/ ) Great stuff!!! I especially like the way you recommend sticking to the 4 basics of writing academic essays. Very few students have mastered how to avoid distractions and focus on a single topic. Many students think that the broad, sweeping statements could give them better grades but they are wrong.

However, I came across a few links that didn’t seem to be working for you. Want me to forward you the short list I jotted down? Cheers Elias

I see some broken links in the comments, but otherwise I’m not sure what you mean.

I found the part about not using my personal opinion or generalities to be very helpful. I am currently writing a 2 page paper and was having a hard time keeping it short. Now I know why. Thanks. Stick to the facts.

This seem to be old but very relevant. Most of what you have stated are things my professor has stated during class trying to prepare us to write a short thesis reading this information verses hearing it was very helpful. You have done an awesome job! I just hope I can take this and apply it to my papers!

Great Post! Thank u!

Thank you for all your effort and help. You´ve taught me a number of things, especially on what college professors´ look for in assigning students short research papers. I am bookmarking your page, and using it as a reference.

Thank you kindly. YOU´VE HELPED A LOST STUDENT FIND HER WAY!

I appreaciate all the help your web site has given to me. I have referred to it many times. I think there may be a typo under the headline of AVOID GLITTERING GENERALITIES: “Throughout the ages, mankind has found many uses for salt. Ancient tribes used it preserve meat;” This is in no way a slight – I thought you might want to know. Please forgive me if I am incorrect. Thank you again – you rock!

You are right — I’ll fix it the next time I’m at my desktop. Thank you!

i would like to say thank you for your detailed information even though it takes time to read as well as we’ve got learnings out from it . even though it’s holiday next week our teacher assigned us to make a short research paper in accordance of our selected topic ! I’m hoping that we can make it cause if we can’t make it, right away, for sure we will get a grade’s that can drop our jaws ! :) ♥ tnx ! keep it up ! ♪♪

Sorry I have not done this for years

Hello I am the mother of a high school student that needs help doing a paper proposal for her senior project. Her topic is Photography. To be honest I have done this for years and I am trying to help, but i am completely lost. What can you recommend since she told me a little late and the paper is due tomorrow 11/11/11.

This page is designed for college students, but I am sure your daughter’s teacher has assigned readings that will guide your daughter through her homework.

Any paper that your daughter writes herself, even if it is late, will be a valuable learning experience — showing her the value of managing her time better for the next time, and preparing her for the day when she will have to tackle grown-up problems on her own.

I am having a hard time with my government essay. I am 55 taking a college course for the first time, and I barely passed high school. Last year I took this course wrote the essay, and did many things wrong. It was all in the typing. I had good story line, excellent site words, and good points of arguments. It wasn’t right on paper. My format is off. Where can I find and print a format. also I need to learn site words.

Most teachers will provide a model to follow. If it’s not already part of the assignment instructions, you could ask your prof. Better yet, bring a near-complete draft to your prof’s office hours, a few days before the due date, and ask for feedback. Your school probably has a writing center or tutoring center, too.

I would like to thank you for such detailed information. I am not a native speaker and I am doing a research paper;so, as you may think, it is really a hard job for me. A friend of mine who saw my draft of Lit. Rev asked me what type of citation format i was using, MLA or APA and I was puzzeled; then I decided to check the net and came across to this! It is being such a help Elsa

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Privacy Policy

Home » Research Paper Format – Types, Examples and Templates

Research Paper Format – Types, Examples and Templates

Table of Contents

Research paper format is an essential aspect of academic writing that plays a crucial role in the communication of research findings . The format of a research paper depends on various factors such as the discipline, style guide, and purpose of the research. It includes guidelines for the structure, citation style, referencing , and other elements of the paper that contribute to its overall presentation and coherence. Adhering to the appropriate research paper format is vital for ensuring that the research is accurately and effectively communicated to the intended audience. In this era of information, it is essential to understand the different research paper formats and their guidelines to communicate research effectively, accurately, and with the required level of detail. This post aims to provide an overview of some of the common research paper formats used in academic writing.

Research Paper Formats

Research Paper Formats are as follows:

• APA (American Psychological Association) format
• MLA (Modern Language Association) format
• Chicago/Turabian style
• IEEE (Institute of Electrical and Electronics Engineers) format
• AMA (American Medical Association) style
• Harvard style
• Vancouver style
• ACS (American Chemical Society) style
• ASA (American Sociological Association) style
• APSA (American Political Science Association) style

APA (American Psychological Association) Format

Here is a general APA format for a research paper:

• Title Page: The title page should include the title of your paper, your name, and your institutional affiliation. It should also include a running head, which is a shortened version of the title, and a page number in the upper right-hand corner.
• Abstract : The abstract is a brief summary of your paper, typically 150-250 words. It should include the purpose of your research, the main findings, and any implications or conclusions that can be drawn.
• Introduction: The introduction should provide background information on your topic, state the purpose of your research, and present your research question or hypothesis. It should also include a brief literature review that discusses previous research on your topic.
• Methods: The methods section should describe the procedures you used to collect and analyze your data. It should include information on the participants, the materials and instruments used, and the statistical analyses performed.
• Results: The results section should present the findings of your research in a clear and concise manner. Use tables and figures to help illustrate your results.
• Discussion : The discussion section should interpret your results and relate them back to your research question or hypothesis. It should also discuss the implications of your findings and any limitations of your study.
• References : The references section should include a list of all sources cited in your paper. Follow APA formatting guidelines for your citations and references.

Some additional tips for formatting your APA research paper:

• Use 12-point Times New Roman font throughout the paper.
• Double-space all text, including the references.
• Use 1-inch margins on all sides of the page.
• Indent the first line of each paragraph by 0.5 inches.
• Use a hanging indent for the references (the first line should be flush with the left margin, and all subsequent lines should be indented).
• Number all pages, including the title page and references page, in the upper right-hand corner.

APA Research Paper Format Template

APA Research Paper Format Template is as follows:

Title Page:

• Title of the paper
• Author’s name
• Institutional affiliation
• A brief summary of the main points of the paper, including the research question, methods, findings, and conclusions. The abstract should be no more than 250 words.

Introduction:

• Background information on the topic of the research paper
• Research question or hypothesis
• Significance of the study
• Overview of the research methods and design
• Brief summary of the main findings
• Participants: description of the sample population, including the number of participants and their characteristics (age, gender, ethnicity, etc.)
• Materials: description of any materials used in the study (e.g., survey questions, experimental apparatus)
• Procedure: detailed description of the steps taken to conduct the study
• Presentation of the findings of the study, including statistical analyses if applicable
• Tables and figures may be included to illustrate the results

Discussion:

• Interpretation of the results in light of the research question and hypothesis
• Implications of the study for the field
• Limitations of the study
• Suggestions for future research

References:

• A list of all sources cited in the paper, in APA format

Formatting guidelines:

• Double-spaced
• 12-point font (Times New Roman or Arial)
• 1-inch margins on all sides
• Page numbers in the top right corner
• Headings and subheadings should be used to organize the paper
• The first line of each paragraph should be indented
• Quotations of 40 or more words should be set off in a block quote with no quotation marks
• In-text citations should include the author’s last name and year of publication (e.g., Smith, 2019)

APA Research Paper Format Example

APA Research Paper Format Example is as follows:

The Effects of Social Media on Mental Health

University of XYZ

This study examines the relationship between social media use and mental health among college students. Data was collected through a survey of 500 students at the University of XYZ. Results suggest that social media use is significantly related to symptoms of depression and anxiety, and that the negative effects of social media are greater among frequent users.

Social media has become an increasingly important aspect of modern life, especially among young adults. While social media can have many positive effects, such as connecting people across distances and sharing information, there is growing concern about its impact on mental health. This study aims to examine the relationship between social media use and mental health among college students.

Participants: Participants were 500 college students at the University of XYZ, recruited through online advertisements and flyers posted on campus. Participants ranged in age from 18 to 25, with a mean age of 20.5 years. The sample was 60% female, 40% male, and 5% identified as non-binary or gender non-conforming.

Data was collected through an online survey administered through Qualtrics. The survey consisted of several measures, including the Patient Health Questionnaire-9 (PHQ-9) for depression symptoms, the Generalized Anxiety Disorder-7 (GAD-7) for anxiety symptoms, and questions about social media use.

Procedure :

Participants were asked to complete the online survey at their convenience. The survey took approximately 20-30 minutes to complete. Data was analyzed using descriptive statistics, correlations, and multiple regression analysis.

Results indicated that social media use was significantly related to symptoms of depression (r = .32, p < .001) and anxiety (r = .29, p < .001). Regression analysis indicated that frequency of social media use was a significant predictor of both depression symptoms (β = .24, p < .001) and anxiety symptoms (β = .20, p < .001), even when controlling for age, gender, and other relevant factors.

The results of this study suggest that social media use is associated with symptoms of depression and anxiety among college students. The negative effects of social media are greater among frequent users. These findings have important implications for mental health professionals and educators, who should consider addressing the potential negative effects of social media use in their work with young adults.

References :

References should be listed in alphabetical order according to the author’s last name. For example:

• Chou, H. T. G., & Edge, N. (2012). “They are happier and having better lives than I am”: The impact of using Facebook on perceptions of others’ lives. Cyberpsychology, Behavior, and Social Networking, 15(2), 117-121.
• Twenge, J. M., Joiner, T. E., Rogers, M. L., & Martin, G. N. (2018). Increases in depressive symptoms, suicide-related outcomes, and suicide rates among U.S. adolescents after 2010 and links to increased new media screen time. Clinical Psychological Science, 6(1), 3-17.

Note: This is just a sample Example do not use this in your assignment.

MLA (Modern Language Association) Format

MLA (Modern Language Association) Format is as follows:

• Page Layout : Use 8.5 x 11-inch white paper, with 1-inch margins on all sides. The font should be 12-point Times New Roman or a similar serif font.
• Heading and Title : The first page of your research paper should include a heading and a title. The heading should include your name, your instructor’s name, the course title, and the date. The title should be centered and in title case (capitalizing the first letter of each important word).
• In-Text Citations : Use parenthetical citations to indicate the source of your information. The citation should include the author’s last name and the page number(s) of the source. For example: (Smith 23).
• Works Cited Page : At the end of your paper, include a Works Cited page that lists all the sources you used in your research. Each entry should include the author’s name, the title of the work, the publication information, and the medium of publication.
• Formatting Quotations : Use double quotation marks for short quotations and block quotations for longer quotations. Indent the entire quotation five spaces from the left margin.
• Formatting the Body : Use a clear and readable font and double-space your text throughout. The first line of each paragraph should be indented one-half inch from the left margin.

MLA Research Paper Template

MLA Research Paper Format Template is as follows:

• Use 8.5 x 11 inch white paper.
• Use a 12-point font, such as Times New Roman.
• Use double-spacing throughout the entire paper, including the title page and works cited page.
• Set the margins to 1 inch on all sides.
• Use page numbers in the upper right corner, beginning with the first page of text.
• Include a centered title for the research paper, using title case (capitalizing the first letter of each important word).
• Include your name, instructor’s name, course name, and date in the upper left corner, double-spaced.

In-Text Citations

• When quoting or paraphrasing information from sources, include an in-text citation within the text of your paper.
• Use the author’s last name and the page number in parentheses at the end of the sentence, before the punctuation mark.
• If the author’s name is mentioned in the sentence, only include the page number in parentheses.

Works Cited Page

• List all sources cited in alphabetical order by the author’s last name.
• Each entry should include the author’s name, title of the work, publication information, and medium of publication.
• Use italics for book and journal titles, and quotation marks for article and chapter titles.
• For online sources, include the date of access and the URL.

Here is an example of how the first page of a research paper in MLA format should look:

Headings and Subheadings

• Use headings and subheadings to organize your paper and make it easier to read.
• Use numerals to number your headings and subheadings (e.g. 1, 2, 3), and capitalize the first letter of each word.
• The main heading should be centered and in boldface type, while subheadings should be left-aligned and in italics.
• Use only one space after each period or punctuation mark.
• Use quotation marks to indicate direct quotes from a source.
• If the quote is more than four lines, format it as a block quote, indented one inch from the left margin and without quotation marks.
• Use ellipses (…) to indicate omitted words from a quote, and brackets ([…]) to indicate added words.

Works Cited Examples

• Book: Last Name, First Name. Title of Book. Publisher, Publication Year.
• Journal Article: Last Name, First Name. “Title of Article.” Title of Journal, volume number, issue number, publication date, page numbers.
• Website: Last Name, First Name. “Title of Webpage.” Title of Website, publication date, URL. Accessed date.

Here is an example of how a works cited entry for a book should look:

Smith, John. The Art of Writing Research Papers. Penguin, 2021.

MLA Research Paper Example

MLA Research Paper Format Example is as follows:

Your Professor’s Name

Course Name and Number

Date (in Day Month Year format)

Word Count (not including title page or Works Cited)

Title: The Impact of Video Games on Aggression Levels

Video games have become a popular form of entertainment among people of all ages. However, the impact of video games on aggression levels has been a subject of debate among scholars and researchers. While some argue that video games promote aggression and violent behavior, others argue that there is no clear link between video games and aggression levels. This research paper aims to explore the impact of video games on aggression levels among young adults.

Background:

The debate on the impact of video games on aggression levels has been ongoing for several years. According to the American Psychological Association, exposure to violent media, including video games, can increase aggression levels in children and adolescents. However, some researchers argue that there is no clear evidence to support this claim. Several studies have been conducted to examine the impact of video games on aggression levels, but the results have been mixed.

Methodology:

This research paper used a quantitative research approach to examine the impact of video games on aggression levels among young adults. A sample of 100 young adults between the ages of 18 and 25 was selected for the study. The participants were asked to complete a questionnaire that measured their aggression levels and their video game habits.

The results of the study showed that there was a significant correlation between video game habits and aggression levels among young adults. The participants who reported playing violent video games for more than 5 hours per week had higher aggression levels than those who played less than 5 hours per week. The study also found that male participants were more likely to play violent video games and had higher aggression levels than female participants.

The findings of this study support the claim that video games can increase aggression levels among young adults. However, it is important to note that the study only examined the impact of video games on aggression levels and did not take into account other factors that may contribute to aggressive behavior. It is also important to note that not all video games promote violence and aggression, and some games may have a positive impact on cognitive and social skills.

Conclusion :

In conclusion, this research paper provides evidence to support the claim that video games can increase aggression levels among young adults. However, it is important to conduct further research to examine the impact of video games on other aspects of behavior and to explore the potential benefits of video games. Parents and educators should be aware of the potential impact of video games on aggression levels and should encourage young adults to engage in a variety of activities that promote cognitive and social skills.

Works Cited:

• American Psychological Association. (2017). Violent Video Games: Myths, Facts, and Unanswered Questions. Retrieved from https://www.apa.org/news/press/releases/2017/08/violent-video-games
• Ferguson, C. J. (2015). Do Angry Birds make for angry children? A meta-analysis of video game influences on children’s and adolescents’ aggression, mental health, prosocial behavior, and academic performance. Perspectives on Psychological Science, 10(5), 646-666.
• Gentile, D. A., Swing, E. L., Lim, C. G., & Khoo, A. (2012). Video game playing, attention problems, and impulsiveness: Evidence of bidirectional causality. Psychology of Popular Media Culture, 1(1), 62-70.
• Greitemeyer, T. (2014). Effects of prosocial video games on prosocial behavior. Journal of Personality and Social Psychology, 106(4), 530-548.

Chicago/Turabian Style

Chicago/Turabian Formate is as follows:

• Margins : Use 1-inch margins on all sides of the paper.
• Font : Use a readable font such as Times New Roman or Arial, and use a 12-point font size.
• Page numbering : Number all pages in the upper right-hand corner, beginning with the first page of text. Use Arabic numerals.
• Title page: Include a title page with the title of the paper, your name, course title and number, instructor’s name, and the date. The title should be centered on the page and in title case (capitalize the first letter of each word).
• Headings: Use headings to organize your paper. The first level of headings should be centered and in boldface or italics. The second level of headings should be left-aligned and in boldface or italics. Use as many levels of headings as necessary to organize your paper.
• In-text citations : Use footnotes or endnotes to cite sources within the text of your paper. The first citation for each source should be a full citation, and subsequent citations can be shortened. Use superscript numbers to indicate footnotes or endnotes.
• Bibliography : Include a bibliography at the end of your paper, listing all sources cited in your paper. The bibliography should be in alphabetical order by the author’s last name, and each entry should include the author’s name, title of the work, publication information, and date of publication.
• Formatting of quotations: Use block quotations for quotations that are longer than four lines. Indent the entire quotation one inch from the left margin, and do not use quotation marks. Single-space the quotation, and double-space between paragraphs.
• Tables and figures: Use tables and figures to present data and illustrations. Number each table and figure sequentially, and provide a brief title for each. Place tables and figures as close as possible to the text that refers to them.
• Spelling and grammar : Use correct spelling and grammar throughout your paper. Proofread carefully for errors.

Chicago/Turabian Research Paper Template

Chicago/Turabian Research Paper Template is as folows:

Title of Paper

Name of Student

Professor’s Name

I. Introduction

A. Background Information

B. Research Question

C. Thesis Statement

II. Literature Review

A. Overview of Existing Literature

B. Analysis of Key Literature

C. Identification of Gaps in Literature

III. Methodology

A. Research Design

B. Data Collection

C. Data Analysis

IV. Results

A. Presentation of Findings

B. Analysis of Findings

C. Discussion of Implications

V. Conclusion

A. Summary of Findings

B. Implications for Future Research

C. Conclusion

VI. References

A. Bibliography

B. In-Text Citations

VII. Appendices (if necessary)

A. Data Tables

C. Additional Supporting Materials

Chicago/Turabian Research Paper Example

Title: The Impact of Social Media on Political Engagement

Name: John Smith

Class: POLS 101

Professor: Dr. Jane Doe

Date: April 8, 2023

I. Introduction:

Social media has become an integral part of our daily lives. People use social media platforms like Facebook, Twitter, and Instagram to connect with friends and family, share their opinions, and stay informed about current events. With the rise of social media, there has been a growing interest in understanding its impact on various aspects of society, including political engagement. In this paper, I will examine the relationship between social media use and political engagement, specifically focusing on how social media influences political participation and political attitudes.

II. Literature Review:

There is a growing body of literature on the impact of social media on political engagement. Some scholars argue that social media has a positive effect on political participation by providing new channels for political communication and mobilization (Delli Carpini & Keeter, 1996; Putnam, 2000). Others, however, suggest that social media can have a negative impact on political engagement by creating filter bubbles that reinforce existing beliefs and discourage political dialogue (Pariser, 2011; Sunstein, 2001).

III. Methodology:

To examine the relationship between social media use and political engagement, I conducted a survey of 500 college students. The survey included questions about social media use, political participation, and political attitudes. The data was analyzed using descriptive statistics and regression analysis.

Iv. Results:

The results of the survey indicate that social media use is positively associated with political participation. Specifically, respondents who reported using social media to discuss politics were more likely to have participated in a political campaign, attended a political rally, or contacted a political representative. Additionally, social media use was found to be associated with more positive attitudes towards political engagement, such as increased trust in government and belief in the effectiveness of political action.

V. Conclusion:

The findings of this study suggest that social media has a positive impact on political engagement, by providing new opportunities for political communication and mobilization. However, there is also a need for caution, as social media can also create filter bubbles that reinforce existing beliefs and discourage political dialogue. Future research should continue to explore the complex relationship between social media and political engagement, and develop strategies to harness the potential benefits of social media while mitigating its potential negative effects.

Vii. References:

• Delli Carpini, M. X., & Keeter, S. (1996). What Americans know about politics and why it matters. Yale University Press.
• Pariser, E. (2011). The filter bubble: What the Internet is hiding from you. Penguin.
• Putnam, R. D. (2000). Bowling alone: The collapse and revival of American community. Simon & Schuster.
• Sunstein, C. R. (2001). Republic.com. Princeton University Press.

IEEE (Institute of Electrical and Electronics Engineers) Format

IEEE (Institute of Electrical and Electronics Engineers) Research Paper Format is as follows:

• Title : A concise and informative title that accurately reflects the content of the paper.
• Abstract : A brief summary of the paper, typically no more than 250 words, that includes the purpose of the study, the methods used, the key findings, and the main conclusions.
• Introduction : An overview of the background, context, and motivation for the research, including a clear statement of the problem being addressed and the objectives of the study.
• Literature review: A critical analysis of the relevant research and scholarship on the topic, including a discussion of any gaps or limitations in the existing literature.
• Methodology : A detailed description of the methods used to collect and analyze data, including any experiments or simulations, data collection instruments or procedures, and statistical analyses.
• Results : A clear and concise presentation of the findings, including any relevant tables, graphs, or figures.
• Discussion : A detailed interpretation of the results, including a comparison of the findings with previous research, a discussion of the implications of the results, and any recommendations for future research.
• Conclusion : A summary of the key findings and main conclusions of the study.
• References : A list of all sources cited in the paper, formatted according to IEEE guidelines.

In addition to these elements, an IEEE research paper should also follow certain formatting guidelines, including using 12-point font, double-spaced text, and numbered headings and subheadings. Additionally, any tables, figures, or equations should be clearly labeled and referenced in the text.

AMA (American Medical Association) Style

AMA (American Medical Association) Style Research Paper Format:

• Title Page: This page includes the title of the paper, the author’s name, institutional affiliation, and any acknowledgments or disclaimers.
• Abstract: The abstract is a brief summary of the paper that outlines the purpose, methods, results, and conclusions of the study. It is typically limited to 250 words or less.
• Introduction: The introduction provides a background of the research problem, defines the research question, and outlines the objectives and hypotheses of the study.
• Methods: The methods section describes the research design, participants, procedures, and instruments used to collect and analyze data.
• Results: The results section presents the findings of the study in a clear and concise manner, using graphs, tables, and charts where appropriate.
• Discussion: The discussion section interprets the results, explains their significance, and relates them to previous research in the field.
• Conclusion: The conclusion summarizes the main points of the paper, discusses the implications of the findings, and suggests future research directions.
• References: The reference list includes all sources cited in the paper, listed in alphabetical order by author’s last name.

In addition to these sections, the AMA format requires that authors follow specific guidelines for citing sources in the text and formatting their references. The AMA style uses a superscript number system for in-text citations and provides specific formats for different types of sources, such as books, journal articles, and websites.

Harvard Style

Harvard Style Research Paper format is as follows:

• Title page: This should include the title of your paper, your name, the name of your institution, and the date of submission.
• Abstract : This is a brief summary of your paper, usually no more than 250 words. It should outline the main points of your research and highlight your findings.
• Introduction : This section should introduce your research topic, provide background information, and outline your research question or thesis statement.
• Literature review: This section should review the relevant literature on your topic, including previous research studies, academic articles, and other sources.
• Methodology : This section should describe the methods you used to conduct your research, including any data collection methods, research instruments, and sampling techniques.
• Results : This section should present your findings in a clear and concise manner, using tables, graphs, and other visual aids if necessary.
• Discussion : This section should interpret your findings and relate them to the broader research question or thesis statement. You should also discuss the implications of your research and suggest areas for future study.
• Conclusion : This section should summarize your main findings and provide a final statement on the significance of your research.
• References : This is a list of all the sources you cited in your paper, presented in alphabetical order by author name. Each citation should include the author’s name, the title of the source, the publication date, and other relevant information.

In addition to these sections, a Harvard Style research paper may also include a table of contents, appendices, and other supplementary materials as needed. It is important to follow the specific formatting guidelines provided by your instructor or academic institution when preparing your research paper in Harvard Style.

Vancouver Style

Vancouver Style Research Paper format is as follows:

The Vancouver citation style is commonly used in the biomedical sciences and is known for its use of numbered references. Here is a basic format for a research paper using the Vancouver citation style:

• Title page: Include the title of your paper, your name, the name of your institution, and the date.
• Abstract : This is a brief summary of your research paper, usually no more than 250 words.
• Introduction : Provide some background information on your topic and state the purpose of your research.
• Methods : Describe the methods you used to conduct your research, including the study design, data collection, and statistical analysis.
• Results : Present your findings in a clear and concise manner, using tables and figures as needed.
• Discussion : Interpret your results and explain their significance. Also, discuss any limitations of your study and suggest directions for future research.
• References : List all of the sources you cited in your paper in numerical order. Each reference should include the author’s name, the title of the article or book, the name of the journal or publisher, the year of publication, and the page numbers.

ACS (American Chemical Society) Style

ACS (American Chemical Society) Style Research Paper format is as follows:

The American Chemical Society (ACS) Style is a citation style commonly used in chemistry and related fields. When formatting a research paper in ACS Style, here are some guidelines to follow:

• Paper Size and Margins : Use standard 8.5″ x 11″ paper with 1-inch margins on all sides.
• Font: Use a 12-point serif font (such as Times New Roman) for the main text. The title should be in bold and a larger font size.
• Title Page : The title page should include the title of the paper, the authors’ names and affiliations, and the date of submission. The title should be centered on the page and written in bold font. The authors’ names should be centered below the title, followed by their affiliations and the date.
• Abstract : The abstract should be a brief summary of the paper, no more than 250 words. It should be on a separate page and include the title of the paper, the authors’ names and affiliations, and the text of the abstract.
• Main Text : The main text should be organized into sections with headings that clearly indicate the content of each section. The introduction should provide background information and state the research question or hypothesis. The methods section should describe the procedures used in the study. The results section should present the findings of the study, and the discussion section should interpret the results and provide conclusions.
• References: Use the ACS Style guide to format the references cited in the paper. In-text citations should be numbered sequentially throughout the text and listed in numerical order at the end of the paper.
• Figures and Tables: Figures and tables should be numbered sequentially and referenced in the text. Each should have a descriptive caption that explains its content. Figures should be submitted in a high-quality electronic format.
• Supporting Information: Additional information such as data, graphs, and videos may be included as supporting information. This should be included in a separate file and referenced in the main text.
• Acknowledgments : Acknowledge any funding sources or individuals who contributed to the research.

ASA (American Sociological Association) Style

ASA (American Sociological Association) Style Research Paper format is as follows:

• Title Page: The title page of an ASA style research paper should include the title of the paper, the author’s name, and the institutional affiliation. The title should be centered and should be in title case (the first letter of each major word should be capitalized).
• Abstract: An abstract is a brief summary of the paper that should appear on a separate page immediately following the title page. The abstract should be no more than 200 words in length and should summarize the main points of the paper.
• Main Body: The main body of the paper should begin on a new page following the abstract page. The paper should be double-spaced, with 1-inch margins on all sides, and should be written in 12-point Times New Roman font. The main body of the paper should include an introduction, a literature review, a methodology section, results, and a discussion.
• References : The reference section should appear on a separate page at the end of the paper. All sources cited in the paper should be listed in alphabetical order by the author’s last name. Each reference should include the author’s name, the title of the work, the publication information, and the date of publication.
• Appendices : Appendices are optional and should only be included if they contain information that is relevant to the study but too lengthy to be included in the main body of the paper. If you include appendices, each one should be labeled with a letter (e.g., Appendix A, Appendix B, etc.) and should be referenced in the main body of the paper.

APSA (American Political Science Association) Style

APSA (American Political Science Association) Style Research Paper format is as follows:

• Title Page: The title page should include the title of the paper, the author’s name, the name of the course or instructor, and the date.
• Abstract : An abstract is typically not required in APSA style papers, but if one is included, it should be brief and summarize the main points of the paper.
• Introduction : The introduction should provide an overview of the research topic, the research question, and the main argument or thesis of the paper.
• Literature Review : The literature review should summarize the existing research on the topic and provide a context for the research question.
• Methods : The methods section should describe the research methods used in the paper, including data collection and analysis.
• Results : The results section should present the findings of the research.
• Discussion : The discussion section should interpret the results and connect them back to the research question and argument.
• Conclusion : The conclusion should summarize the main findings and implications of the research.
• References : The reference list should include all sources cited in the paper, formatted according to APSA style guidelines.

In-text citations in APSA style use parenthetical citation, which includes the author’s last name, publication year, and page number(s) if applicable. For example, (Smith 2010, 25).

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Delimitations in Research – Types, Examples and...

Research Objectives – Types, Examples and...

APA Table of Contents – Format and Example

Theoretical Framework – Types, Examples and...

Appendices – Writing Guide, Types and Examples

Research Methodology – Types, Examples and...

• Foundations
• Write Paper

Search form

• Experiments
• Anthropology
• Self-Esteem
• Social Anxiety

• Research Paper >

Research Paper Example

To fully understand what information particular parts of the paper should discuss, here’s another example of a research paper.

This article is a part of the guide:

• Outline Examples
• Example of a Paper
• Write a Hypothesis
• Introduction

Browse Full Outline

• 1 Write a Research Paper
• 2 Writing a Paper
• 3.1 Write an Outline
• 3.2 Outline Examples
• 4.1 Thesis Statement
• 4.2 Write a Hypothesis
• 5.2 Abstract
• 5.3 Introduction
• 5.4 Methods
• 5.5 Results
• 5.6 Discussion
• 5.7 Conclusion
• 5.8 Bibliography
• 6.1 Table of Contents
• 6.2 Acknowledgements
• 6.3 Appendix
• 7.1 In Text Citations
• 7.2 Footnotes
• 7.3.1 Floating Blocks
• 7.4 Example of a Paper
• 7.5 Example of a Paper 2
• 7.6.1 Citations
• 7.7.1 Writing Style
• 7.7.2 Citations
• 8.1.1 Sham Peer Review
• 8.1.2 Advantages
• 8.1.3 Disadvantages
• 8.2 Publication Bias
• 8.3.1 Journal Rejection
• 9.1 Article Writing
• 9.2 Ideas for Topics

It includes some key parts of the paper such as the Abstract , Introduction , Discussion and References :

Text center-aligned and placed at the middle of the page, stating the title of the paper, name of author and affiliation.

A Study on the Factors Affecting the Infant Feeding Practices

Of Mothers in Las Piñas City

By [Author], University of the Philippines

The abstract starts on the next page, page 2. The text starts at the top, left flushed, double-spaced.

Abstract [Abstract here]

The body text starts on the next page, page 3. The text starts at the top, left flushed, double-spaced.

Introduction The melamine controversy that erupted during the last quarter of year 2008 brought people’s attention back to the debates between breastfeeding and the use of breast milk substitutes like commercial infant formula. This wasn’t the first time that infant formula had caused illnesses and even deaths to infants worldwide - hence the continuous campaign of World Health Organization (WHO) and UNICEF along with other breastfeeding advocates, for mothers to breastfeed their children at least until 6 months of age. Infant feeding practices refer generally to meet the nutritional and immunological needs of the baby. A study of infant feeding practices was carried out on a sample of 100 mother and infant pairs. The results revealed that only 20% of mothers in the study currently exclusively breastfeed their babies. It also shows that socio-economic factors like mother’s work status, marital status and educational attainment had direct bearing on these practices. Employed mothers tend to cease from breastfeeding their babies and eventually stop and just resort to formula feeding as they go back to work. The study also showed that mothers who are married and living with their partners are more likely to breastfeed their infants than single mothers. Those with higher educational attainment resort more to formula feeding and mixed feeding than those with lower educational attainment. Health care professionals influence mothers the most when it comes to infant feeding decisions. Methodology Type of Research The type of research that will be used in this study is qualitative research and quantitative research. Qualitative researchers aim to gather an in-depth understanding of human behavior and the reasons that govern such behavior. The discipline investigates the “why” and “how” of decision making. Besides this, the researcher will also examine the phenomenon through observations in numerical representations and through statistical analysis. Along with questionnaires that will be given out to respondents for the statistical representation of the findings in the study, interviews with the respondents and a few experts in this field will also be conducted. Sampling Method The research sampling method that will be used in this study is random sampling to obtain a more scientific result that could be used to represent the entirety of the population. A list of all health care facilities (maternity and lying-in clinics, public and private hospitals, health centers) was acquired from the Las Piñas City Hall. From 20 barangays, 3 will be picked through random sampling. The health care facilities and institutions in these three barangays will then be the target sources of respondents of the researcher. The health care facilities and institutions will be contacted to obtain a verbal consent to administer the questionnaire to mothers at their places. A letter of consent will also be sent to them along with a sample copy of the questionnaire that will be used, as well as the protocol of the researcher. A letter was also addressed to the City Health Officer to obtain endorsement and consent to conduct a research in selected barangays and distribute questionnaires to the mothers in the vicinity. Data collection was conducted throughout the facilities‟ and health centers‟ operating hours from Mondays through Sundays in order to include both working and non-working mothers. Respondents The respondents in this research will all be coming from one single location - Las Piñas City, specifically the randomly selected barangays of Pamplona I, CAA/BF International and Pamplona III. The researcher chose Las Piñas City because of the socio-economic conditions present in the area that is relevant to the study and also as it fits the time frame and resources of the researcher. The randomly sampled respondents will be asked by the researcher for consent and approval to answer the questionnaire until the desired number of respondents which is 100 is reached. The opinion of experts will also be sought in this research to provide explanations regarding the respondents‟ infant feeding behaviors and practices. Questionnaire The questionnaire requires information about the socio-economic and demographic background of the mother. It also has questions related to previous infant feeding practices and the birth of her youngest infant and also regarding the baby’s general health and age. Statements that are perceived to be factors that influence mothers‟ infant feeding decisions were presented. The description of the type of infant formula given by formula and mixed feeding mothers will also be asked in the material. Conclusion Majority of the mothers formula feed their child and only a minority exclusively breastfeeds their children, especially as per recommendation of the World Health Organization. While majority of the mothers in this study showed a positive attitude towards breastfeeding, most of them decided only to formula feed due to the reasons of insufficient milk supply and work. Based on the results of the study, the educational attainment, work status, marital status, and seminars in the barangay the respondents are part of, about breastfeeding, are the significant factors that affect the infant feeding decision of mothers in Las Piñas City. Majority of the mothers that served as respondents in this study fall under the age range of 17-30 years old. More than half of them were also college graduates while a significant number are undergraduates and have only reached until high school. Most of the mothers are housewives and the others remaining have full-time jobs, part-time jobs and self-employed. A few of them are still students. While majority of them were married, a lot were still in a status of live-in and are single. More than half of the mothers did not have previous children before the current one. Majority of the respondents also have an annual gross household income that does not exceed P50,000. Among the several information sources namely, media through televisions/radios and printed/published materials, the social support system comprised of the mother’s family, friends and other relatives and health institutions, the mothers who give their babies infant formula are influenced the most by health care institutions through health professionals and other health care personnel. They influence the mothers in deciding to feed the baby with formula and in choosing, as well, which brand of formula is best for their babies. Mothers trust their baby’s doctor because of their expertise in the said field hence this kind of relation is achieved. Mothers were overall not concerned about the possible side effects of breastfeeding as a few were only worried as shown in the data presented.       It can be concluded that numerous internal as well as external factors influence a mother in making infant feeding decisions, and a greater fraction of these is socio-economic in nature.

• Psychology 101
• Flags and Countries
• Capitals and Countries

Explorable.com (Jun 26, 2009). Research Paper Example. Retrieved Jun 28, 2024 from Explorable.com: https://explorable.com/research-paper-example

You Are Allowed To Copy The Text

The text in this article is licensed under the Creative Commons-License Attribution 4.0 International (CC BY 4.0) .

This means you're free to copy, share and adapt any parts (or all) of the text in the article, as long as you give appropriate credit and provide a link/reference to this page.

That is it. You don't need our permission to copy the article; just include a link/reference back to this page. You can use it freely (with some kind of link), and we're also okay with people reprinting in publications like books, blogs, newsletters, course-material, papers, wikipedia and presentations (with clear attribution).

Related articles

Example of a Research Paper

Want to stay up to date? Follow us!

Check out the official book.

Learn how to construct, style and format an Academic paper and take your skills to the next level.

(also available as ebook )

Save this course for later

Don't have time for it all now? No problem, save it as a course and come back to it later.

Footer bottom

• Privacy Policy

• Subscribe to our RSS Feed
• Like us on Facebook
• Follow us on Twitter

Sociology Research Paper

This sample sociology research paper features: 10800 words (approx. 36 pages), an outline, and a bibliography with 59 sources. Browse other research paper examples for more inspiration. If you need a thorough research paper written according to all the academic standards, you can always turn to our experienced writers for help. This is how your paper can get an A! Feel free to contact our writing service for professional assistance. We offer high-quality assignments for reasonable rates.

More Sociology Research Paper Examples:

• Disaster Research Paper
• Gambling Research Paper
• Group Dynamics Research Paper
• Human Ecology Research Paper
• Knowledge Research Paper
• Love Research Paper
• Quality of Life Research Paper
• Social Change Research Paper
• Social Movements Research Paper
• Social Networks Research Paper
• Social Stratification Research Paper
• Socialization Research Paper
• Sociology of Religion Research Paper

Introduction

The early sociology, the foundation of social science: statistical studies, the rise of american sociology, the substance of the sociological perspective, the passion for sociology, conclusion: the future of sociology.

• Bibliography

A commonly accepted definition of sociology as a special science is that it is the study of social aggregates and groups in their institutional organization, of institutions and their organization, and of the causes and consequences of changes in institutions and social organization. (Albert J. Reiss, Jr. 1968:1)

Academic Writing, Editing, Proofreading, And Problem Solving Services

Get 10% off with 24start discount code.

Within the contemporary context, sociologists are interested in human social interaction as people take one another into account as each behaves toward the other. Sociologists also take into analytical consideration the systemic units of interaction within social groups, social relations, and social organizations. As stated by Reiss (1968), the purview of sociology extends to

Governments, corporations, and school systems to such territorial organizations as communities or to the schools, factories, and churches . . . that are components of communities. . . . are also concerned with social aggregates, or populations, in their institutional organization. (P. 1) (adsbygoogle = window.adsbygoogle || []).push({});

Sociology is, as Touraine (1990) suggests, an interpretation of social experience and is thus a part of the reality that the practitioners of the discipline attempt to observe and explain. To these areas we can add that sociology is a discipline that demystifies its subject matter, and it is, as Dennis H. Wrong (1990:21–22) notes, a debunker of popular beliefs, holds skeptical and critical views of the institutions that are studied (Smelser 1990), and challenges myth making (Best 2001).

The early history of sociology is a history of ideas developed in the European tradition, whereas the sociological approach of the last 150 years involved the development of concepts, methodology, and theories, especially in the United States (Goudsblom and Heilbron 2001). As American sociologists trained in the traditional theory and methods developed during the first eight decades of the twentieth century, we acknowledge our intellectual debt to the European founders. But beyond an earnest recognition of the classic work of the early founders, including Auguste Comte, Émile Durkheim, Alexis de Tocqueville, Frederic LePlay, Marcell Mauss, Max Weber, Karl Marx, and Harriet Martineau, most of whom were attracted to the European environment that included the liberalism, radicalism, and conservatism of the early to mid-nineteenth century (Nisbet 1966; Friedrichs 1970) and to what C. Wright Mills (1959) refers to as the sociological imagination that “enables us to grasp history and biography and the relations between the two within society” (p. 6), our approach to sociology is deeply embedded with and indebted to those individuals who established the Chicago, Harvard, Iowa, and Berkeley schools of thought. Similarly, as practitioners, our approach to the discipline of sociology is reflected in these distinctive American scholarly perspectives.

The American tradition of sociology has focused on social policy issues relating to social problems, the recognition of which grew out of the dynamic periods of social transformation wrought by the Industrial Revolution, the Progressive Era, world crises engendered by war, worldwide population shifts, increasing mechanization, and the effort of sociologists to create a specific niche for the discipline within a growing scientific community. This effort occurred first in North America and Western Europe and then, similar to cultural transitions of the past, within a global context. In every instance, the motives embedded within a science of society lie in the attempt to understand and offer proposals for solutions to whatever problems gain significant attention at a particular point in time.

In a most interesting work, Goudsblom and Heilbron (2001) pose that sociology represents a great diversity, or what some analysts may refer to as fragmentation, because the discipline grew as a part of the processes affecting societies and cultures worldwide throughout the nineteenth and twentieth centuries. Thus, as we move well into a new era and a new stage of academic development, it remains important that we recognize the sociological heritage as identified and discussed by these analysts. The five stages that sociology has experienced to date are (1) the predisciplinary stage prior to 1830, further identified as “protosociologies”; (2) the formation of the intellectual discipline, 1830–1890; (3) the formation of an academic discipline with diverging national traditions, 1890–1930; (4) the establishment of an international academic discipline, 1930–1970; and (5) a period of crisis, fragmentation, and attempts to develop a new synthesis, 1970–2000 (Goudsblom and Heilbron 2001:14574–80).

Consistent with the fifth stage, for almost four decades we have been witness to major changes in the substantive topics that undergo sociological inquiry both in the United States and, given the influence on the discipline by Canadian, European, and Scandinavian scholars, internationally. Among the areas more fully developed that might be identified as fragmentation are many of the most interesting sociological topics, including deviant behavior, the family, religion, gender, aging, health, the environment, science and technology, among so many seemingly unrelated topics. The unique conceptual paradigms of sociology serve as a template or pattern for seeing the social world in a special way. Every discipline and, indeed, every occupation employs templates or patterns to see and accomplish things in a unique fashion. Disciplines such as sociology rely on intellectual templates based on certain conceptual schemes or paradigms that have evolved through the development of a body of knowledge in those disciplines.

In its early era of the mid- to late nineteenth century, sociology was understood to represent anything relating to the study of social problems. Indeed, it was thought that the methods of the social sciences could be applied to social problems and used to develop solutions (Bernard and Bernard 1943). In focusing on such substance, O’Neill (1967:168–69) notes that periodicals of this early period had a sociological section in which news items relating to family matters, poverty, and labor often appeared. These early social scientists did not hold any special talents other than their training in theology. This situation was similar in the United States as well. It is not difficult, then, to imagine that, as Bramson (1961) notes, “For many American sociologists these problems evoked a moral response” (p. 75). Thus, the process of solving the problems of society was attempted by application of the conventional morality and the validation of Christian principles of piety rather than reform or progress.

Sociology was born as a result of a process, a process that directed a method of inquiry away from philosophy and toward positivism (MacIver 1934). Sociology was the result of a process caused by two major forces—namely, the Industrial Revolution and the French Revolution. The events, changes, and ideas that emerged from these two revolutions are found in the nineteenth-century thought pertaining to social order (Eisenstadt 1968). Following in the wake of the Age of Reason and the Renaissance, according to Nisbet (1966), this was a period of word formation:

Perhaps the richest period of word formation in history . . . which were either invented during this period or were modified to their present meanings: industry, industrialist, democracy, class, middle class, ideology, intellectual, rationalism, humanitarian, atomistic, masses, commercialism, proletariat, collectivism, equalitarian, liberal, conservative, scientist, crisis . . . [among others]. (P. 23)

These were words that held great moral and partisan interest in the European economy and culture; such passions were identified with politics as well.

Identified with European conservatism, which became infused by and with science, the visionary perspective promoted by Auguste Comte during the 1830s in his six-volume Positive Philosophy, later translated from the French and condensed into two volumes by Harriet Martineau, was based on the medieval model of European society.

This model of family, community, authority, tradition, and the sacred became the core of scientific sociology that was to serve notice that a science of society was essential to provide for more than commonsense analysis and to reestablish social order (MacIver 1934). Although unsuccessful in his quest to secure a professorship, Auguste Comte was a positivist, mathematician, and promoter of the scientific identity of the engineering profession (Noble 1999). Comte argued that positivism and the still-to-beidentified area of “sociology” would serve as a means of supporting his intention to create a unique perspective of human relations and a system to reestablish the social order and organization of society. Reestablishment of this new social order was to proceed in accordance with the positivist stage of evolution with its ineluctable natural laws that could and would be established through engaging the scientific perspective. Along with the arts, the science of sociology, according to Comte, was to emerge as the queen of the sciences, the scientia scientorum, and would ultimately supplant biology and cosmology.

If the restoration of order in French society was a preoccupation for many early-nineteenth-century scholars, including Auguste Comte, it was also the case, as Bramson (1961) notes, that

many of the key concepts of sociology illustrate this concern with the maintenance and conservation of order; ideas such as status, hierarchy ritual, integration, social function and social control are themselves a part of the history of the reaction to the ideals of the French Revolution. What conservative critics saw as resulting from these movements was not the progressive liberation of individuals, but increasing insecurity and alienation, the breakdown of traditional associations and group ties. (Pp. 13–14)

For social scientists of the early nineteenth century, many of the problems of the time were much more well defined than is the case in the contemporary experience.

Comte was fervently religious, and he believed those interested in science would constitute a “priesthood of positivism” that would ultimately lead to a new social order. According to Noble (1999),

A theist in spite of himself, Comte declared that the existence of the Great Being “is deeply stamped on all its creations, in moral, in the arts and sciences, in industry,” and he insisted, as had previous like-minded prophets since Erigena, that all such manifestations of divinity were equally vital means of mankind’s regeneration . . . Comte was convinced that people like himself, science-minded engineering savants occupied with the study of the sciences of observation are the only men whose capacity and intellectual culture fulfill the necessary conditions. (P. 85)

The legacy of this enthusiastic perspective is that sociology has been at the heart of the positivists’ contribution to the understanding of the human condition. It was also to serve in part as a basis for the reactions of conflict theorist Karl Marx, especially as these writings referred to the religious opiate of the masses deemed by Comte as critical to the reorganization of society (Noble 1999:87). The discipline continues to present an array of perspectives that have served to stimulate much controversy within both society and the discipline (see Turner 2001).

Although the sociological legacy of Harriet Martineau is substantial, as outlined by Lengermann and NiebruggeBrantley (1998), it was Martineau’s effort to translate and condense Auguste Comte’s six-volume magnum opus into a two-volume set of writings published in 1853 that allowed this important work to be available to the Englishspeaking world. Interestingly, Comte’s English translation came after Martineau’s sociological contributions, the richness of which was finally recognized by feminist researchers during the 1980s and 1990s. Martineau engaged in “participant observation” of the United States during the mid-1830s and subsequently published the two-volume Society in America (1836/1837), which is based on this excursion to the North American continent. Because of this experience, Martineau was able to lay the foundation for her treatise on research methodology in How to Observe Morals and Manners (1838).

Perhaps it is ironic that the distinctive difference between the European theoretical sociology and the empirical sociology practiced in the United States was advanced by events in Europe. Indeed, the origin of empirical sociology is rooted in Europe. Statistical studies began in the 1660s, thereby preceding the birth of all of the social sciences by a couple of centuries. The early statistical gatherers and analysts were involved in “political arithmetic” or the gathering of data considered relevant to public policy matters of the state, and as noted by Reiss (1968), the gathering of such data may have been accelerated to meet the needs of the newly emerging insurance industry and other commercial activities of the time. But it was the early work of the moral statisticians interested in reestablishing social order in the emerging industrial societies that was to lay the quantitative foundation for the discipline, especially the early scientific work of the French sociologist Émile Durkheim (Whitt 2001:229–35).

The second stage in the early history of quantification may have been related to the development of probability theory, the rise of the insurance industry, other commercial activities, and political necessity (Lecuyer and Oberschall 1968; Reiss 1968). English political arithmeticians, including John Graunt and William Petty, were destined to be followed by the efforts of the moral statisticians who engaged in data gathering in Belgium and France. Indeed, as early as 1831, the Belgian Adolphe Quetelet and the Frenchman Andre Michel de Guerry de Champneuf, in building on the early efforts of the practitioners of the “political arithmetic” that first began in the 1660s, were engaging in the government-sponsored data-gathering activity pertaining to data on moral topics, including suicide, prostitution, and illegitimacy. Such activities would prove quite instrumental in the establishment of the empirical social sciences. Even many of the methodologies developed during this same era of the early nineteenth century, as well as awareness of important ecological methodological issues such as statistical interactions, the ecological fallacy, and spuriousness, were developed by early moral statisticians such as Andre-Michel de Guerry and Adolphe Quetelet. Later, the work of Henry Morselli, Enrico Ferri, and Alfred Maury during this same century were to serve well the needs of aspiring European sociologists and even later members of the Chicago School of Sociology (Whitt 2001:229–31).

American sociology is one of the intellectual creations that has most deeply influenced our century. No other society ( the American ) has been more actively involved in understanding its own organizational change for the sake of knowledge itself. (Touraine 1990:252)

The birth of the social sciences in general and of sociology in particular is traced to the liberal democratic ideas generated by the British social philosophies of the seventeenth century—ideas that later were to be enhanced by the French Enlightenment of the eighteenth century and then transformed in the United States where these ideas served as the foundation for practical democratic society. The rise of American sociology can be traced to the early-nineteenthcentury social science movement, a movement that by the mid-1800s became a new discipline that was widely introduced into college and university curricula. The movement also led to the establishment of a national social science association that was to later spawn various distinctive social sciences, including sociology, as well as social reform associations (Bernard and Bernard 1943:1–8).

Although the promotion of the social sciences in the United States began as early as 1865 with the establishment of the American Association for the Promotion of Social Sciences and then, in 1869, creation of the American Social Science Association with its associationsponsored publication the Journal of Social Science, prior to the 1880s there had been no organized and systematic scientific research in the United States. This was the case simply because, as Howard W. Odum ([1927] 1965:3–20) noted, there was no university per se in which research as a scientific pursuit could be conducted. It is within the context of the movement to organize such a university that sociology and many other social sciences were embraced as viable academic disciplines, thereby allowing systematic research to be conducted in a rigorous manner. This also was a period of great emphasis on pursuing answers to new research questions through the evaluation of knowledge and the employment of methodological and statistical tools within an interdisciplinary context. Indeed, L. L. Bernard and Jessie Bernard (1943) posit that the vision of the founders of the American Social Science Association was “to establish a unified science of society which could and would see all human problems in their relationships and make an effort to solve these problems as unified wholes” (p. 601).

Thus, the social sciences in general and sociology in particular owe a great intellectual debt to the American intellects who studied at length with the masters of Europe. Included among these are notables such as William Graham Sumner, Lester Frank Ward, Albion Woodbury Small, Franklin Henry Giddings, John William Burgess, Herbert B. Adams, Thorstein Veblen, Frederick Jackson

Turner, James Harvey Robinson, George Vincent, Charles Horton Cooley, Edward Alsworth Ross, George Howard, Frank W. Blackmar, Ulysses G. Weatherly, John R. Commons, and Richard T. Ely (see Odum 1951, [1927] 1965); each of whom were well versed in scholarly areas other than sociology, including history, theology, economics, political science, and statistics. With the decline of the social science movement and its national association, the general discipline that emerged from the remains of social science was in fact sociology (Bernard and Bernard 1943:835).

The development of an intellectual and academic American sociology, like sociology in any part of the world, was and continues to be dependent on the social and political conditions of the country. In the United States, a liberal political climate and, in the aftermath of the Civil War, the advent of a system of a mass public education system, American sociology flourished. Thus, in countries in which the structure of the system of higher education was open to free inquiry, research was supported by private foundations and government contributions (Wright 1895), and the university was organized albeit loosely, sociology, subject to the polemics of its status as an academic science, gained entry if not acceptance among university faculty. Where education was available to the elite rather than the masses, sociology was less apt to flourish (Reiss 1968).

Another important factor is that American sociology arose basically without roots other than the growing influence of the social science movement in the United States and the emphasis on the virtues of science that permeated the intellectual and social environs of this same period. As noted by Neil J. Smelser (1990:49–60), American sociology did not experience the yoke of either European feudalism or any peculiar intellectual history. Rather, sociology came into being within American higher education during the 1880s and only after several other disciplines, including psychology and economics, had been accepted within the academy. Attempts among adherents of these other disciplines led to the establishment of the scientific theme within the social sciences. Early sociologists embraced this same scientific theme.

A second factor that had a profound effect on the early adherents of the sociological perspective is the social reform theme of the 1890s. The legacy of these two themes—namely, scientific respectability and social reform—became the dual platforms on which the unique American sociological perspective was to be based.

Although there was a great, direct influence of European thought, research, and the philosophy of the British Social Science Association on sociology to focus on attempting to solve America’s problems (Odum 1951:36–50), the rise of American sociology, at least during the first half of the twentieth century, was concomitant with the most dynamic period of technological, economic, and social reform changes ever recorded. In this context, Howard W. Odum (1951:52) views sociology as a product of the American social and cultural experience and places sociology’s heritage to be as “American as American literature,American culture, and the freedoms of the new world democracy” (p. 3). American sociology is thus part European and part American. Indeed, American sociology was envisioned early on as a social science that could and would assist policymakers and concerned citizens in creating the “American Dream.”

Consistent with this ideology, Odum (1951:59–60) identified three unique American developments, each of which influenced the direction of American sociology throughout the entire twentieth century. The first of these developments is the symbiotic relationship between the discipline and the American society and culture. The ideology that focused on the American Dream and its realization had a great influence.

The second development, according to Odum, is the emphasis on moral development and the motivation to establish ethics as a component of the educational curricula,American literature, and the social sciences, especially as these relate to ethical conduct, social justice, and public morality. Within sociology, this orientation is found in the application of sociological principles into economic and organizational behavior and the founding of the American Institute of Christian Sociology.

Finally, Odum (1951) notes, the American experience led to a research emphasis on social problems of a moral and economic nature. In an effort to better understand these social problems, sociologists organized the systematic study of issues such as waves of immigration, the working class, public disorder, neglect of children, violence toward women, intergroup conflict, urbanism, alcoholism, suicide, crime, mental illness, delinquency, and poverty (see also Fine 2006). This was the application side of sociology that held important social policy implication. However, there was also an early emphasis on a “general sociology” as opposed to a “special sociology” as was found at the more elite institutions of higher learning. Clearly, this difference foreshadowed the pure versus applied dichotomy that has generated so much discussion within the discipline (see Odum 1951:51–74).

Because of the important influence of the social science movement in the United States, there is some disagreement pertaining to who the founders and members of the first generation of American sociologists are (see Odum 1951, [1927] 1965). But publication of Lester Ward’s book Dynamic Sociology in 1883 does appear to mark the beginning of American sociology (Bramson 1961:84–85). On the other hand, there does not seem to be any disagreement as to the purpose of the American founders, and that was to establish a scientific theoretical base. Later, at the University of Chicago the goals were to establish a relationship between sociology and the classical problems of philosophy by focusing on process issues relating to elements of social control, such as conflict, competition, and accommodation (Kurtz 1986:95).

American sociology emerged concomitant with the challenges to legal philosophy and the discussion of questions relating to myriad questions that arose as the effects of industrialization were observed Calhoun (1919). Such questions have their focus on marriage, divorce, immigration, poverty, and health and how to employ the emerging scientific model to topical data that had been gathered by the nineteenth-century moral statisticians.

Leon Bramson (1961:47–48) observed that the most interesting aspect of American sociology in the first half of the twentieth century is that when affected by European theories of mass behavior and collective behavior, American sociologists, in their haste to establish a role for sociology in America, either transformed the meaning of the concepts to meet their needs or created new concepts to apply to the more liberal American social and political context. American sociologists, according to Bramson, also applied European theoretical concepts such as social pathology, social disorganization, and social control to the data referring to the American experience without regard for whatever special conditions should have been accounted for or even possible theoretical distortions; this issue is also discussed by Lester R. Kurtz (1986:60–83) in his evaluation of the Chicago School of Sociology.

Albert J. Reiss, Jr. (1968) notes that the first formal instruction of a sociology course in the United States was offered by William Graham Sumner, a professor of political and social science at Yale University, during 1876. The first, second, and third American Departments of Sociology were established at Brown University, the University of Chicago, and Columbia University, respectively (Kurtz 1986:93–97). Between 1889 and 1892, 18 American colleges and universities offered instruction in sociology, but in 1893, the University of Chicago was the first to develop a program that led to the granting of a Ph.D.

Despite the recognition of the emerging field of sociology as a distinctive area of inquiry, the focal point of a religious orientation and perhaps fervor expressed by social commentators in their discussions and analyses of the social issues that were to constitute the purview of sociology also engaged the attention of other early practitioners of the discipline. The social problems identified in the wake of expansion of the American West and the building of the railroads included issues relating to “the influx of immigrants, the rise of the factory system and the concentration of people in big cities. These comprised the now familiar catalogue of crime, delinquency, divorce, poverty, suicide, alcoholism, minority problems and slums” (Bramson 1961:75).

Alfred McClung Lee (1978:69) notes that ever since that time, sociologists have been attempting to divorce themselves from an ancestry that is historically rooted in the clergy, the police, utopian ideologues, social reformers, conservative apologists, journalistic muckrakers, radical thinkers, agitators, and civil libertarians.

Given the moral tone of much of the writing of many early American sociologists, it is noteworthy that in articulating the six “aims” of the American Journal of Sociology established at the University of Chicago in 1895, the scientific view of sociological concern so clearly defined several decades later by E. A. Ross (1936) was not so clear to many if not all of the moral philosophers of this earlier period. Witness the following comments offered by the founding editor of the American Journal of Sociology, Albion W. Small (1895):

Sociology has a foremost place in the thought of modern men. Approve or deplore the fact at pleasure, we cannot escape it. . . . To many possible readers the most important question abut the conduct of the Journal will be with reference to its attitude toward “Christian Sociology.” The answer is, in a word, towards Christian sociology sincerely deferential, toward “Christian sociologists” severely suspicious. (Pp. 1, 15)

These comments were of particular significance given that the American Journal of Sociology was not only the first journal of sociology created anywhere, but it was also, until 1936, the official journal of the American Sociological Society. Thus, the influence of both the Chicago School and the large number of contributions by its faculty and students to the American Journal of Sociology placed the work of the Chicago School at the forefront in shaping the early direction and substance of American, Canadian, and Polish sociology (Kurtz 1986:93–97). This was especially true in the subareas of urban and community studies, race and ethnic relations, crime and juvenile delinquency, deviance, communications and public opinion, and political sociology.

Leon Bramson (1961:73–95) identified three important phases in the rise of American sociology. The first period began in 1883 with the publication of Lester Ward’s Dynamic Sociology to about 1915 or 1918 with the publication of Robert E. Park’s essay on the city and/or the end of World War I, respectively. During this period, the founders began their earnest quest to establish the theoretical foundation as it related to the American experience focusing on “a liberal sociology of change and process, rather than one of conservation and equilibrium” (Bramson 1961:85).

This focus on change and process became even more evident during the second stage of American sociology, identified as the period between the two world wars. This was a period of academic expansion, with major increases in faculty and students, but even more important, led by sociologists at the University of Chicago, this was a period of specialization and the beginning of differentiation within sociology as the quest to develop a viable methodology began in earnest. This also was a meaningful period during which sociologists worked to establish the scientific status of the discipline and to earn respectability and academic legitimization. It was also a period during which many of the conceptual problems of sociology first began to emerge as its practitioners developed an increasingly complex technical vocabulary, a vast array of classification schema, and other abstract systems categories of thought. Perhaps assuming the need to compensate for a past that included so many nonscientifically moral reformistoriented representatives of the discipline, sociologists responded during this phase of development by creating complex theories that, for an extended period of time, were not only unintelligible to the layperson, but also the abstract nature of these grand theories exceeded the ability of social scientists to create methodologies appropriate to empirically test these theoretical models (Lee 1978). But despite this theoretical/methodological problem, this second stage of sociological development was also one in which much substance was created.

The history of sociology in America from prior to World War I to approximately the mid-1930s is, according to Kurtz (1986), a history of the school of thought promoted by the University of Chicago. If the second phase of American sociology is to be distinguished as a period dominated by the Chicago sociologists, it is also one that led Pitirim Sorokin to observe that American sociology was emerging as a distinctive brand:

The bulk of the sociological works in America are marked by their quantitative and empirical character while the bulk of the sociological literature of Europe is still marked by an analytical elaboration of concepts and definitions; by a philosophical and epistemological polishing of words. (Cited in Bramson 1961:89)

The period is characterized by a marked increase in the development of new and expanding methodologies and measurement. These new techniques included a plethora of scales intended to measure the theoretical concepts developed previously.

As noted, Goudsblom and Heilbron (2001) identify five phases of development of the discipline that cover the period prior to 1830 to the very end of the twentieth century. But the third phase of the development of American sociology, identified by Bramson (1961) as covering the period from 1940 to 1960, is noteworthy because this was a period during which the development and adoption of theories of the “middle-range” advocated by Robert K. Merton led to even greater specialization and differentiation of the discipline. In turn, sociologists began to develop ever-expanding areas of inquiry. Robert K. Merton ([1957] 1968), who wrote in reaction to the abstractness of the previous dominant position of the functionalist school of sociology, stated that theories of the middle range are

theories that lie between the minor but necessary working hypotheses that evolve in abundance during day-to-day research and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organization and social change. (P. 39)

The all-inclusive efforts refer, of course, to the contributions of Talcott Parsons in The Structure of Social Action, originally published in 1937, and in 1951 with the appearance of The Social System.

The third phase of development can be characterized as the most enthusiastic period during which greater emphasis was placed on the application of sociological knowledge. As the field expanded, new outlets for sociological studies and knowledge were created, sociologists found employment in nonacademic settings such as government and business, and the new specialty areas of interest reflected the changes in American society, including a growing rise in membership in the middle class, the expansion of the suburbs, more leisure time, and the growth of bureaucracy. In lieu of the previous sociological interest in the reform of society and the more traditional social problems orientation of the discipline, the new sociology opted to leave such concerns to the social work profession and to special studies programs such as criminology. Thus, specialty areas emerged—areas such as the sociology of marriage and the family, and aging (later to be defined as gerontology), industrial sociology, public opinion, organizations, communications, and social psychiatry (later called mental health). From this point forward, the continued rise to respectability of sociology is attributed by analysts such as Robert Nisbet (1966) to the public recognition that societal problems are more integrative in nature than previously thought. This may also serve as a partial explanation for why the discipline is viewed by some as fragmented.

The logic and ethos of science is the search for the truth, the objective truth. Thus, the most fundamental problem the social scientist confronts, according to Gunnar Myrdal (1969), is this:

What is objectivity, and how can the student attain objectivity in trying to find out the facts and the causal relationships between facts? [That is,] How can a biased view be avoided? The challenge is to maintain an objectivity of that which the sociologist is a part. (P. 3)

Although the sociologies of the United States and Europe differ in perspective, both attempt to answer similar albeit distinguishable questions. In his discussion of “the two faces of sociology,” Touraine (1990:240) states that these differences lie in the scholarly research response to two problems: (1) How does society exist? (2) How are culture and society historically created and transformed by work, by the specific way nature and its resources are put to use, and through systems of political, economic, and social organization? Because the intellectual legacy of American sociological thought has been shaped to a large extent by the historical experience of creating a nation in which the rights and the will of the American people have been dominant, American sociologists have long focused on “institution” as a central concept and the significance of efforts of reform movements within the American society to affect its social organization. Thus, the substance of American sociology has been on topics such as the family, social organization, community, the criminal justice system, and law and society among the numerous institutionallevel areas of inquiry that are evaluated within the context of yet another American theoretical focus—namely, the emphasis on theories of the middle range. European sociologists, on the other hand, tend to focus on the second question while emphasizing the concept “revolution” in their analyses. Thus, even when similar topics such as social movements serve as the focus of inquiry, the American and European sociology responds from a different perspective (Touraine 1990). To understand the importance of this difference in perspective between the two sociologies, Alain Touraine (1990) poses the view that American sociology has a symbiotic relationship between culture and society, whereas European sociology integrates society and its history. Americans sociologists focus on society; the European sociology is focused on the rich history that serves as the backdrop for any attempt to understand social change.

Because the American experience is predicated on building a nation through the rule of law; the concepts of individualism, capitalism, and territorial conquest; and the attempt at integration of successive waves of immigrants to the North American continent,American sociology began its rise in prominence through an elitist intellectual process that dominated the academy during the early formative years of the discipline. Thus, it is perhaps ironic that an American sociology housed within the university setting would assume a critical teaching and research posture toward an elitist system of institutions that the early sociology assisted in creating. Within the context of certain kinds of social problems areas, such as ethnic studies, discrimination, and segregation, sociology and sociologists have been able to exert some influence. But in other important areas within which issues relating to elitist society may be involved, such as social class relations and economic and political power, the official and public perceptions of the efforts of American sociologists may not be as well received.

Many analysts of the past can be called on to render testimony in support of or apologize for the past efforts of sociologists to provide useful information, but none is perhaps more relevant than the following statement offered by George A. Lundberg (1947): “Good intentions are not a substitute for good techniques in either achieving physical or social goals” (p. 135). During the 1960s and 1970s, sociology, psychology, and other social science undergraduate job candidates customarily responded to interviewer queries with “I want to help people.” Similar to those who attended graduate school after World War II, these individuals were influenced by the potential of sociology to make a difference. But good intentions aside, the real issue is, How do we go about assisting/helping people? Perhaps the more educated and sophisticated we become, the more difficult are the answers to social problems and social arrangements that are deemed inappropriate or at least in need of some form of rearrangement. That is, the more we believe we already know the answers, the less apt we are to recognize the importance of the sociological perspective. Within this context, sociology necessarily must adhere to and advocate the use of the methods of science in approaching any social problem, whether this is local or international in scope.

Sociology has utility beyond addressing social problems and contributing to the development of new social policy. Indeed, the sociological perspective is empowering. Those who use it are in a position to bring about certain behavior in others. It has been said that “behavior that can be understood can be predicted, and behavior that can be predicted can likely be controlled.” It is not surprising that sociologists are often used to help select juries, develop effective advertising campaigns, plan political strategies for elections, and solve human relations problems in the workplace. As Peter Berger (1963) phrases it, “Sociological understanding can be recommended to social workers, but also to salesmen, nurses, evangelists and politicians—in fact to anyone whose goals involve the manipulation of men, for whatever purpose and with whatever moral justification” (p. 5). In some ways, it might be said that the sociological perspective puts one “in control.”

The manipulation of others, even for commendable purposes, however, is not without critical reaction or detractors. Some years back, industrial sociologists who worked for, or consulted with, industrial corporations to aid them to better address problems in the workplace were sometimes cynically labeled as “cow sociologists” because “they helped management milk the workers.” Knowledge is power that can be used for good or evil. The sociological perspective is utilitarian and empowering in that it can accomplish things for whatever purposes. Berger (1963) goes on to reflect the following:

If the sociologist can be considered a Machiavellian figure, then his talents can be employed in both humanly nefarious and humanly liberating enterprises. If a somewhat colorful metaphor may be allowed here, one can think of the sociologist as a condottiere of social perception. Some condottieri fight for the oppressors of men, others for their liberators. Especially if one looks around beyond the frontiers of America as well as within them, one can find enough grounds to believe that there is a place in today’s world for the latter type of condottiere. (P. 170)

Responding to the question, “Can science save us?” George A. Lundberg (1947) states “yes,” but he also equates the use of brain (the mind) as tantamount to employing science. Lundberg also posed the following: “Shall we place our faith in science or in something else?” (p. 142). Physical science is not capable of responding to human social issues. If sociologists have in a vain effort failed to fulfill the promise of the past, this does not indicate that they will not do so at some future time. Again, as Lundberg (1947) heeded long ago, “Science is at best a growth, not a sudden revelation. We also can use it imperfectly and in part while it is developing” (pp. 143–144).

And a few years later but prior to the turmoil that was to embroil the decades of the 1960s and 1970s, John Madge (1962) urged that a century after the death of the positivist Auguste Comte (now 150 years later) the structure of sociology remains incomplete. However, Madge recognized and demonstrates in The Origins of Scientific Sociology that sociology was slowly gaining in maturity and with this growth was on the verge of or within reach of achieving the status of a science. But it is also important to keep in focus the goals of science as articulated by Gunnar Myrdal (1969)—more specifically, “The goals of objectivity and effectiveness in research are honesty, clarity, and effectiveness” (p. 72). If the results of sociological research have been less than to the liking of policymakers and government and corporate leaders, then yet another of Myrdal’s insights is especially germane. That is,

Research is always and by logical necessity based on moral and political valuations, and the researcher should be obligated to account for them explicitly. When these valuations are brought out into the open any one who finds a particular piece of research to have been founded on what is considered wrong valuation can challenge it on that ground. (P. 74)

There are other reasons as well, reasons that complicate the delivery of the important message promoted by the discipline’s practitioners, for as noted by Joel Best (2003:11), sociology “is a perspective built on relativism, built on the recognition that people understand the world differently.” Indeed, many years earlier George C. Homans (1967) observed,

If some of the social sciences seem to have made little progress, at least in the direction of generalizing and explanatory science, the reason lies neither in lack of intelligence on the part of the scientists nor in the newness of the subject as an academic discipline. It lies rather in what is out there in the world of nature. (P. 89)

Such statements lie at the heart of the epistemological debate that began in the 1920s (see Reiss 1968:10–11) and continues into the modern era. Despite the vastness of sociological inquiry, it is obvious that a strong orientation toward the scientific study of human behavior, social interaction, and organizations continues and that this scientific focus is predicated on the assumption that such study is possible because it is based on the examination of phenomena that are subject to the operation of universal laws, a point not lost in the minds of the discipline’s founders. The counterpoint that the social sciences are cultural sciences and thereby fundamentally different from the physical sciences and also subject to different methodology and other evaluative criteria is representative of a longstanding European influence that also began in the 1920s.

Given the diversity and fluidity of the topics addressed and the levels of theories employed by sociologists, it is not surprising that many others do not agree. The counterargument is based on the premise that given the circumstances behind the evolution of science and the support it received in the past and the more repressive attention it receives in the contemporary experience from powerful interest groups, objective social science and the establishment of universal laws that are based on such inquiry may not be possible (see Turner 2001).

Whether or not one argues that the study of human society is unique, it is still extraordinary given the vast array of extant theories used to express the human experience and capacity. Witness the statement of one contemporary analyst who, in an intriguing assessment of the contemporary American “wilding” experience, wrote,

Sociology arose as an inquiry into the dangers of modern individualism, which could potentially kill society itself. The prospect of the death of society gave birth to the question . . . what makes society possible and prevents it from disintegrating into a mass of sociopathic and self-interested isolates? This core question of sociology has become the vital issue of our times. (Charles Derber 2003:18)

Only in part is Derber referring to the American experience. His assessment also speaks to the experience of Western Europe. Much social change has taken place, and the efforts of sociologists to describe and explain this change and to draw upon these insights to develop predictive models has led to a diversity of theories. Indeed, over time, the scientific paradigm shifts more generally described by Thomas Kuhn ([1962] 1970) are obvious in our discipline (see Friedrichs 1970). There have been, there are at present, and there undoubtedly will be future paradigm shifts within this evolving and apparently expanding discipline of sociology, many of which will focus, as has been the case in the past, on the social change process. And for all the so-called objectivity of a scientific sociology advocated by analysts such as George A. Lundberg (1947), the development of which is so eloquently described by Leon Bramson (1961)), sociologists have been involved in social activism and social engineering, that first occurred during the embryonic years of the discipline’s development (Volkart 1968). Such activism occurred again during the 1960s and 1970s, in many social justice areas, and in occupational settings such as those of the criminal justice system.

At present, sociological inquiry represents a vast array of topics and offers many competing theoretical models while its practitioners attempt to make sense of a rapidly changing world. For all its middle-range theories and studies that reflect the efforts of those dedicated to cumulative knowledge, it is also important that we recognize that the building of a paradigm as well as challenges to an extant paradigm are not relegated to the gathering of information alone. Indeed, if sociology is to advantage itself in the twenty-first century, it may be imperative that a dominant paradigm begins to identify the kinds of community needs that it can usually serve, for as Joseph R. Gusfield (1990) so clearly notes, sociology has been at odds with and a critic of the classical economic and individualistic interpretations of American life. Thus, whatever issues sociology may need to address at this juncture, perhaps we are hampered only by the limits of the sociological imagination. Again, the following comment by Homans (1967) is noteworthy:

The difficulties of social science lie in explanation rather than discovery. . . . Our trouble has not been with making discoveries but with organizing them theoretically—showing how they follow under a variety of given conditions from a few general principles. (Pp. 79, 105)

The present diversity of the discipline welcomed by so many social critics also serves as a barrier to the creation of a dominant theoretical paradigm. Without this focus, sociology remains in the minds of many of the discipline’s representatives a less-than-coherent discipline. Perhaps this is not different from the struggle of the 1960s as described by Gouldner (1970), a period that also was far less than organized and coherent and certainly far less civil in disagreement. It is important that sociologists take stock of their trade and question in earnest the utility of the work we do. As noted by Herbert L. Gans (1990),

By and large, we sociologists have been too distant from the society in which we operate and in which we are embedded, which funds us even if too poorly and which influences us surely more than we influence it. We are too busy trying to understand how that society functions . . . that we rarely think about our own functions—and dysfunctions. To some extent our failure to do so stems from a typical professional blindness, which results in our inability to distance ourselves sufficiently from ourselves and our routines to look systematically at what we are for and to whom. (Pp. 12–13)

Not all may agree, of course. Indeed, sociology in the United States and in Europe has been a critique of modern urban life with its emphasis on the individual, capitalism, and bureaucracy. In some instances, this critique of American society has been radical and reformist in its thrust (Gusfield 1990:31–46). And although American sociology had been shaped in part by psychology in establishing its methodology during the first two-thirds of the twentieth century, especially through a common socialpsychological area (see, e.g., Reiss 1968), it can be safely stated that American sociology has been transformed during the latter decades of the twentieth century.

Sociologists may be accused of engaging in an affair with their work. Witness the stirring comments of one colleague:

I fell in love with sociology when I was twelve. . . . Sociology was my savior. It saved me from the vexing confusion caused by my once despising the mundaneness of everyday life and deeply loving and admiring my people. It stabilized me by articulating the dedication that I felt for social justice. (Shahidian 1999:303–04)

We share this passionate approach to social science based on the insightful development of theory and empirical research, an approach that has, in turn, led to a vast array of subject matter. In light of these impressive contributions, the only aspect of this endeavor that may seem perplexing to some is that as we move further into the twenty-first century, there are those who continue to believe in and practice the scientific method; there also are those who argue that if the logic of science and the methods of scientific objectivity are to be carried to an extreme, sociology will lose or has already lost its humanistic perspective and, with this loss, the inclination toward active community involvement through social policy advocacy and practical intervention. As Peter L. Berger (1963) phrases it,

At the same time it is quite true that some sociologists, especially in America, have become so preoccupied with methodological questions that they have ceased to be interested in society at all. As a result, they have found out nothing of significance about any aspect of social life, since in science as in love a concentration on technique is quite likely to lead to impotence. (P. 13)

This dichotomy certainly is a matter of considerable debate, but perhaps most advocates and active practitioners of the discipline would fall somewhere in between these two orientations (see, e.g., Reiss 1968:10–11). In this regard, we are also optimistic that the sociological imagination will continue to be an important part of the work of sociologists as they take into consideration “a quality of mind that will help them to use information and to develop reason in order to achieve lucid summations of what is going on in the world and of what may be happening within themselves” (Mills 1959:5).

More than 170 years ago, sociology began to emerge from its philosophical and biological roots to it current status as an important social science. Early sociologists achieved renown based on their interest in providing information useful to appraise social policy issues. However, in the contemporary instance, there are strong indicators that sociology has not achieved the eminent position envisioned by the founders. Note the less-than-enthusiastic assessment offered by Black (1999):

The problems endemic to the discipline of sociology include the lack of a paradigm, disciplinary fragmentation, and the irreconcilability of science, ideology, and politics . . . and the lack of an occupational niche—[all these] place sociologists in the position of having constantly to defend the profession. (Pp. 261, 263)

Thus, as we move well into the twenty-first century, it is clear that sociology is engaged in yet another struggle to (re)identify itself. Perhaps such a struggle is to be expected of any science of human behavior. And nowhere is this situation more contentious than in the responses of representatives of the discipline to the question as to whether sociology is or is not yet considered an activity worthy of the label “scientific activity.”

At the center of this struggle lies the heart of any discipline—namely, sociological theory. Among the eminent theorists reporting on the status of sociology in this Handbook are individuals who represent the very best of what the discipline has to offer. That the message is suggestive of a continuing debate within the discipline is both disheartening and encouraging. It is disheartening in that after a period of more than 175 years, representatives of the discipline should be able to exclaim with great pride the accomplishments of so much activity instead of debating their scientific worth. It is encouraging because the current debate over the theory and the substance of the work sociologists engage in can only lead to the exploration of new and challenging frontiers. But the substance of sociological inquiry also represents a matter of contention for many research- and practitioner-oriented representatives of the discipline. Some contemporary analysts who have observed the developments within the academy during the past several decades call for a critical reevaluation of that which sociologists identify as the substance of research and understanding. Sociology has given birth to and generated intense interest in many areas of study that are no longer identified with the discipline. Because the specific subareas developed by sociologists became well accepted as legitimate applied disciplines within the academy, independent, overlapping units within the academy have been created.

If the 1960s represent the golden era of sociology, it is also a period, as described by Turner and Sica (2006), that is “remembered as a time of violence, massive social change, and personal transformation” (p. 4). The period had a profound effect on an entire generation of students, many of whom were instrumental in creating the new sociological emphasis that today is criticized for its diversity, the lack of continuity, and a failure to develop a unified paradigm. Whatever reservations that may continue to exist as we progress well into the twenty-first century, these can be hailed as a challenge. Thus, at the same time that community involvement and applied research are increasingly being devalued in the academic world, there is a distinct pressure, according to Harris and Wise (1998), for sociologists to become increasingly involved in the community and society.

This call to establish a public sociology may well combine with the three types of knowledge identified by Burawoy (2005)—the professional, critical, and policyspecific databases. In each of these areas, the initiative would be consistent with enthusiastic proclamations of the past. George A. Lundberg’s (1947) Can Science Save Us? serves as but one important example of those who promoted the application of social science insights to solve social problems. Of course, one major difference between the time when Lundberg wrote and now is that we are not rebounding from the tragedy of a world war. Indeed, it was during the post-World War II period and during the subsequent several decades that American sociology assumed its theoretical and empirical dominance (Odum 1951), especially in the area of deviant behavior (see Touraine 1990). Yet another important difference between then and now, as Harris and Wise (1998) suggest, is that sociologists need to be perceived as problem solvers rather than as social critics, and similar to the pleas of Marion Talbot (1896) at the end of the nineteenth century, much of the sociological may necessarily become interdisciplinary in nature. This perspective is supported as a portion of a more scholarly editorial philosophy articulated by Wharton (2006:1–2). Most noteworthy for our purpose are points three and four:

(3) Be aware and reflective about the . . . broader contributions to scholarship, policy, and/or activism . . . ; (4) produce useful knowledge—not merely in the applied sense of solving problems, but knowledge that is useful as basic research that can help people better understand and transform the social world. (P. 1)

These same kinds of issues—social activism and public policy research—were recognized at the end of the nineteenth century as strengths of the new discipline.

Thus, there appears to be hopeful as well as worrisome aspects of sociology at the end of the twentieth century (Lewis 1999). But this kind of enthusiasm and concern appears to be periodic throughout the history of the discipline as sociologists attempt to both define and then redefine the parameters of what some argue is too extensive a range of topics to allow practitioners of the discipline to be definitively identified (Best 2003). Witness the statement attributed to one of the coeditors of this Handbook who, in the early 1980s, wrote the following:

Future prospects for sociology(ists) no doubt will depend upon our ability to identify and respond to community needs, to compete for funds available from nontraditional sources, to work in applied areas, and to establish creative problemsolving strategies. The challenge before us should generate a healthy response. (Peck 1982:319–20)

Since that time and in the wake of a declining influence of the social sciences, there has been a response as evidenced by the many new areas of inquiry, many interdisciplinary in nature, that currently curry attention from sociologists. Indeed, there does appear to be a fragmentation, but this so-called fragmentation is consistent with an assessment offered by Beck (1999), “Sociology today, as throughout its history, is not unified. . . . we have never been able to sustain . . . unanimity and consistency for very long. Thank goodness” (p. 121).

Perhaps we do not engage in “normal science,” at least not in the sense that Thomas Kuhn ([1962] 1970) refers to it. That is, academic sociologists continue to function quite well even though they are outside the single frame of reference that usually serves as the paradigmatic foundation for the physical sciences. Normal science is rigid, but it is also burdened by uncertainty and inconsistency, as Friedrichs (1970) observes. In the case of sociology, this is found in the diversity of theoretical models and topical areas. Although some analysts lament the current state of the discipline, Jacobs (2004) recently observed that “some might view this diversity [of topics] as evidence of excessive fragmentation, (but) there are important theoretical connections” (p. v). Of course, the substance of manuscripts submitted for possible publication, the rubrics under which the research can be categorized, is quite different from the search for a common sociological paradigm. To wit, classic studies do exist, but none serve to forge a single paradigm. Thus, the future of the discipline will depend, as usual, on the contributions of those who may be relatively silent in the wake of less-than-acceptable “scholarship,” as suggested by Lewis (1999), but who nonetheless commit themselves to excellence by producing significant contributions to theory and application (see, e.g., Rossi 1999) that should, in the long run, counter the myriad productions that are less significant. Concomitant with this effort will be an increased awareness of and involvement in the applied and an earnest effort to again be a viable force in the policy-related aspects of sociology and society. In other words, we believe there will be a reawakening of and involvement in those aspects of sociology that served the discipline well during its early years of development in the United States (see Ross 1936) even as the applied social work-oriented practitioners broke away to form their own professional association (Odum 1951; Rossi 1999). Indeed, there exists a need for answers to myriad policy-oriented questions as well as applied concerns at all governmental levels.

But in the end, sociologists may, as Beck (1999:123) suggests, go where they go, where they want to go. This may again mean that sociologists will abandon important areas of inquiry that they helped to establish, leaving the sociological legacy to others. Sociologists will also move to create other areas of inquiry while questioning past and present assumptions and knowledge claims in an ongoing quest to better understand social arrangements and to engage in, as Beck (1999) observes, “life, liberty, the pursuit of happiness, and the sociological imagination” (p. 124). To this we can add the quest to establish the meaning of social justice in a rapidly changing democratic society.

Thus, contrary to dubious predictions of an ominous obscure future, the content of this Handbook attests to a much more positive and grand future orientation within the discipline that will include much more than the rigorous efforts to clean up conceptual problems that sociologists are supposedly noted for. Moreover, the epistemological debates of the past will undoubtedly continue as Turner (2001) and Best (2003) suggest, but in so doing, the future of academic sociology will again be broadened. This expansion will again, we think, involve the applied aspects of the discipline and engagement of the public through active involvement of sociologists in the four traditional areas—namely, through a public sociology with an emphasis on further development of the profession and a critical civic activism with the intent to broadly influence social policy. Moreover, the increasing influence of European sociology in the global community will undoubtedly continue; this influence is not only important, it is most welcome. Given the above, it may well be that another call to arms will result. There has been a movement, albeit a small movement, among highly regarded intellectuals (the National Association of Scholars) to enhance the substance and quality of academic teaching and scholarly activity. This, too, is welcome in sociology.

The world that engages a scientist, as noted by Friedrichs (1970), is one that emerges from a scientific tradition, along with its special vocabulary and grammar and environment. Sociology’s laboratory is the social world and on occasion its practitioners are criticized by those who argue the arcane nature of all that is considered scientific. If the normal science, as described by Thomas Kuhn ([1962] 1970) and Robert W. Friedrichs (1970), is to be realized within the discipline of sociology, then it may depend on efforts of young sociologists (see, e.g., Frickel and Gross 2005) who may capture the essence of such a paradigm in a general theory of scientific/intellectual movements. Such work may also serve to stimulate more thought as to the requisite initiatives essential for subsequently developing the kind of intellectual movement that will define once again, and actively promote, the substance of the sociological perspective.

If the emphasis of American sociology at the beginning of the twentieth century was unsophisticated, armchair science that “featured the study of general society and the ‘system’ of social theory, it reflected not only the almost universal philosophical approach but also the consistency of the best minds in interaction with European philosophy and American higher education” (Odum 1951:421–22). In the mid-twentieth century, sociology, similar to other social and physical sciences, struggled to determine whether the future of the discipline would continue to pursue a general systems theory of society or whether the discipline’s practitioners would develop more theory and then relate these theories to research and the scientific method (Odum 1951:422). At this critical midpoint of the century past, and in recognition of the importance of the discipline, Odum (1951) wrote that there is

the extraordinary need in the contemporary world for a social science to seek special knowledge of human society and welfare and meet the crises brought on by science and technology, so often out of perspective to human relations, and so to provide the basis for not only a social morale in an age of science but for societal survival as well. (P. 3)

At the end of the twentieth century, these comments rang clear, and as we move forward and well into the greater twenty-first-century experience, Odum’s words seem no less germane today than in the past.

Toward establishing the prospects for the future of this great academic discipline, we hasten to add how critical it is and will be to again acknowledge the important work of the founding mothers and fathers of sociology. Thus, at the end of the twentieth century, the state of sociology may have been debatable, but during the initial decades of the twenty-first century, sociologists will undoubtedly take up the challenge to pursue answers to vexing social problems that are, as Fine (2006:14–15) states, embedded with complex, dynamic, interconnected social systems. Some of the solutions to be tendered in the near future may not serve well the needs of all citizens, but these should nonetheless address policy issues relating to social freedom, social justice, and social equality while recognizing that such policies determine the behavior of those actors whom sociologists are intent to study. Herein American sociologists may now have achieved the requisite disciplinary maturity to employ the kind of sociological imagination envisioned by C. Wright Mills (1959) half a century ago. Such a sociology would, in the tradition of Europe, encompass a biography and history within society, thereby allowing sociology to represent not only a scientific enterprise but also to serve as a sensitizing discipline that allows us to continue to view the world in a new and interpretive fashion.

Finally, in some peculiar ways, the vexing problems that capture our attention during the early portion of the twenty-first century parallel those of the early twentieth century; this is true at all levels of society and perhaps even more so within those sectors that heretofore were barricaded from a critical analyses. The actors may have changed but, in general, the public concerns regarding the kinds of behavior tolerated and considered to be appropriate tend to remain the same. And as the moral entrepreneurs of the twenty-first century push their agendas, the new prohibitionist movements continue to capture the attention of policymakers, which may of necessity be cause for some sociologists at least to revisit many of the same topics that held sway in the past. Thus, we will continue to use templates in our lives to understand the world, physical and social, in which we exist. The sociological templates derived from the many conceptual constructs available provide us with a unique and perceptive perspective. As sociology further develops, new conceptual constructs will be added and will contribute to its unique perspective, thereby enhancing our ability to better analyze and understand human social behavior.

Bibliography:

• Beck, Bernard. 1999. “The Future of Sociology.” Sociological Inquiry 69:121–29.
• Berger, Peter L. 1963. Invitation to Sociology: A Humanistic Perspective. Garden City, NY: Anchor Books.
• Bernard, L. L. and Jessie Bernard. 1943. Origins of American Sociology. New York: Thomas Y. Crowell.
• Best, Joel. 2001. Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists. Berkeley: University of California Press.
• Best, Joel. 2003. “Killing the Messenger: The Social Problems of Sociology.” Social Problems 50:1–13.
• Black, Timothy. 1999. “Going Public: How Sociology Might Matter Again.” Special Section: Saving Sociology (Part II) Sociological Inquiry 690:257–75.
• Bramson, Leon. 1961. The Political Context of Sociology. Princeton, NJ: Princeton University Press.
• Burawoy, Michael. 2005. “For Public Sociology.” American Sociological Review 70:4–28.
• Calhoun, Arthur W. 1919. A Social History of the American Family: From Colonial Times to the Present. 3. Cleveland, OH: Arthur H. Clark.
• Crothers, Charles. 2001. “History of Sociology.” Pp. 1550–52 in Reader’s Guide to the Social Sciences, edited by J. Michie. Chicago, IL: Fitzroy Dearborn.
• Derber, Charles. 2003. The Wilding of America: Money, Mayhem, and the New American Dream. New York: Worth.
• Eisenstadt, Shmuel N. 1968. “The Development of Sociological Thought.” Pp. 23–36 in International Encyclopedia of the Social Sciences, edited by D. L. Sills. New York: Macmillan/Free Press.
• Fine, Gary Alan. 2006. “The Chaining of Social Problems: Solution, and Unintended Consequences in the Age of Betrayal.” Social Problems 53:3–17.
• Frickel, Scott and Neil Gross. 2005. “A General Theory of Scientific/Intellectual Movements.” American Journal of Sociology 70:204–32.
• Friedrichs, Robert W. 1970. A Sociology of Sociology. New York: Free Press.
• Gans, Herbert J., ed. 1990. Sociology in America. Newbury Park, CA: Sage.
• Goudsblom, J. and J. Heilbron. 2001. “History of Sociology.” Pp. 14574–80 in International Encyclopedia of the Social & Behavioral Sciences, edited by N. J. Smelser and P. B. Baltes. Amsterdam: Elsevier.
• Gouldner, Alvin W. 1970. The Coming Crisis in Western Sociology. New York: Basic Books.
• Gusfield, Joseph R. 1990. “Sociology’s Effects on Society.” Pp. 31–46 in Sociology in America, edited by H. J. Gans. Newbury Park, CA: Sage.
• Harris, Catherine and Michael Wise. 1998. “Grassroots Sociology and the Future of the Discipline.” American Sociologist Winter:29–47.
• Hollander, Paul. 1999. “Saving Sociology?” Sociological Inquiry 69:130–47.
• Homans, George C. 1967. The Nature of Social Science. New York: Harcourt, Brace & World.
• Jacobs, Jerry A. 2004. “ASR: Yesterday, Today, Tomorrow.” American Sociological Review 69:v–vi.
• Kuhn, Thomas S. [1962] 1970. The Structure of Scientific Revolutions. 2d ed. Chicago, IL: University of Chicago Press.
• Kurtz, Lester R. 1986. Evaluating Chicago Sociology. Chicago, IL: University of Chicago Press.
• Lecuyer, Bernard and Anthony R. Oberschall. 1968. “The Early History of Social Research.” Pp. 36–52 in International Encyclopedia of the Social Sciences, edited by D. L. Sills. New York: Macmillan/Free Press.
• Lee, Alfred McClung. 1978. Sociology for Whom? New York: Oxford University Press.
• Lengermann, Patricia Madoo and Jill Niebrugge-Brantley. 1998. The Women Founders: Sociology and Social Theory, 1830–1930. Boston, MA: McGraw-Hill.
• Lewis, Michael. 1999. “Introduction: Saving Sociology (Part I).” Sociological Inquiry 69:106–109.
• Lundberg, George A. 1947. Can Science Save Us? New York: David McKay.
• MacIver, R. M. 1934. “Sociology.” Pp. 232–46 in Encyclopedia of the Social Sciences, edited by E. R. A. Seligman and A. Johnson. New York: Macmillan.
• Madge, John. 1962. The Origins of Scientific Sociology. New York: Free Press.
• Martineau, Harriet. 1836/1837 . Society in America. London, England: Saunders & Otley .
• Martineau, Harriet. 1838. How to Observe Morals and Manners. London, England: Charles Knight.
• Merton, Robert K. [1957] 1968. Social Theory and Social Structure. New York: Free Press.
• Mills, C. Wright. 1959. The Sociological Imagination. New York: Oxford University Press.
• Myrdal, Gunnar. 1969. Objectivity in Social Research. New York: Pantheon Books.
• Nisbet, Robert A. 1966. The Sociological Tradition. New York: Basic Books.
• Noble, David F. 1999. The Religion of Technology. New York: Penguin Putnam.
• Odum, Howard W. 1951. American Sociology: The Story of Sociology in the United States through 1950. New York: Longmans, Green.
• Odum, Howard W. [1927] 1965. American Masters of Social Science. Port Washington, NY: Kennikat Press.
• O’Neill, William L. 1967. Divorce in the Progressive Era. New Haven, CT: Yale University Press.
• Peck, Dennis L. 1982. “Future Prospects for Sociology(ists): Observations of a Non-Rural Sociologist.” The Rural Sociologist 2:315–21 .
• Reiss, Albert J. 1968. “Sociology: The Field.” Pp. 1–22 in International Encyclopedia of the Social Sciences, edited by D. L. Sills. New York: Macmillan/Free Press.
• Ross, E. A. 1936. “Some Contributions of Sociology to the Guidance of Society.” American Sociological Review 1:29–32.
• Rossi, Peter H. 1999. “Saving Academic Sociology.” Sociological Inquiry 69:110–20.
• Shahidian, Hammed. 1999. “Saving the Savior.” Sociological Inquiry 69:303–27.
• Small, A. W. 1895. “The Era of Sociology.” American Journal of Sociology 1:1–15.
• Smelser, Neil J. 1990. “External Influences on Sociology.” Pp. 49–60 in Sociology in America, edited by H. J. Gans. Newbury Park, CA: Sage.
• Talbot, Marion. 1896. “Sanitation and Sociology.” American Journal of Sociology 2:74–81.
• Touraine, Alain. 1990. “American Sociology Viewed from Abroad.” Pp. 239–52 in Sociology in America, edited by H. J. Gans. Newbury Park, CA: Sage.
• Turner, Jonathan H. 2001. “Sociology, Survey.” Pp. 1538–50 in Reader’s Guide to the Social Sciences, edited by J. Michie. Chicago, IL: Fitzroy Dearborn.
• Turner, Stephen and Alan Sica. 2006. “Two Scholars Examine Golden Decade’s Imprint on Today’s Sociologists.” Footnotes 34(2):4.
• Volkart, E. H. 1968. “Thomas, W. I.” Pp. 1–6 in International Encyclopedia of the Social Sciences, 16, edited by D. L. Sills. New York: Macmillan/Free Press.
• Ward, Lester A. 1896. “Contributions to Social Philosophy. IX. The Purpose of Sociology.” American Journal of Sociology 2:446–60.
• Wharton, Amy S. 2006. “Letter from the Editor.” Social Problems 59:1–2.
• Whitt, Hugh P. 2001. “The Moral Statisticians” Pp. 229–35 in Encyclopedia of Criminology and Deviant Behavior, Historical, Conceptual, and Theoretical Issues, 1, edited by P. A. Adler, P. Adler, and J. Corzine. Philadelphia, PA: Brunner-Routledge.
• Wright, Carroll D. 1895. “Contributions of the United States Government to Social Science.” American Journal of Sociology 1(3):241–75.
• Wrong, Dennis H. 1990. “Sociology’s Effects on America.” Pp. 19–30 in Sociology in America, edited by H. J. Gans. Newbury Park, CA: Sage.

ORDER HIGH QUALITY CUSTOM PAPER

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 19 June 2024

Detecting hallucinations in large language models using semantic entropy

• Sebastian Farquhar   ORCID: orcid.org/0000-0002-9185-6415 1   na1 ,
• Jannik Kossen 1   na1 ,
• Lorenz Kuhn 1   na1 &
• Yarin Gal   ORCID: orcid.org/0000-0002-2733-2078 1  

Nature volume  630 ,  pages 625–630 ( 2024 ) Cite this article

70k Accesses

1 Citations

1458 Altmetric

Metrics details

• Computer science
• Information technology

Large language model (LLM) systems, such as ChatGPT 1 or Gemini 2 , can show impressive reasoning and question-answering capabilities but often ‘hallucinate’ false outputs and unsubstantiated answers 3 , 4 . Answering unreliably or without the necessary information prevents adoption in diverse fields, with problems including fabrication of legal precedents 5 or untrue facts in news articles 6 and even posing a risk to human life in medical domains such as radiology 7 . Encouraging truthfulness through supervision or reinforcement has been only partially successful 8 . Researchers need a general method for detecting hallucinations in LLMs that works even with new and unseen questions to which humans might not know the answer. Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations. Our method addresses the fact that one idea can be expressed in many ways by computing uncertainty at the level of meaning rather than specific sequences of words. Our method works across datasets and tasks without a priori knowledge of the task, requires no task-specific data and robustly generalizes to new tasks not seen before. By detecting when a prompt is likely to produce a confabulation, our method helps users understand when they must take extra care with LLMs and opens up new possibilities for using LLMs that are otherwise prevented by their unreliability.

Similar content being viewed by others

Testing theory of mind in large language models and humans

Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT

ThoughtSource: A central hub for large language model reasoning data

‘Hallucinations’ are a critical problem 9 for natural language generation systems using large language models (LLMs), such as ChatGPT 1 or Gemini 2 , because users cannot trust that any given output is correct.

Hallucinations are often defined as LLMs generating “content that is nonsensical or unfaithful to the provided source content” 9 , 10 , 11 but they have come to include a vast array of failures of faithfulness and factuality. We focus on a subset of hallucinations which we call ‘confabulations’ 12 for which LLMs fluently make claims that are both wrong and arbitrary—by which we mean that the answer is sensitive to irrelevant details such as random seed. For example, when asked a medical question “What is the target of Sotorasib?” an LLM confabulates by sometimes answering KRASG12 ‘C’ (correct) and other times KRASG12 ‘D’ (incorrect) despite identical instructions. We distinguish this from cases in which a similar ‘symptom’ is caused by the following different mechanisms: when LLMs are consistently wrong as a result of being trained on erroneous data such as common misconceptions 13 ; when the LLM ‘lies’ in pursuit of a reward 14 ; or systematic failures of reasoning or generalization. We believe that combining these distinct mechanisms in the broad category hallucination is unhelpful. Our method makes progress on a portion of the problem of providing scalable oversight 15 by detecting confabulations that people might otherwise find plausible. However, it does not guarantee factuality because it does not help when LLM outputs are systematically bad. Nevertheless, we significantly improve question-answering accuracy for state-of-the-art LLMs, revealing that confabulations are a great source of error at present.

We show how to detect confabulations by developing a quantitative measure of when an input is likely to cause an LLM to generate arbitrary and ungrounded answers. Detecting confabulations allows systems built on LLMs to avoid answering questions likely to cause confabulations, to make users aware of the unreliability of answers to a question or to supplement the LLM with more grounded search or retrieval. This is essential for the critical emerging field of free-form generation in which naive approaches, suited to closed vocabulary and multiple choice, fail. Past work on uncertainty for LLMs has focused on simpler settings, such as classifiers 16 , 17 and regressors 18 , 19 , whereas the most exciting applications of LLMs relate to free-form generations.

The term hallucination in the context of machine learning originally comes from filling in ungrounded details, either as a deliberate strategy 20 or as a reliability problem 4 . The appropriateness of the metaphor has been questioned as promoting undue anthropomorphism 21 . Although we agree that metaphor must be used carefully with LLMs 22 , the widespread adoption of the term hallucination reflects the fact that it points to an important phenomenon. This work represents a step towards making that phenomenon more precise.

To detect confabulations, we use probabilistic tools to define and then measure the ‘semantic’ entropy of the generations of an LLM—an entropy that is computed over meanings of sentences. High entropy corresponds to high uncertainty 23 , 24 , 25 —so semantic entropy is one way to estimate semantic uncertainties. Semantic uncertainty, the broader category of measures we introduce, could be operationalized with other measures of uncertainty, such as mutual information, instead. Entropy in free-form generation is normally hard to measure because answers might mean the same thing (be semantically equivalent) despite being expressed differently (being syntactically or lexically distinct). This causes naive estimates of entropy or other lexical variation scores 26 to be misleadingly high when the same correct answer might be written in many ways without changing its meaning.

By contrast, our semantic entropy moves towards estimating the entropy of the distribution of meanings of free-form answers to questions, insofar as that is possible, rather than the distribution over the ‘tokens’ (words or word-pieces) which LLMs natively represent. This can be seen as a kind of semantic consistency check 27 for random seed variation. An overview of our approach is provided in Fig. 1 and a worked example in Supplementary Table 1 .

a , Naive entropy-based uncertainty measures variation in the exact answers, treating ‘Paris’, ‘It’s Paris’ and ‘France’s capital Paris’ as different. But this is unsuitable for language tasks for which sometimes different answers mean the same things. Our semantic entropy clusters answers which share meanings before computing the entropy. A low semantic entropy shows that the LLM is confident about the meaning. b , Semantic entropy can also detect confabulations in longer passages. We automatically decompose a long generated answer into factoids. For each factoid, an LLM generates questions to which that factoid might have been the answer. The original LLM then samples  M possible answers to these questions. Finally, we compute the semantic entropy over the answers to each specific question, including the original factoid. Confabulations are indicated by high average semantic entropy for questions associated with that factoid. Here, semantic entropy classifies Fact 1 as probably not a confabulation because generations often mean the same thing, despite very different wordings, which a naive entropy would have missed.

Intuitively, our method works by sampling several possible answers to each question and clustering them algorithmically into answers that have similar meanings, which we determine on the basis of whether answers in the same cluster entail each other bidirectionally 28 . That is, if sentence A entails that sentence B is true and vice versa, then we consider them to be in the same semantic cluster. We measure entailment using both general-purpose LLMs and natural language inference (NLI) tools developed specifically for detecting entailment for which we show direct evaluations in Supplementary Tables 2 and 3 and Supplementary Fig. 1 . Textual entailment has previously been shown to correlate with faithfulness 10 in the context of factual consistency 29 as well as being used to measure factuality in abstractive summarization 30 , especially when applied at the right granularity 31 .

Semantic entropy detects confabulations in free-form text generation across a range of language models and domains, without previous domain knowledge. Our evaluations cover question answering in trivia knowledge (TriviaQA 32 ), general knowledge (SQuAD 1.1; ref. 33 ), life sciences (BioASQ 34 ) and open-domain natural questions (NQ-Open 35 ) derived from actual queries to Google Search 36 . In addition, semantic entropy detects confabulations in mathematical word problems (SVAMP 37 ) and in a biography-generation dataset, FactualBio, accompanying this paper.

Our results for TriviaQA, SQuAD, BioASQ, NQ-Open and SVAMP are all evaluated context-free and involve sentence-length answers (96 ± 70 characters, mean ± s.d.) and use LLaMA 2 Chat (7B, 13B and 70B parameters) 38 , Falcon Instruct (7B and 40B) 39 and Mistral Instruct (7B) 40 . In the Supplementary Information , we further consider short-phrase-length answers. Results for FactualBio (442 ± 122 characters) use GPT-4 (ref. 1 ). At the time of writing, GPT-4 (ref. 1 ) did not expose output probabilities 41 or hidden states, although it does now. As a result, we propose a discrete approximation of our estimator for semantic entropy which allows us to run experiments without access to output probabilities, which we use for all GPT-4 results in this paper and which performs similarly well.

Our confabulation detection with semantic entropy is more robust to user inputs from previously unseen domains than methods which aim to ‘learn’ how to detect confabulations from a set of example demonstrations. Our method is unsupervised, meaning that we do not need labelled examples of confabulations. By contrast, supervised methods detect confabulations by learning patterns behind examples of confabulations, assuming that future questions preserve these patterns. But this assumption is often untrue in new situations or with confabulations that human overseers are unable to identify (compare Fig. 17 of ref. 24 ). As a strong supervised baseline, we compare to an embedding regression method inspired by ref. 24 which trains a logistic regression classifier to predict whether the model correctly answered a question on the basis of the final ‘embedding’ (hidden state) of the LLM. We also use the P (True) method 24 which looks at the probability with which an LLM predicts that the next token is ‘True’ when few-shot prompted to compare a main answer with ‘brainstormed’ alternatives.

Confabulations contribute substantially to incorrect answers given by language models. We show that semantic entropy can be used to predict many incorrect model answers and to improve question-answering accuracy by refusing to answer those questions the model is uncertain about. Corresponding to these two uses, we evaluate two main metrics. First, the widely used area under the receiver operating characteristic (AUROC) curve for the binary event that a given answer is incorrect. This measure captures both precision and recall and ranges from 0 to 1, with 1 representing a perfect classifier and 0.5 representing an un-informative classifier. We also show a new measure, the area under the ‘rejection accuracy’ curve (AURAC). This studies the case in which the confabulation detection score is used to refuse to answer the questions judged most likely to cause confabulations. Rejection accuracy is the accuracy of the answers of the model on the remaining questions and the area under this curve is a summary statistic over many thresholds (representative threshold accuracies are provided in Supplementary Material ). The AURAC captures the accuracy improvement which users would experience if semantic entropy was used to filter out questions causing the highest entropy.

Detecting confabulations in QA and math

In Fig. 2 , we show that both semantic entropy and its discrete approximation outperform our best baselines for sentence-length generations. These results are averaged across datasets and provide the actual scores on the held-out evaluation dataset. We report the raw average score across held-out evaluation datasets without standard error because the distributional characteristics are more a property of the models and datasets selected than the method. Consistency of relative results across different datasets is a stronger indicator of variation in this case.

Semantic entropy outperforms leading baselines and naive entropy. AUROC (scored on the y -axes) measures how well methods predict LLM mistakes, which correlate with confabulations. AURAC (likewise scored on the y -axes) measures the performance improvement of a system that refuses to answer questions which are judged likely to cause confabulations. Results are an average over five datasets, with individual metrics provided in the Supplementary Information .

Semantic entropy greatly outperforms the naive estimation of uncertainty using entropy: computing the entropy of the length-normalized joint probability of the token sequences. Naive entropy estimation ignores the fact that token probabilities also express the uncertainty of the model over phrasings that do not change the meaning of an output.

Our methods also outperform the supervised embedding regression method both in- and out-of-distribution. In pale-yellow bars we show that embedding regression performance deteriorates when its training data do not match the deployment distribution—which mirrors the common real-world case in which there is a distribution shift between training and deployment 42 —the plotted value is the average metric for embedding regression trained on one of the four ‘off-distribution’ datasets for that evaluation. This is critical because reliable uncertainty is most important when the data distribution shifts. Semantic entropy also outperforms P (True) which is supervised ‘in-context’; that is, it is adapted to the deployment task with a few training examples provided in the LLM prompt itself. The discrete variant of semantic entropy performs similarly to our standard estimator, despite not requiring exact output probabilities.

Averaged across the 30 combinations of tasks and models we study, semantic entropy achieves the best AUROC value of 0.790 whereas naive entropy (0.691), P (True) (0.698) and the embedding regression baseline (0.687) lag behind it. Semantic entropy performs well consistently, with stable performance (between 0.78 and 0.81 AUROC) across the different model families (LLaMA, Falcon and Mistral) and scales (from 7B to 70B parameters) which we study (we report summary statistics for each dataset and model as before). Although semantic entropy outperforms the baselines across all model sizes, P (True) seems to improve with model size, suggesting that it might become more competitive for very capable honest models in settings that the model understands well (which are, however, not the most important cases to have good uncertainty). We use ten generations to compute entropy, selected using analysis in Supplementary Fig. 2 . Further results for short-phrase generations are described in Supplementary Figs. 7 – 10 .

The results in Fig. 2 offer a lower bound on the effectiveness of semantic entropy at detecting confabulations. These evaluations determine whether semantic entropy and baseline methods can detect when the answers of the model are incorrect (which we validate against human correctness evaluations in Supplementary Table 4 ). In addition to errors from confabulations (arbitrary incorrectness), this also includes other types of mistakes for which semantic entropy is not suited, such as consistent errors learned from the training data. The fact that methods such as embedding regression are able to spot other kinds of errors, not just confabulations, but still are outperformed by semantic entropy, suggests that confabulations are a principal category of errors for actual generations.

Examples of questions and answers from TriviaQA, SQuAD and BioASQ, for LLaMA 2 Chat 70B, are shown in Table 1 . These illustrate how only semantic entropy detects when the meaning is constant but the form varies (the first row of the table) whereas semantic entropy and naive entropy both correctly predict the presence of confabulations when the form and meaning vary together (second row) and predict the absence of confabulations when the form and meaning are both constant across several resampled generations (third row). In the final row, we give an example in which semantic entropy is erroneously high as a result of overly sensitive semantic clustering relative to the reference answer. Our clustering method distinguishes the answers which provide a precise date from those which only provide a year. For some contexts that would have been correct but in this context the distinction between the specific day and the year is probably irrelevant. This highlights the importance of context and judgement in clustering, especially in subtle cases, as well as the shortcomings of evaluating against fixed reference answers which do not capture the open-ended flexibility of conversational deployments of LLMs.

Detecting confabulations in biographies

Semantic entropy is most natural for sentences that express a single proposition but the idea of semantic equivalence is trickier to apply to longer passages which express many propositions which might only agree partially 43 . Nevertheless, we can use semantic entropy to detect confabulations in longer generations, such as entire paragraphs of text. To show this, we develop a dataset of biographical generations from GPT-4 (v.0613) for 21 individuals notable enough to have their own Wikipedia page but without extensive online biographies. From each biography generated by GPT-4, we automatically extract propositional factual claims about the individual (150 factual claims in total), which we manually label as true or false.

Applying semantic entropy to this problem is challenging. Naively, one might simply regenerate each sentence (conditioned on the text so far) and then compute semantic entropy over these regenerations. However, the resampled sentences often target different aspects of the biography: for example, one time describing family and the next time profession. This is analogous to the original problem semantic entropy was designed to resolve: the model is uncertain about the right ordering of facts, not about the facts themselves. To address this, we break down the entire paragraph into factual claims and reconstruct questions which might have been answered by those claims. Only then do we apply semantic entropy (Fig. 1 ) by generating three new answers to each question (selected with analysis in Supplementary Figs. 3 and 4 ) and computing the semantic entropy over those generations plus the original factual claim. We aggregate these by averaging the semantic entropy over all the questions to get an uncertainty score for each proposition, which we use to detect confabulations. Unaggregated results are shown in Supplementary Figs. 5 and 6 .

As GPT-4 did not allow access to the probability of the generation at the time of writing, we use a discrete variant of semantic entropy which makes the further approximation that we can infer a discrete empirical distribution over semantic meaning clusters from only the generations ( Methods ). This allows us to compute semantic entropy using only the black-box outputs of an LLM. However, we were unable to compute the naive entropy baseline, the standard semantic entropy estimator or the embedding regression baseline for GPT-4 without output probabilities and embeddings.

In Fig. 3 we show that the discrete variant of semantic entropy effectively detects confabulations on this dataset. Its AUROC and AURAC are higher than either a simple ‘self-check’ baseline—which just asks the LLM whether the factoid is likely to be true—or a variant of P (True) which has been adapted to work for the paragraph-length setting. Discrete semantic entropy has better rejection accuracy performance until 20% of the questions have been rejected at which point P (True) has a narrow edge. This indicates that the questions predicted to cause confabulations are indeed more likely to be wrong.

The discrete variant of our semantic entropy estimator outperforms baselines both when measured by AUROC and AURAC metrics (scored on the y -axis). The AUROC and AURAC are substantially higher than for both baselines. At above 80% of questions being answered, semantic entropy has the highest accuracy. Only when the top 20% of answers judged most likely to be confabulations are rejected does the answer accuracy on the remainder for the P (True) baseline exceed semantic entropy.

Our probabilistic approach, accounting for semantic equivalence, detects an important class of hallucinations: those that are caused by a lack of LLM knowledge. These are a substantial portion of the failures at present and will continue even as models grow in capabilities because situations and cases that humans cannot reliably supervise will persist. Confabulations are a particularly noteworthy failure mode for question answering but appear in other domains too. Semantic entropy needs no previous domain knowledge and we expect that algorithmic adaptations to other problems will allow similar advances in, for example, abstractive summarization. In addition, extensions to alternative input variations such as rephrasing or counterfactual scenarios would allow a similar method to act as a form of cross-examination 44 for scalable oversight through debate 45 .

The success of semantic entropy at detecting errors suggests that LLMs are even better at “knowing what they don’t know” than was argued by ref. 24 —they just don’t know they know what they don’t know. Our method explicitly does not directly address situations in which LLMs are confidently wrong because they have been trained with objectives that systematically produce dangerous behaviour, cause systematic reasoning errors or are systematically misleading the user. We believe that these represent different underlying mechanisms—despite similar ‘symptoms’—and need to be handled separately.

One exciting aspect of our approach is the way it makes use of classical probabilistic machine learning methods and adapts them to the unique properties of modern LLMs and free-form language generation. We hope to inspire a fruitful exchange of well-studied methods and emerging new problems by highlighting the importance of meaning when addressing language-based machine learning problems.

Semantic entropy as a strategy for overcoming confabulation builds on probabilistic tools for uncertainty estimation. It can be applied directly to any LLM or similar foundation model without requiring any modifications to the architecture. Our ‘discrete’ variant of semantic uncertainty can be applied even when the predicted probabilities for the generations are not available, for example, because access to the internals of the model is limited.

In this section we introduce background on probabilistic methods and uncertainty in machine learning, discuss how it applies to language models and then discuss our contribution, semantic entropy, in detail.

Uncertainty and machine learning

We aim to detect confabulations in LLMs, using the principle that the model will be uncertain about generations for which its output is going to be arbitrary.

One measure of uncertainty is the predictive entropy of the output distribution, which measures the information one has about the output given the input 25 . The predictive entropy (PE) for an input sentence x is the conditional entropy ( H ) of the output random variable Y with realization y given x ,

A low predictive entropy indicates an output distribution which is heavily concentrated whereas a high predictive entropy indicates that many possible outputs are similarly likely.

Aleatoric and epistemic uncertainty

We do not distinguish between aleatoric and epistemic uncertainty in our analysis. Researchers sometimes separate aleatoric uncertainty (uncertainty in the underlying data distribution) from epistemic uncertainty (caused by having only limited information) 46 . Further advances in uncertainty estimation which separate these kinds of uncertainty would enhance the potential for our semantic uncertainty approach by allowing extensions beyond entropy.

Joint probabilities of sequences of tokens

Generative LLMs produce strings of text by selecting tokens in sequence. Each token is a wordpiece that often represents three or four characters (though especially common sequences and important words such as numbers typically get their own token). To compute entropies, we need access to the probabilities the LLM assigns to the generated sequence of tokens. The probability of the entire sequence, s , conditioned on the context, x , is the product of the conditional probabilities of new tokens given past tokens, whose resulting log-probability is \(\log P({\bf{s}}| {\boldsymbol{x}})={\sum }_{i}\log P({s}_{i}| {{\bf{s}}}_{ < i},{\boldsymbol{x}})\) , where s i is the i th output token and s < i denotes the set of previous tokens.

Length normalization

When comparing the log-probabilities of generated sequences, we use ‘length normalization’, that is, we use an arithmetic mean log-probability, \(\frac{1}{N}{\sum }_{i}^{N}\log P({s}_{i}| {{\bf{s}}}_{ < i},{\boldsymbol{x}})\) , instead of the sum. In expectation, longer sequences have lower joint likelihoods because of the conditional independence of the token probabilities 47 . The joint likelihood of a sequence of length N shrinks exponentially in N . Its negative log-probability therefore grows linearly in N , so longer sentences tend to contribute more to entropy. We therefore interpret length-normalizing the log-probabilities when estimating the entropy as asserting that the expected uncertainty of generations is independent of sentence length. Length normalization has some empirical success 48 , including in our own preliminary experiments, but little theoretical justification in the literature.

Principles of semantic uncertainty

If we naively calculate the predictive entropy directly from the probabilities of the generated sequence of tokens, we conflate the uncertainty of the model over the meaning of its answer with the uncertainty over the exact tokens used to express that meaning. For example, even if the model is confident in the meaning of a generation, there are still usually many different ways for phrasing that generation without changing its meaning. For the purposes of detecting confabulations, the uncertainty of the LLM over meanings is more important than the uncertainty over the exact tokens used to express those meanings.

Our semantic uncertainty method therefore seeks to estimate only the uncertainty the LLM has over the meaning of its generation, not the choice of words. To do this, we introduce an algorithm that clusters model generations by meaning and subsequently calculates semantic uncertainty. At a high level this involves three steps:

Generation: sample output sequences of tokens from the predictive distribution of a LLM given a context x .

Clustering: cluster sequences by their meaning using our clustering algorithm based on bidirectional entailment.

Entropy estimation: estimate semantic entropy by summing probabilities of sequences that share a meaning following equation ( 2 ) and compute their entropy.

Generating a set of answers from the model

Given some context x as input to the LLM, we sample M sequences, { s (1) , …,  s ( M ) } and record their token probabilities, { P ( s (1) ∣ x ), …,  P ( s ( M ) ∣ x )}. We sample all our generations from a single model, varying only the random seed used for sampling from the token probabilities. We do not observe the method to be particularly sensitive to details of the sampling scheme. In our implementation, we sample at temperature 1 using nucleus sampling ( P  = 0.9) (ref. 49 ) and top- K sampling ( K  = 50) (ref. 50 ). We also sample a single generation at low temperature (0.1) as an estimate of the ‘best generation’ of the model to the context, which we use to assess the accuracy of the model. (A lower sampling temperature increases the probability of sampling the most likely tokens).

Clustering by semantic equivalence

To estimate semantic entropy we need to cluster generated outputs from the model into groups of outputs that mean the same thing as each other.

This can be described using ‘semantic equivalence’ which is the relation that holds between two sentences when they mean the same thing. We can formalize semantic equivalence mathematically. Let the space of tokens in a language be \({\mathcal{T}}\) . The space of all possible sequences of tokens of length N is then \({{\mathcal{S}}}_{N}\equiv {{\mathcal{T}}}^{N}\) . Note that N can be made arbitrarily large to accommodate whatever size of sentence one can imagine and one of the tokens can be a ‘padding’ token which occurs with certainty for each token after the end-of-sequence token. For some sentence \({\bf{s}}\in {{\mathcal{S}}}_{N}\) , composed of a sequence of tokens, \({s}_{i}\in {\mathcal{T}}\) , there is an associated meaning. Theories of meaning are contested 51 . However, for specific models and deployment contexts many considerations can be set aside. Care should be taken comparing very different models and contexts.

Let us introduce a semantic equivalence relation, E (  ⋅  ,  ⋅  ), which holds for any two sentences that mean the same thing—we will operationalize this presently. Recall that an equivalence relation is any reflexive, symmetric and transitive relation and that any equivalence relation on a set corresponds to a set of equivalence classes. Each semantic equivalence class captures outputs that can be considered to express the same meaning. That is, for the space of semantic equivalence classes \({\mathcal{C}}\) the sentences in the set \(c\in {\mathcal{C}}\) can be regarded in many settings as expressing a similar meaning such that \(\forall {\bf{s}},{{\bf{s}}}^{{\prime} }\in c:E({\bf{s}},{{\bf{s}}}^{{\prime} })\) . So we can build up these classes of semantically equivalent sentences by checking if new sentences share a meaning with any sentences we have already clustered and, if so, adding them into that class.

We operationalize E (  ⋅  ,  ⋅  ) using the idea of bidirectional entailment, which has a long history in linguistics 52 and natural language processing 28 , 53 , 54 . A sequence, s , means the same thing as a second sequence, s ′, only if the sequences entail (that is, logically imply) each other. For example, ‘The capital of France is Paris’ entails ‘Paris is the capital of France’ and vice versa because they mean the same thing. (See later for a discussion of soft equivalence and cases in which bidirectional entailment does not guarantee equivalent meanings).

Importantly, we require that the sequences mean the same thing with respect to the context—key meaning is sometimes contained in the context. For example, ‘Paris’ does not entail ‘The capital of France is Paris’ because ‘Paris’ is not a declarative sentence without context. But in the context of the question ‘What is the capital of France?’, the one-word answer does entail the longer answer.

Detecting entailment has been the object of study of a great deal of research in NLI 55 . We rely on language models to predict entailment, such as DeBERTa-Large-MNLI 56 , which has been trained to predict entailment, or general-purpose LLMs such as GPT-3.5 (ref. 57 ), which can predict entailment given suitable prompts.

We then cluster sentences according to whether they bidirectionally entail each other using the algorithm presented in Extended Data Fig. 1 . Note that, to check if a sequence should be added to an existing cluster, it is sufficient to check if the sequence bidirectionally entails any of the existing sequences in that cluster (we arbitrarily pick the first one), given the transitivity of semantic equivalence. If a sequence does not share meaning with any existing cluster, we assign it its own cluster.

Computing the semantic entropy

Having determined the classes of generated sequences that mean the same thing, we can estimate the likelihood that a sequence generated by the LLM belongs to a given class by computing the sum of the probabilities of all the possible sequences of tokens which can be considered to express the same meaning as

Formally, this treats the output as a random variable whose event-space is the space of all possible meaning-classes, C , a sub- σ -algebra of the standard event-space S . We can then estimate the semantic entropy (SE) as the entropy over the meaning-distribution,

There is a complication which prevents direct computation: we do not have access to every possible meaning-class c . Instead, we can only sample c from the sequence-generating distribution induced by the model. To handle this, we estimate the expectation in equation ( 3 ) using a Rao–Blackwellized Monte Carlo integration over the semantic equivalence classes C ,

where \(P({C}_{i}| {\boldsymbol{x}})=\frac{P({c}_{i}| {\boldsymbol{x}})}{{\sum }_{c}P(c| {\boldsymbol{x}})}\) estimates a categorical distribution over the cluster meanings, that is, ∑ i P ( C i ∣ x ) = 1. Without this normalization step cluster ‘probabilities’ could exceed one because of length normalization, resulting in degeneracies. Equation ( 5 ) is the estimator giving our main method that we refer to as semantic entropy throughout the text.

For scenarios in which the sequence probabilities are not available, we propose a variant of semantic entropy which we call ‘discrete’ semantic entropy. Discrete semantic entropy approximates P ( C i ∣ x ) directly from the number of generations in each cluster, disregarding the token probabilities. That is, we approximate P ( C i ∣ x ) as \({\sum }_{1}^{M}\frac{{I}_{c={C}_{i}}}{M}\) , the proportion of all the sampled answers which belong to that cluster. Effectively, this just assumes that each output that was actually generated was equally probable—estimating the underlying distribution as the categorical empirical distribution. In the limit of M the estimator converges to equation ( 5 ) by the law of large numbers. We find that discrete semantic entropy results in similar performance empirically.

We provide a worked example of the computation of semantic entropy in Supplementary Note  1 .

Semantic entropy is designed to detect confabulations, that is, model outputs with arbitrary meaning. In our experiments, we use semantic uncertainty to predict model accuracy, demonstrating that confabulations make up a notable fraction of model mistakes. We further show that semantic uncertainty can be used to improve model accuracy by refusing to answer questions when semantic uncertainty is high. Last, semantic uncertainty can be used to give users a way to know when model generations are probably unreliable.

We use the datasets BioASQ 34 , SQuAD 33 , TriviaQA 32 , SVAMP 37 and NQ-Open 35 . BioASQ is a life-sciences question-answering dataset based on the annual challenge of the same name. The specific dataset we use is based on the QA dataset from Task B of the 2023 BioASQ challenge (11B). SQuAD is a reading comprehension dataset whose context passages are drawn from Wikipedia and for which the answers to questions can be found in these passages. We use SQuAD 1.1 which excludes the unanswerable questions added in v.2.0 that are deliberately constructed to induce mistakes so they do not in practice cause confabulations to occur. TriviaQA is a trivia question-answering dataset. SVAMP is a word-problem maths dataset containing elementary-school mathematical reasoning tasks. NQ-Open is a dataset of realistic questions aggregated from Google Search which have been chosen to be answerable without reference to a source text. For each dataset, we use 400 train examples and 400 test examples randomly sampled from the original larger dataset. Note that only some of the methods require training, for example semantic entropy does not use the training data. If the datasets themselves are already split into train and test (or validation) samples, we sample our examples from within the corresponding split.

All these datasets are free-form, rather than multiple choice, because this better captures the opportunities created by LLMs to produce free-form sentences as answers. We refer to this default scenario as our ‘sentence-length’ experiments. In Supplementary Note  7 , we also present results for confabulation detection in a ‘short-phrase’ scenario, in which we constrain model answers on these datasets to be as concise as possible.

To make the problems more difficult and induce confabulations, we do not provide the context passages for any of the datasets. When the context passages are provided, the accuracy rate is too high for these datasets for the latest generations of models to meaningfully study confabulations.

For sentence-length generations we use: Falcon 39 Instruct (7B and 40B), LLaMA 2 Chat 38 (7B, 13B and 70B) and Mistral 40 Instruct (7B).

In addition to reporting results for semantic entropy, discrete semantic entropy and naive entropy, we consider two strong baselines.

Embedding regression is a supervised baseline inspired by the P (IK) method 24 . In that paper, the authors fine-tune their proprietary LLM on a dataset of questions to predict whether the model would have been correct. This requires access to a dataset of ground-truth answers to the questions. Rather than fine-tuning the entire LLM in this way, we simply take the final hidden units and train a logistic regression classifier to make the same prediction. By contrast to their method, this is much simpler because it does not require fine-tuning the entire language model, as well as being more reproducible because the solution to the logistic regression optimization problem is not as seed-dependent as the fine-tuning procedure. As expected, this supervised approach performs well in-distribution but fails when the distribution of questions is different from that on which the classifier is trained.

The second baseline we consider is the P (True) method 24 , in which the model first samples M answers (identically to our semantic entropy approach) and then is prompted with the list of all answers generated followed by the highest probability answer and a question whether this answer is “(a) True” or “(b) False”. The confidence score is then taken to be the probability with which the LLM responds with ‘a’ to the multiple-choice question. The performance of this method is boosted with a few-shot prompt, in which up to 20 examples from the training set are randomly chosen, filled in as above, but then provided with the actual ground truth of whether the proposed answer was true or false. In this way, the method can be considered as supervised ‘in-context’ because it makes use of some ground-truth training labels but can be used without retraining the model. Because of context-size constraints, this method cannot fit a full 20 few-shot examples in the context when input questions are long or large numbers of generations are used. As a result, we sometimes have to reduce the number of few-shot examples to suit the context size and we note this in the  Supplementary Material .

Entailment estimator

Any NLI classification system could be used for our bidirectional entailment clustering algorithm. We consider two different kinds of entailment detector.

One option is to use an instruction-tuned LLM such as LLaMA 2, GPT-3.5 (Turbo 1106) or GPT-4 to predict entailment between generations. We use the following prompt:

We are evaluating answers to the question {question} Here are two possible answers: Possible Answer 1: {text1} Possible Answer 2: {text2} Does Possible Answer 1 semantically entail Possible Answer 2? Respond with entailment, contradiction, or neutral.

Alternatively, we consider using a language model trained for entailment prediction, specifically the DeBERTa-large model 56 fine-tuned on the NLI dataset MNLI 58 . This builds on past work towards paraphrase identification based on embedding similarity 59 , 60 and BERT-style models 61 , 62 . We template more simply, checking if DeBERTa predicts entailment between the concatenation of the question and one answer and the concatenation of the question and another answer. Note that DeBERTa-large is a relatively lightweight model with only 1.5B parameters which is much less powerful than most of the LLMs under study.

In Supplementary Note 2 , we carefully evaluate the benefits and drawbacks of these methods for entailment prediction. We settle on using GPT-3.5 with the above prompt, as its entailment predictions agree well with human raters and lead to good confabulation detection performance.

In Supplementary Note  3 , we provide a discussion of the computational cost and choosing the number of generations for reliable clustering.

Prompting templates

We use a simple generation template for all sentence-length answer datasets:

Answer the following question in a single brief but complete sentence. Question: {question} Answer:

Metrics and accuracy measurements

We use three main metrics to evaluate our method: AUROC, rejection accuracy and AURAC. Each of these is grounded in an automated factuality estimation measurement relative to the reference answers provided by the datasets that we use.

AUROC, rejection accuracy and AURAC

First, we use the AUROC curve, which measures the reliability of a classifier accounting for both precision and recall. The AUROC can be interpreted as the probability that a randomly chosen correct answer has been assigned a higher confidence score than a randomly chosen incorrect answer. For a perfect classifier, this is 1.

Second, we compute the ‘rejection accuracy at X %’, which is the question-answering accuracy of the model on the most-confident X % of the inputs as identified by the respective uncertainty method. If an uncertainty method works well, predictions on the confident subset should be more accurate than predictions on the excluded subset and the rejection accuracy should increase as we reject more inputs.

To summarize this statistic we compute the AURAC—the total area enclosed by the accuracies at all cut-off percentages X %. This should increase towards 1 as given uncertainty method becomes more accurate and better at detecting likely-inaccurate responses but it is more sensitive to the overall accuracy of the model than the AUROC metric.

In Supplementary Note  5 , we provide the unaggregated rejection accuracies for sentence-length generations.

Assessing accuracy

For the short-phrase-length generation setting presented in Supplementary Note  7 , we simply assess the accuracy of the generations by checking if the F1 score of the commonly used SQuAD metric exceeds 0.5. There are limitations to such simple scoring rules 63 but this method is widely used in practice and its error is comparatively small on these standard datasets.

For our default scenario, the longer sentence-length generations, this measure fails, as the overlap between the short reference answer and our long model answer is invariably too small. For sentence-length generations, we therefore automatically determine whether an answer to the question is correct or incorrect by using GPT-4 to compare the given answer to the reference answer. We use the template:

We are assessing the quality of answers to the following question: {question} The expected answer is: {reference answer} The proposed answer is: {predicted answer} Within the context of the question, does the proposed answer mean the same as the expected answer? Respond only with yes or no.

We make a small modification for datasets with several reference answers: line two becomes “The following are expected answers to this question:” and the final line asks “does the proposed answer mean the same as any of the expected answers?”.

In Supplementary Note 6 , we check the quality of our automated ground-truth evaluations against human judgement by hand. We find that GPT-4 gives the best results for determining model accuracy and thus use it in all our sentence-length experiments.

In this section we describe the application of semantic entropy to confabulation detection in longer model generations, specifically paragraph-length biographies.

We introduce a biography-generation dataset—FactualBio—available alongside this paper. FactualBio is a collection of biographies of individuals who are notable enough to have Wikipedia pages but not notable enough to have large amounts of detailed coverage, generated by GPT-4 (v.0613). To generate the dataset, we randomly sampled 21 individuals from the WikiBio dataset 64 . For each biography, we generated a list of factual claims contained in each biography using GPT-4, with 150 total factual claims (the total number is only coincidentally a round number). For each of these factual claims, we manually determined whether the claim was correct or incorrect. Out of 150 claims, 45 were incorrect. As before, we apply confabulation detection to detect incorrect model predictions, even though there may be model errors which are not confabulations.

Prompting and generation

Given a paragraph-length piece of LLM-generated text, we apply the following sequence of steps:

Automatically decompose the paragraph into specific factual claims using an LLM (not necessarily the same as the original).

For each factual claim, use an LLM to automatically construct Q questions which might have produced that claim.

For each question, prompt the original LLM to generate M answers.

For each question, compute the semantic entropy of the answers, including the original factual claim.

Average the semantic entropies over the questions to arrive at a score for the original factual claim.

We pursue this slightly indirect way of generating answers because we find that simply resampling each sentence creates variation unrelated to the uncertainty of the model about the factual claim, such as differences in paragraph structure.

We decompose the paragraph into factual claims using the following prompt:

Please list the specific factual propositions included in the answer above. Be complete and do not leave any factual claims out. Provide each claim as a separate sentence in a separate bullet point.

We found that we agreed with the decompositions in all cases in the dataset.

We then generate six questions for each of the facts from the decomposition. We generate these questions by prompting the model twice with the following:

Following this text: {text so far} You see the sentence: {proposition} Generate a list of three questions, that might have generated the sentence in the context of the preceding original text, as well as their answers. Please do not use specific facts that appear in the follow-up sentence when formulating the question. Make the questions and answers diverse. Avoid yes-no questions. The answers should not be a full sentence and as short as possible, e.g. only a name, place, or thing. Use the format “1. {question} – {answer}”.

These questions are not necessarily well-targeted and the difficulty of this step is the main source of errors in the procedure. We generate three questions with each prompt, as this encourages diversity of the questions, each question targeting a different aspect of the fact. However, we observed that the generated questions will sometimes miss obvious aspects of the fact. Executing the above prompt twice (for a total of six questions) can improve coverage. We also ask for brief answers because the current version of GPT-4 tends to give long, convoluted and highly hedged answers unless explicitly told not to.

Then, for each question, we generate three new answers using the following prompt:

We are writing an answer to the question “{user question}”. So far we have written: {text so far} The next sentence should be the answer to the following question: {question} Please answer this question. Do not answer in a full sentence. Answer with as few words as possible, e.g. only a name, place, or thing.

We then compute the semantic entropy over these answers plus the original factual claim. Including the original fact ensures that the estimator remains grounded in the original claim and helps detect situations in which the question has been interpreted completely differently from the original context. We make a small modification to handle the fact that GPT-4 generations often include refusals to answer questions. These refusals were not something we commonly observe in our experiments with LLaMA 2, Falcon or Mistral models. If more than half of the answers include one of the strings ‘not available’, ‘not provided’, ‘unknown’ or ‘unclear’ then we treat the semantic uncertainty as maximal.

We then average the semantic entropies for each question corresponding to the factual claim to get an entropy for this factual claim.

Despite the extra assumptions and complexity, we find that this method greatly outperforms the baselines.

To compute semantic entailment between the original claim and regenerated answers, we rely on the DeBERTa entailment prediction model as we find empirically that DeBERTa predictions result in higher train-set AUROC than other methods. Because DeBERTa has slightly lower recall than GPT-3.5/4, we use a modified set-up for which we say the answers mean the same as each other if at least one of them entails the other and neither is seen to contradict the other—a kind of ‘non-defeating’ bidirectional entailment check rather than true bidirectional entailment. The good performance of DeBERTa in this scenario is not surprising as both factual claims and regenerated answers are relatively short. We refer to Supplementary Notes 2 and 3 for ablations and experiments regarding our choice of entailment estimator for paragraph-length generations.

We implement two baselines. First, we implement a variant of the P (True) method, which is adapted to the new setting. For each factoid, we generate a question with answers in the same way as for semantic entropy. We then use the following prompt:

Question: {question} Here are some brainstormed ideas: {list of regenerated answers} Possible answer: {original answer} Is the possible answer true? Respond with “yes” or “no”.

As we cannot access the probabilities GPT-4 assigns to predicting ‘yes’ and ‘no’ as the next token, we approximate this using Monte Carlo samples. Concretely, we execute the above prompt ten times (at temperature 1) and then take the fraction of answers which was ‘yes’ as our unbiased Monte Carlo estimate of the token probability GPT-4 assigns to ‘yes’.

As a second, simpler, baseline we check if the model thinks the answer is true. We simply ask:

Following this text: {text so far} You see this statement: {proposition} Is it likely that the statement is true? Respond with ‘yes’ or ‘no’.

It is interesting that this method ought to perform very well if we think that the model has good ‘self-knowledge’ (that is, if “models mostly know what they don’t know” 24 ) but in fact semantic entropy is much better at detecting confabulations.

Data availability

The data used for the short-phrase and sentence-length generations are publicly available and the released code details how to access it. We release a public version of the FactualBio dataset as part of the code base for reproducing the paragraph-length experiments.

Code availability

We release all code used to produce the main experiments. The code for short-phrase and sentence-length experiments can be found at github.com/jlko/semantic_uncertainty and https://doi.org/10.5281/zenodo.10964366 (ref. 65 ). The code for paragraph-length experiments can be found at github.com/jlko/long_hallucinations and https://doi.org/10.5281/zenodo.10964366 (ref. 65 ).

GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).

Gemini: a family of highly capable multimodal models. Preprint at https://arxiv.org/abs/2312.11805 (2023).

Xiao, Y. & Wang, W. Y. On hallucination and predictive uncertainty in conditional language generation. In Proc. 16th Conference of the European Chapter of the Association for Computational Linguistics 2734–2744 (Association for Computational Linguistics, 2021).

Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T. & Saenko, K. Object hallucination in image captioning. In Proc. 2018 Conference on Empirical Methods in Natural Language Processing (eds Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J.) 4035–4045 (Association for Computational Linguistics, 2018).

Weiser, B. Lawyer who used ChatGPT faces penalty for made up citations. The New York Times (8 Jun 2023).

Opdahl, A. L. et al. Trustworthy journalism through AI. Data Knowl. Eng . 146 , 102182 (2023).

Shen, Y. et al. ChatGPT and other large language models are double-edged swords. Radiology 307 , e230163 (2023).

Article   PubMed   Google Scholar  

Schulman, J. Reinforcement learning from human feedback: progress and challenges. Presented at the Berkeley EECS Colloquium. YouTube www.youtube.com/watch?v=hhiLw5Q_UFg (2023).

Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. 55 , 248 (2023).

Maynez, J., Narayan, S., Bohnet, B. & McDonald, R. On faithfulness and factuality in abstractive summarization. In Proc. 58th Annual Meeting of the Association for Computational Linguistics (eds Jurafsky, D., Chai, J., Schluter, N. & Tetreault, J.) 1906–1919 (Association for Computational Linguistics, 2020).

Filippova, K. Controlled hallucinations: learning to generate faithfully from noisy data. In Findings of the Association for Computational Linguistics: EMNLP 2020 (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 864–870 (Association for Computational Linguistics, 2020).

Berrios, G. Confabulations: a conceptual history. J. Hist. Neurosci. 7 , 225–241 (1998).

Article   CAS   PubMed   Google Scholar  

Lin, S., Hilton, J. & Evans, O. Teaching models to express their uncertainty in words. Transact. Mach. Learn. Res. (2022).

Evans, O. et al. Truthful AI: developing and governing AI that does not lie. Preprint at https://arxiv.org/abs/2110.06674 (2021).

Amodei, D. et al. Concrete problems in AI safety. Preprint at https://arxiv.org/abs/1606.06565 (2016).

Jiang, Z., Araki, J., Ding, H. & Neubig, G. How can we know when language models know? On the calibration of language models for question answering. Transact. Assoc. Comput. Linguist. 9 , 962–977 (2021).

Article   Google Scholar  

Desai, S. & Durrett, G. Calibration of pre-trained transformers. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 295–302 (Association for Computational Linguistics, 2020).

Glushkova, T., Zerva, C., Rei, R. & Martins, A. F. Uncertainty-aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021 (eds Moens, M-F., Huang, X., Specia, L. & Yih, S.) 3920–3938 (Association for Computational Linguistics, 2021).

Wang, Y., Beck, D., Baldwin, T. & Verspoor, K. Uncertainty estimation and reduction of pre-trained models for text regression. Transact. Assoc. Comput. Linguist. 10 , 680–696 (2022).

Baker, S. & Kanade, T. Hallucinating faces. In Proc. Fourth IEEE International Conference on Automatic Face and Gesture Recognition . 83–88 (IEEE, Catalogue no PR00580, 2002).

Eliot, L. AI ethics lucidly questioning this whole hallucinating AI popularized trend that has got to stop. Forbes Magazine (24 August 2022).

Shanahan, M. Talking about large language models. Commun. Assoc. Comp. Machinery 67 , 68–79 (2024).

MacKay, D. J. C. Information-based objective functions for active data selection. Neural Comput. 4 , 590–604 (1992).

Kadavath, S. et al. Language models (mostly) know what they know. Preprint at https://arxiv.org/abs/2207.05221 (2022).

Lindley, D. V. On a measure of the information provided by an experiment. Ann. Math. Stat. 27 , 986–1005 (1956).

Article   MathSciNet   Google Scholar  

Xiao, T. Z., Gomez, A. N. & Gal, Y. Wat zei je? Detecting out-of-distribution translations with variational transformers. In Workshop on Bayesian Deep Learning at the Conference on Neural Information Processing Systems (NeurIPS, Vancouver, 2019).

Christiano, P., Cotra, A. & Xu, M. Eliciting Latent Knowledge (Alignment Research Center, 2021); https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit .

Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D. & Marchetti, A. Divide and conquer: crowdsourcing the creation of cross-lingual textual entailment corpora. In Proc. 2011 Conference on Empirical Methods in Natural Language Processing 670–679 (Association for Computational Linguistics, 2011).

Honovich, O. et al. TRUE: Re-evaluating factual consistency evaluation. In Proc. Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering 161–175 (Association for Computational Linguistics, 2022).

Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I. & Gurevych, I. Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In Proc. 57th Annual Meeting of the Association for Computational Linguistics 2214–2220 (Association for Computational Linguistics, 2019).

Laban, P., Schnabel, T., Bennett, P. N. & Hearst, M. A. SummaC: re-visiting NLI-based models for inconsistency detection in summarization. Trans. Assoc. Comput. Linguist. 10 , 163–177 (2022).

Joshi, M., Choi, E., Weld, D. S. & Zettlemoyer, L. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proc. 55th Annual Meeting of the Association for Computational Linguistics 1601–1611 (Association for Computational Linguistics. 2017).

Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ questions for machine compression of text. In Proc. 2016 Conference on Empirical Methods in Natural Language Processing (eds Su, J., Duh, K. & Carreras, X.) 2383–2392 (Association for Computational Linguistics, 2016).

Tsatsaronis, G. et al. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics 16 , 138 (2015).

Article   PubMed   PubMed Central   Google Scholar  

Lee, K., Chang, M.-W. & Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Proc. 57th Annual Meeting of the Association for Computational Linguistics 6086–6096 (Association for Computational Linguistics, 2019).

Kwiatkowski, T. et al. Natural questions: a benchmark for question answering research. Transact. Assoc. Comput. Linguist. 7 , 452–466 (2019).

Patel, A., Bhattamishra, S. & Goyal, N. Are NLP models really able to solve simple math word problems? In Proc. 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Toutanova, K. et al.) 2080–2094 (Assoc. Comp. Linguistics, 2021).

Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https://arxiv.org/abs/2307.09288 (2023).

Penedo, G. et al. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. In Proc. 36th Conference on Neural Information Processing Systems (eds Oh, A. et al.) 79155–79172 (Curran Associates, 2023)

Jiang, A. Q. et al. Mistral 7B. Preprint at https://arxiv.org/abs/2310.06825 (2023).

Manakul, P., Liusie, A. & Gales, M. J. F. SelfCheckGPT: Zero-Resource Black-Box hallucination detection for generative large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 (eds Bouamor, H., Pino, J. & Bali, K.) 9004–9017 (Assoc. Comp. Linguistics, 2023).

Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. & Gal, Y. Deep deterministic uncertainty: a new simple baseline. In IEEE/CVF Conference on Computer Vision and Pattern Recognition 24384–24394 (Computer Vision Foundation, 2023).

Schuster, T., Chen, S., Buthpitiya, S., Fabrikant, A. & Metzler, D. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022 (eds Goldberg, Y. et al.) 394–412 (Association for Computational Linguistics, 2022).

Barnes, B. & Christiano, P. Progress on AI Safety via Debate. AI Alignment Forum www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1 (2020).

Irving, G., Christiano, P. & Amodei, D. AI safety via debate. Preprint at https://arxiv.org/abs/1805.00899 (2018).

Der Kiureghian, A. & Ditlevsen, O. Aleatory or epistemic? Does it matter? Struct. Saf. 31 , 105–112 (2009).

Malinin, A. & Gales, M. Uncertainty estimation in autoregressive structured prediction. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=jN5y-zb5Q7m (2021).

Murray, K. & Chiang, D. Correcting length bias in neural machine translation. In Proc. Third Conference on Machine Translation (eds Bojar, O. et al.) 212–223 (Assoc. Comp. Linguistics, 2018).

Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=rygGQyrFvH (2020).

Fan, A., Lewis, M. & Dauphin, Y. Hierarchical neural story generation. In Proc. 56th Annual Meeting of the Association for Computational Linguistics (eds Gurevych, I. & Miyao, Y.) 889–898 (Association for Computational Linguistics, 2018).

Speaks, J. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford Univ., 2021).

Culicover, P. W. Paraphrase generation and information retrieval from stored text. Mech. Transl. Comput. Linguist. 11 , 78–88 (1968).

Google Scholar  

Padó, S., Cer, D., Galley, M., Jurafsky, D. & Manning, C. D. Measuring machine translation quality as semantic equivalence: a metric based on entailment features. Mach. Transl. 23 , 181–193 (2009).

Androutsopoulos, I. & Malakasiotis, P. A survey of paraphrasing and textual entailment methods. J. Artif. Intell. Res. 38 , 135–187 (2010).

MacCartney, B. Natural Language Inference (Stanford Univ., 2009).

He, P., Liu, X., Gao, J. & Chen, W. Deberta: decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations https://openreview.net/forum?id=XPZIaotutsD (2021).

Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33 , 1877–1901 (2020).

Williams, A., Nangia, N. & Bowman, S. R. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Walker, M. et al.) 1112–1122 (Assoc. Comp. Linguistics, 2018).

Yu, L., Hermann, K. M., Blunsom, P. & Pulman, S. Deep learning for answer sentence selection. Preprint at https://arxiv.org/abs/1412.1632 (2014).

Socher, R., Huang, E., Pennin, J., Manning, C. D. & Ng, A. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of the 24th Conference on Neural Information Processing Systems (eds Shawe-Taylor, J. et al.) (2011)

He, R., Ravula, A., Kanagal, B. & Ainslie, J. Realformer: Transformer likes residual attention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (eds Zhong, C., et al.) 929–943 (Assoc. Comp. Linguistics, 2021).

Tay, Y. et al. Charformer: fast character transformers via gradient-based subword tokenization. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=JtBRnrlOEFN (2022).

Kane, H., Kocyigit, Y., Abdalla, A., Ajanoh, P. & Coulibali, M. Towards neural similarity evaluators. In Workshop on Document Intelligence at the 32nd conference on Neural Information Processing (2019).

Lebret, R., Grangier, D. & Auli, M. Neural text generation from structured data with application to the biography domain. In Proc. 2016 Conference on Empirical Methods in Natural Language Processing (eds Su, J. et al.) 1203–1213 (Association for Computational Linguistics, 2016).

Kossen, J., jlko/semantic_uncertainty: Initial release v.1.0.0. Zenodo https://doi.org/10.5281/zenodo.10964366 (2024).

Download references

Acknowledgements

We thank G. Irving, K. Perlin, J. Richens, L. Rimell and M. Turpin for their comments or discussion related to this work. We thank K. Handa for his help with the human evaluation of our automated accuracy assessment. We thank F. Bickford Smith and L. Melo for their code review. Y.G. is supported by a Turing AI Fellowship funded by the UK government’s Office for AI, through UK Research and Innovation (grant reference EP/V030302/1), and delivered by the Alan Turing Institute.

Author information

These authors contributed equally: Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn

Authors and Affiliations

OATML, Department of Computer Science, University of Oxford, Oxford, UK

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn & Yarin Gal

You can also search for this author in PubMed   Google Scholar

Contributions

S.F. led the work from conception to completion and proposed using bidirectional entailment to cluster generations as a way of computing entropy in LLMs. He wrote the main text, most of the Methods and Supplementary Information and prepared most of the figures. J.K. improved the mathematical formalization of semantic entropy; led the extension of semantic entropy to sentence- and paragraph-length generations; wrote the code for, and carried out, all the experiments and evaluations; wrote much of the Methods and Supplementary Information and prepared drafts of many figures; and gave critical feedback on the main text. L.K. developed the initial mathematical formalization of semantic entropy; wrote code for, and carried out, the initial experiments around semantic entropy and its variants which demonstrated the promise of the idea and helped narrow down possible research avenues to explore; and gave critical feedback on the main text. Y.G. ideated the project, proposing the idea to differentiate semantic and syntactic diversity as a tool for detecting hallucinations, provided high-level guidance on the research and gave critical feedback on the main text; he runs the research laboratory in which the work was carried out.

Corresponding author

Correspondence to Sebastian Farquhar .

Ethics declarations

Competing interests.

S.F. is currently employed by Google DeepMind and L.K. by OpenAI. For both, this paper was written under their University of Oxford affiliation. The remaining authors declare no competing interests.

Peer review

Peer review information.

Nature thanks Mirella Lapata and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended data fig. 1 algorithm outline for bidirectional entailment clustering..

Given a set of outputs in response to a context, the bidirectional entailment answer returns a set of sets of outputs which have been classified as sharing a meaning.

Supplementary information

Supplementary information.

Supplementary Notes 1–7, Figs. 1–10, Tables 1–4 and references. Includes, worked example for semantic entropy calculation, discussion of limitations and computational cost of entailment clustering, ablation of entailment prediction and clustering methods, discussion of automated accuracy assessment, unaggregated results for sentence-length generations and further results for short-phrase generations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Farquhar, S., Kossen, J., Kuhn, L. et al. Detecting hallucinations in large language models using semantic entropy. Nature 630 , 625–630 (2024). https://doi.org/10.1038/s41586-024-07421-0

Download citation

Received : 17 July 2023

Accepted : 12 April 2024

Published : 19 June 2024

Issue Date : 20 June 2024

DOI : https://doi.org/10.1038/s41586-024-07421-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Starting the research process
• 10 Research Question Examples to Guide Your Research Project

10 Research Question Examples to Guide your Research Project

Published on October 30, 2022 by Shona McCombes . Revised on October 19, 2023.

The research question is one of the most important parts of your research paper , thesis or dissertation . It’s important to spend some time assessing and refining your question before you get started.

The exact form of your question will depend on a few things, such as the length of your project, the type of research you’re conducting, the topic , and the research problem . However, all research questions should be focused, specific, and relevant to a timely social or scholarly issue.

Once you’ve read our guide on how to write a research question , you can use these examples to craft your own.

Note that the design of your research question can depend on what method you are pursuing. Here are a few options for qualitative, quantitative, and statistical research questions.

Other interesting articles

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

Methodology

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, October 19). 10 Research Question Examples to Guide your Research Project. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/research-process/research-question-examples/

Is this article helpful?

Shona McCombes

Other students also liked, writing strong research questions | criteria & examples, how to choose a dissertation topic | 8 steps to follow, evaluating sources | methods & examples, what is your plagiarism score.

More From Forbes

Harvard business school announces 3 new application essays.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

Harvard Business School.

Harvard Business School announced a surprising departure from its single, open-ended application essay to three short essays with specific prompts. The HBS website sums up the kind of applicant the school is seeking: “We are looking for future leaders who are passionate about business, leadership, and growth.”

The prompts for the class that will begin in fall 2025 instruct applicants to address each topic in turn.

• Business-Minded Essay : Please reflect on how your experiences have influenced your career choices and aspirations and the impact you will have on the businesses, organizations, and communities you plan to serve. (up to 300 words)
• Leadership-Focused Essay : What experiences have shaped who you are, how you invest in others, and what kind of leader you want to become? (up to 250 words)
• Growth-Oriented Essay : Curiosity can be seen in many ways. Please share an example of how you have demonstrated curiosity and how that has influenced your growth. (up to 250 words)

The prompts ask applicants to go beyond simply asserting their allegiance to the ideals of business, leadership and growth. Each of the three questions asks for evidence: “experiences,” “experiences” and “an example,” respectively.

The prompts do not expect a straightforward list of what happened in the past. Rather, they encourage reflection on how these experiences affected present realities and future goals.

Applicants are asked to reflect on past, present and future as an ongoing process of becoming who they are now and who they wish to become. Even the “Business-Minded Essay” is about past choices and future impact; it also assumes you “plan to serve.” The “Leadership-Focused Essay” does not ask applicants to recite a list of titles, but to discuss who they are and how they relate to others; not what title they aspire to, but “what kind of leader you wish to become.”

Perhaps the most surprising essay prompt is No. 3, which asks about curiosity. It opens the door for applicants to discuss a more personal aspect of their candidacies. The prompt asks not about end result, but about the process of change. Once again, the emphasis is on “growth.”

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

In short, the prompts ask about person and process.

How The 3 New Prompts Differ From Last Year’s Single Question

This year’s prompts give applicants more direction than the previous open-ended instruction, which was: “As we review your application, what more would you like us to know as we consider your candidacy for the Harvard Business School MBA program?”

Applicants may find it easier to follow these more detailed instructions and to stay on topic. They no longer need to face an open question and a blank page.

Another aid is the shorter word limit. The essay on being business-minded has a limit of 300 words, and the essays on leadership and growth through curiosity are limited to 250 words each.

A third difference is the specific inquiry about business. Last year’s prompt allowed candidates to choose anything they thought would be important for HBS to consider. Some applicants struggled to decide whether to focus on business or something beyond work. While the “Business-Minded Essay” is still personal, it does ask applicants to reflect on their careers.

One might also speculate that the new, more directive prompts makes it easier for the admissions committee to compare essays across applications, while still leaving room for considerable variation in how applicants choose to address the essay prompts.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.

The Authorship of Federalist Paper No. 51: James Madison’s Enduring Influence

This essay about Federalist Paper No. 51, written by James Madison in 1788 under the pseudonym “Publius,” explores the principles of governmental structure and checks and balances. It highlights Madison’s arguments for the separation of powers among the legislative, executive, and judicial branches to prevent tyranny. The essay underscores Madison’s influence on American political theory and the enduring relevance of his ideas in contemporary constitutional discussions.

How it works

Number of Paper of Federalist 51, a central piece published in 1788 how part of Papers of Federalist, becomes the native stones of American political conversation. It is Created by James Madison under collective pseudonym of “Publius,” this essay is known for his research of governmental structure and principles checks and balances.

James Madison, key architect of Constitution of the united states and later his fourth President, locked Number of Paper of Federalist 51 with an aim clear:, to protect for the acceptance of Constitution by New York by gentlemen.

In that, Madison lays out on the requirement of department of governmental plenary powers in expressive branches-legislative, specialist, and judicial. Him hinges of argument on an idea, that every branch must have ability to check plenary powers of other, thus preventing to the arbitrary only branch to own excessive plenary powers.

Penetrating of Madison in the human wild and political management is especially obvious in his statement, that department of plenary powers and checks and balances are substantial guarantees against tyranny. His known statement, that “ambition must be done to counteract ambition” of his encapsulates faith that egoism can serve as a mechanism for maintenance of freedom within the limits of strukturuje governmental structure.

Number of Paper of Federalist 51 prolongs to philosophize in modern discussions about a constitutional law and management. Defence of Madison for the balanced distribution of plenary powers has deeply influenced on development of American political establishments. His ideas put foundation for the system that not only divides power but and guarantees, that not a single legal individual can prevail above other without responsibility.

After his historical value, bits and pieces of Number of Paper of Federalist 51, relevant today how the testament of foresight of Parents, what Founds, in treatment of strong structure for a democratic management. Arguments of Madison stood, as they call to the fundamental questions about nature of power and his proper placing in free society.

Upon completion, authorship of Madison survived James Numer of Paper of Federalist 51 of his underscores influence on the American political theory. Through this seminal essay, Madison articulated principles that formed the constitutional structure of the united states, doing an accent on importance checks and balances in maintenance of individual freedom and distraction of governmental cunning. His additions prolong to inform modern debates of role and frame of government, distinguishing his strong expediency to the idea in the guard of democratic principles.

Cite this page

The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence. (2024, Jun 28). Retrieved from https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/

"The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence." PapersOwl.com , 28 Jun 2024, https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/

PapersOwl.com. (2024). The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence . [Online]. Available at: https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/ [Accessed: 28 Jun. 2024]

"The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence." PapersOwl.com, Jun 28, 2024. Accessed June 28, 2024. https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/

"The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence," PapersOwl.com , 28-Jun-2024. [Online]. Available: https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/. [Accessed: 28-Jun-2024]

PapersOwl.com. (2024). The Authorship of Federalist Paper No. 51: James Madison's Enduring Influence . [Online]. Available at: https://papersowl.com/examples/the-authorship-of-federalist-paper-no-51-james-madisons-enduring-influence/ [Accessed: 28-Jun-2024]

Don't let plagiarism ruin your grade

Hire a writer to get a unique paper crafted to your needs.

Our writers will help you fix any mistakes and get an A+!

Please check your inbox.

You can order an original essay written according to your instructions.

Trusted by over 1 million students worldwide

1. Tell Us Your Requirements

2. Pick your perfect writer

3. Get Your Paper and Pay

Hi! I'm Amy, your personal assistant!

Don't know where to start? Give me your paper requirements and I connect you to an academic expert.

short deadlines

100% Plagiarism-Free

Certified writers

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Why Are Companies That Lose Money Still So Successful?

• Vijay Govindarajan,
• Shivaram Rajgopal,
• Anup Srivastava,
• Aneel Iqbal,
• Elnaz Basirian

New research on how to identify investments that produce delayed but real profits — not just those that produce short-term accounting profits.

In a well-functioning capital market, profits should be the sole criterion for firm survival; that is, firms reporting losses should disappear. Of late, however, loss-making firms are highly sought after by investors — often more than some profitable firms. Unicorns, or startups with valuations exceeding a billion dollars, are examples of such loss-making firms. What has changed over time? When and why did losses lose their meaning? The authors’ series of new research papers provide some answers, guiding managers to make the right investments: those that produce delayed but real profits — not just those that produce short-term accounting profits but decimate shareholder wealth in long run.

In 1979, psychologists Daniel Kahneman and Amos Tversky famously posited that losses loom larger than gains in human decision-making. For example, a dollar of loss affects our behavior more than a dollar of profits . Likewise, when a firm announces losses, its stock price declines more dramatically than it increases for the same dollar amount of profits. Investors abandon and lenders tend to stop financing loss-making firms , which then start restructuring their business lines and laying off employees. Some firms go even further, conducting M&A transactions without substance and “managing earnings” to report profits instead of a loss.

• Vijay Govindarajan is the Coxe Distinguished Professor at Dartmouth College’s Tuck School of Business, an executive fellow at Harvard Business School, and faculty partner at the Silicon Valley incubator Mach 49. He is a New York Times and Wall Street Journal bestselling author. His latest book is Fusion Strategy: How Real-Time Data and AI Will Power the Industrial Future . His Harvard Business Review articles “ Engineering Reverse Innovations ” and “ Stop the Innovation Wars ” won McKinsey Awards for best article published in HBR. His HBR articles “ How GE Is Disrupting Itself ” and “ The CEO’s Role in Business Model Reinvention ” are HBR all-time top-50 bestsellers. Follow him on LinkedIn . vgovindarajan
• Shivaram Rajgopal is the Roy Bernard Kester and T.W. Byrnes Professor of Accounting and Auditing and Vice Dean of Research at Columbia Business School. His research examines financial reporting and executive compensation issues and he is widely published in both accounting and finance.
• Anup Srivastava holds Canada Research Chair in Accounting, Decision Making, and Capital Markets and is a full professor at Haskayne School of Business, University of Calgary. In a series of HBR articles, he examines the management implications of digital disruption. He specializes in the valuation and financial reporting challenges of digital companies. Follow Anup on  LinkedIn .
• Aneel Iqbal is an assistant professor at Thunderbird School of Global Management, Arizona State University. He examines the accounting measurement and financial disclosures for new-economy firms and incorporates his wide-ranging industry experience into his research and teaching. He is a seasoned accounting and finance professional with diverse experience in auditing, financial analysis, business advisory, performance management, and executive training. Follow Aneel on LinkedIn .
• Elnaz Basirian is a PhD student at the Haskayne School of Business. She examines the influence and role of intangibles in accounting and finance, aimed at improving valuation and market efficiency. She brings a decade of work experience in international financial markets. Follow Elnaz on LinkedIn .

Partner Center