research paper citation website

• Plagiarism and grammar
• School access

The best papers start with EasyBib®

Powered by chegg.

Start a new citation or manage your existing projects.

Scan your paper for plagiarism and grammar errors.

Check your paper for grammar and plagiarism

Catch plagiarism and grammar mistakes with our paper checker

Wipe out writing errors with EasyBib® Plus

Double check for plagiarism mistakes and advanced grammar errors before you turn in your paper.

• expert check

Know you're citing correctly

No matter what citation style you're using (APA, MLA, Chicago, etc.) we'll help you create the right bibliography

Check for unintentional plagiarism

Scan your paper the way your teacher would to catch unintentional plagiarism. Then, easily add the right citation

Strengthen your writing

Give your paper an in-depth check. Receive feedback within 24 hours from writing experts on your paper's main idea, structure, conclusion, and more.

Find and fix grammar errors

Don't give up sweet paper points for small mistakes. Our algorithms flag grammar and writing issues and provide smart suggestions

Choose your online writing help

Easybib® guides & resources, mla format guide.

This is the total package when it comes to MLA format. Our easy to read guides come complete with examples and step-by-step instructions to format your full and in-text citations, paper, and works cited in MLA style. There’s even information on annotated bibliographies.

Works Cited | In-Text Citations | Bibliography | Annotated Bibliography | Website | Book | Journal | YouTube | View all MLA Citation Examples

APA Format Guide

Get the facts on citing and writing in APA format with our comprehensive guides. Formatting instructions, in-text citation and reference examples, and sample papers provide you with the tools you need to style your paper in APA.

Reference Page | In-Text Citations | Annotated Bibliography | Website | Books | Journal | YouTube | View all APA citation Examples

Chicago Format Guide

Looking to format your paper in Chicago style and not sure where to start? Our guide provides everything you need! Learn the basics and fundamentals to creating references and footnotes in Chicago format. With numerous examples and visuals, you’ll be citing in Chicago style in no time.

Footnotes | Website | Book | Journal

Harvard Referencing Guide

Learn the requirements to properly reference your paper in Harvard style. The guides we have provide the basics and fundamentals to give credit to the sources used in your work.

In-Text Citations | Books | Article | YouTube | View all Harvard Referencing Examples

Check Your Paper

Avoid common grammar mistakes and unintentional plagiarism with our essay checker. Receive personalized feedback to help identify citations that may be missing, and help improve your sentence structure, punctuation, and more to turn in an error-free paper.

Grammar Check | Plagiarism Checker | Spell Check

Learn From Our Innovative Blog

Our blog features current and innovative topics to keep you up to speed on citing and writing. Whether you’re an educator, student, or someone who lives and breathes citations (it’s not as uncommon as you might think!), our blog features new and exciting articles to discover and learn from.

Looking for Other Tools and Resources?

Our Writing Center is jam-packed with tons of exciting resources. Videos, infographics, research guides, and many other citation-related resources are found here. Check it out to find what you need to succeed!

• EasyBib® Plus
• Citation Guides
• Chicago Style Format
• Terms of Use
• Global Privacy Policy
• Cookie Notice
• DO NOT SELL MY INFO

• Free Tools for Students
• MLA Citation Generator

Free MLA Citation Generator

Generate accurate citations in MLA format automatically, with MyBib!

😕 What is an MLA Citation Generator?

An MLA citation generator is a software tool designed to automatically create academic citations in the Modern Language Association (MLA) citation format. The generator will take information such as document titles, author, and URLs as in input, and output fully formatted citations that can be inserted into the Works Cited page of an MLA-compliant academic paper.

The citations on a Works Cited page show the external sources that were used to write the main body of the academic paper, either directly as references and quotes, or indirectly as ideas.

👩‍🎓 Who uses an MLA Citation Generator?

MLA style is most often used by middle school and high school students in preparation for transition to college and further education. Ironically, MLA style is not actually used all that often beyond middle and high school, with APA (American Psychological Association) style being the favored style at colleges across the country.

It is also important at this level to learn why it's critical to cite sources, not just how to cite them.

🙌 Why should I use a Citation Generator?

Writing citations manually is time consuming and error prone. Automating this process with a citation generator is easy, straightforward, and gives accurate results. It's also easier to keep citations organized and in the correct order.

The Works Cited page contributes to the overall grade of a paper, so it is important to produce accurately formatted citations that follow the guidelines in the official MLA Handbook .

⚙️ How do I use MyBib's MLA Citation Generator?

It's super easy to create MLA style citations with our MLA Citation Generator. Scroll back up to the generator at the top of the page and select the type of source you're citing. Books, journal articles, and webpages are all examples of the types of sources our generator can cite automatically. Then either search for the source, or enter the details manually in the citation form.

The generator will produce a formatted MLA citation that can be copied and pasted directly into your document, or saved to MyBib as part of your overall Works Cited page (which can be downloaded fully later!).

MyBib supports the following for MLA style:

Daniel is a qualified librarian, former teacher, and citation expert. He has been contributing to MyBib since 2018.

• Plagiarism and grammar
• Citation guides

Cite a Website

Don't let plagiarism errors spoil your paper, citing a website in apa.

Once you’ve identified a credible website to use, create a citation and begin building your reference list. Citation Machine citing tools can help you create references for online news articles, government websites, blogs, and many other website! Keeping track of sources as you research and write can help you stay organized and ethical. If you end up not using a source, you can easily delete it from your bibliography. Ready to create a citation? Enter the website’s URL into the search box above. You’ll get a list of results, so you can identify and choose the correct source you want to cite. It’s that easy to begin!

If you’re wondering how to cite a website in APA, use the structure below.

Author Last Name, First initial. (Year, Month Date Published). Title of web page . Name of Website. URL

Example of an APA format website:

Austerlitz, S. (2015, March 3). How long can a spinoff like ‘Better Call Saul’ last? FiveThirtyEight. http://fivethirtyeight.com/features/how-long-can-a-spinoff-like-better-call-saul-last/

Keep in mind that not all information found on a website follows the structure above. Only use the Website format above if your online source does not fit another source category. For example, if you’re looking at a video on YouTube, refer to the ‘YouTube Video’ section. If you’re citing a newspaper article found online, refer to ‘Newspapers Found Online’ section. Again, an APA website citation is strictly for web pages that do not fit better with one of the other categories on this page.

Social media:

When adding the text of a post, keep the original capitalization, spelling, hashtags, emojis (if possible), and links within the text.

Facebook posts:

Structure: Facebook user’s Last name, F. M. (Year, Monday Day of Post). Up to the first 20 words of Facebook post [Source type if attached] [Post type]. Facebook. URL

Source type examples: [Video attached], [Image attached]

Post type examples: [Status update], [Video], [Image], [Infographic]

Gomez, S. (2020, February 4). Guys, I’ve been working on this special project for two years and can officially say Rare Beauty is launching in [Video]. Facebook. https://www.facebook.com/Selena/videos/1340031502835436/

Life at Chegg. (2020, February 7) It breaks our heart that 50% of college students right here in Silicon Valley are hungry. That’s why Chegg has [Images attached] [Status update]. Facebook. https://www.facebook.com/LifeAtChegg/posts/1076718522691591

Twitter posts:

Structure: Account holder’s Last name, F. M. [Twitter Handle]. (Year, Month Day of Post). Up to the first 20 words of tweet [source type if attached] [Tweet]. Twitter. URL

Source type examples: [Video attached], [Image attached], [Poll attached]

Example: Edelman, J. [Edelman11]. (2018, April 26). Nine years ago today my life changed forever. New England took a chance on a long shot and I’ve worked [Video attached] [Tweet]. Twitter. https://twitter.com/Edelman11/status/989652345922473985

Instagram posts:

APA citation format: Account holder’s Last name, F. M. [@Instagram handle]. (Year, Month Day). Up to the first 20 words of caption [Photograph(s) and/or Video(s)]. Instagram. URL

Example: Portman, N. [@natalieportman]. (2019, January 5). Many of my best experiences last year were getting to listen to and learn from so many incredible people through [Videos]. Instagram. https://www.instagram.com/p/BsRD-FBB8HI/?utm_source=ig_web_copy_link

If this guide hasn’t helped solve all of your referencing questions, or if you’re still feeling the need to type “how to cite a website APA” into Google, then check out our APA citation generator on CitationMachine.com, which can build your references for you!

Featured links:

APA Citation Generator  |  Website   | Books | Journal Articles | YouTube | Images | Movies | Interview | PDF

• Citation Machine® Plus
• Citation Guides
• Chicago Style
• Harvard Referencing
• Terms of Use
• Global Privacy Policy
• Cookie Notice
• DO NOT SELL MY INFO

Referencing Generator

Powered by chegg.

• Select style:
• Archive material
• Chapter of an edited book
• Conference proceedings
• Dictionary entry
• Dissertation
• DVD, video, or film
• E-book or PDF
• Edited book
• Encyclopedia article
• Government publication
• Music or recording
• Online image or video
• Presentation
• Press release
• Religious text

What Is Cite This For Me’s Reference Generator?

Cite This For Me’s open-access generator is an automated citation machine that turns any of your sources into references in just a click. Using a reference generator helps students to integrate referencing into their research and writing routine; turning a time-consuming ordeal into a simple task.

A referencing generator accesses information from across the web, drawing the relevant information into a fully-formatted bibliography that clearly presents all of the sources that have contributed to your work.

If you don’t know how to reference a website correctly, or have a fast-approaching deadline, Cite This For Me’s accurate and intuitive reference generator will lend you the confidence to realise your full academic potential. In order to get a grade that reflects all your hard work, your references must be accurate and complete. Using a citation machine not only saves you time but also ensures that you don’t lose valuable marks on your assignment.

Not sure how to format your citations, what citations are, or just want to find out more about Cite This For Me’s reference generator? This guide outlines everything you need to know to equip yourself with the know-how and confidence to research and cite a wide range of diverse sources in your work.

Why Do I Need To Reference?

Simply put, when another source contributes to your work, you have to give the original owner the appropriate credit. After all, you wouldn’t steal someone else’s possessions so why would you steal their ideas?

Regardless of whether you are referencing a website, an article or a podcast, any factual material or ideas you take from another source must be acknowledged in a citation unless it is common knowledge (e.g. Winston Churchill was English). Failing to credit all of your sources, even when you’ve paraphrased or completely reworded the information, is plagiarism. Plagiarising will result in disciplinary action, which can range from losing precious marks on your assignment to expulsion from your university.

What’s more, attributing your research infuses credibility and authority into your work, both by supporting your own ideas and by demonstrating the breadth of your research. For many students, crediting sources can be a confusing and tedious process, but it’s a surefire way to improve the quality of your work so it’s essential to get it right. Luckily for you, using Cite This For Me’s reference generator makes creating accurate references easier than ever, leaving more time for you to excel in your studies.

In summary, the citing process serves three main functions:

• To validate the statements and conclusions in your work by providing directions to other sound sources that support and verify them.
• To help your readers locate, read and check your sources, as well as establishing their contribution to your work.
• To give credit to the original author and hence avoid committing intellectual property theft (known as ‘plagiarism’ in academia).

How Do I Cite My Sources With The Cite This For Me Referencing Generator?

Cite This For Me’s reference generator is the most accurate citation machine available, so whether you’re not sure how to format in-text references or are looking for a foolproof solution to automate a fully-formatted bibliography, this referencing generator will solve all of your citing needs.

Crediting your source material doesn’t just prevent you from losing valuable marks for plagiarism, it also provides all of the information to help your reader find for themselves the book, article, or other item you are citing. The accessible interface of the reference generator makes it easy for you to identify the source you have used – simply enter its unique identifier into the citation machine search bar. If this information is not available you can search for the title or author instead, and then select from the search results that appear below the reference generator.

Don’t know how to reference a website? The good news is that by using tools such as Cite This For Me’s reference generator, which help you work smarter, you don’t need to limit your research to sources that are traditional to cite. In fact, there are no limits to what you can cite, whether you are referencing a website, a YouTube video or a tweet.

To use the reference generator, simply:

• Select your style from Harvard, APA, OSCOLA and many more*
• Choose the type of source you would like to cite (e.g. website, book, journal, video)
• Enter the URL , DOI , ISBN , title, or other unique source information to find your source
• Click the ‘Cite’ button on the reference generator
• Copy your new citation straight from the referencing generator into your bibliography
• Repeat for each source that has contributed to your work.

*If you require another style for your paper, essay or other academic work, you can select from over 1,000 styles by creating a free Cite This For Me account.

Once you have created your Cite This For Me account you will be able to use the reference generator to create multiple references and save them into a project. Use Cite This For Me’s highly-rated iOS or Android apps to generate references in a flash with your smartphone camera, export your complete bibliography in one go, and much more.

What Will The Reference Generator Create For Me?

Cite This For Me’s reference generator will create your citation in two parts: an in-text citation and a full citation to be copied straight into your work.

The reference generator will auto-generate the correct formatting for your bibliography depending on your chosen style. For instance, if you select a parenthetical style the reference generator will generate an in-text citation in parentheses, along with a full citation to slot into your bibliography. Likewise, if the reference generator is set to a footnote style then it will create a fully-formatted citation for your reference list and bibliography, as well as a corresponding footnote to insert at the bottom of the page containing the relevant source.

Parenthetical style examples:

In-text example: A nation has been defined as an imagined community (Anderson, 2006).* Alternative format: Anderson (2006) defined a nation as an imagined community.

*The reference generator will create your references in the first style, but this should be edited if the author’s name already appears in the text.

Bibliography / Works Cited list example: Anderson, B. (2006). Imagined Communities. London: Verso.

What Are Citation Styles?

A citation style is a set of rules that you, as an academic writer, must follow to ensure the quality and relevance of your work. There are thousands of styles that are used in different academic institutions around the world, but in the UK the most common are Harvard, APA and Oscola.

The style you need to use will depend on the preference of your lecturer, discipline or academic institution – so if you’re unsure which style you should be using, consult your department and follow their guidelines exactly, as this is what you’ll be evaluated on when it comes to marking. You can also find your university’s style by logging into your Cite This For Me account and setting your institution in ‘My Profile’.

Citing isn’t just there to guard against plagiarism – presenting your research in a clear and consistent way eases the reader’s comprehension. Each style has a different set of rules for formatting both the page and your references. Be sure to adhere to formatting rules such as font type, font size and line spacing to ensure that your work is easily legible. Furthermore, if your work is published as part of an anthology or collected works, each entry will need to be presented in the same style to maintain uniformity throughout. It is important to make sure that you don’t jump from one style to another, so follow the rules carefully to ensure your reference list and bibliography are both accurate and complete.

If you need a hand with your citations then why not try Cite This For Me’s reference generator? It’s the quickest and easiest way to cite any source, in any style. The reference generator above will create your citations in the Harvard referencing style as standard, but it can generate fully-formatted references in over 1,000 styles – including university variations of each style. So, whether your lecturer has asked you to adopt APA referencing , or your subject requires you to use OSCOLA referencing , we’re sure to have the style you need. To access all of them, simply go to Cite This For Me’s website to create your free Cite This For Me account and search for your specific style such as MLA or Vancouver .

How Do I Format A Reference List Or Bibliography?

Drawing on a wide range of sources greatly enhances the quality of your work, and reading above and beyond your recommended reading list – and then using these sources to support your own thesis – is an excellent way to impress your reader. A clearly presented reference list or bibliography demonstrates the lengths you have gone to in researching your chosen topic.

Typically, a reference list starts on a new page at the end of the main body of text and includes a complete list of the sources you have actually cited in your paper. This list should contain all the information needed for the reader to locate the original source of the information, quote or statistic that directly contributed to your work. On the other hand, a bibliography is a comprehensive list of all the material you may have consulted throughout your research and writing process. Both provide the necessary information for readers to retrieve and check the sources cited in your work.

Each style’s guidelines will define the terminology of ‘reference list’ and ‘bibliography’, as well as providing formatting guidelines for font, line spacing and page indentations. In addition, it will instruct you on how to order each list – this will usually be either alphabetical or chronological (meaning the order that these sources appear in your work). Before submitting your work, be sure to check that you have formatted your whole paper according to your style’s formatting guidelines.

Sounds complicated? Citing has never been so easy; Cite This For Me’s reference generator will automatically generate fully-formatted citations for your reference list or bibliography in your chosen style. Sign in to your Cite This For Me account to save and export your bibliography.

How Do References Actually Work?

Although the reference generator will create your bibliography for you in record time, it is still useful to understand how this system works behind the scenes. As well as saving you time with its referencing generator, Cite This For Me provides the learning resources to help you fully understand the citing process and the benefits of adopting great citing standards.

The referencing process:

• Find a book, journal, website or other source that will contribute to your work
• Save the quote, image, data or other information that you will use in your work
• Save the source information that enables you to find it again (i.e. URL, ISBN, DOI etc.)
• Format the source information into a citation
• Copy and paste the citation into the body of the text
• Repeat for each source that contributes to your work.
• Export or copy and paste the fully-formatted citation into your bibliography.

Manage all your references in one place

Create projects, add notes, cite directly from the browser and scan books’ barcodes with a mobile app.

Sign up to Cite This For Me – the ultimate reference management tool.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Referencing

A Quick Guide to Harvard Referencing | Citation Examples

Published on 14 February 2020 by Jack Caulfield . Revised on 15 September 2023.

Referencing is an important part of academic writing. It tells your readers what sources you’ve used and how to find them.

Harvard is the most common referencing style used in UK universities. In Harvard style, the author and year are cited in-text, and full details of the source are given in a reference list .

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Harvard in-text citation, creating a harvard reference list, harvard referencing examples, referencing sources with no author or date, frequently asked questions about harvard referencing.

A Harvard in-text citation appears in brackets beside any quotation or paraphrase of a source. It gives the last name of the author(s) and the year of publication, as well as a page number or range locating the passage referenced, if applicable:

Note that ‘p.’ is used for a single page, ‘pp.’ for multiple pages (e.g. ‘pp. 1–5’).

An in-text citation usually appears immediately after the quotation or paraphrase in question. It may also appear at the end of the relevant sentence, as long as it’s clear what it refers to.

When your sentence already mentions the name of the author, it should not be repeated in the citation:

Sources with multiple authors

When you cite a source with up to three authors, cite all authors’ names. For four or more authors, list only the first name, followed by ‘ et al. ’:

Sources with no page numbers

Some sources, such as websites , often don’t have page numbers. If the source is a short text, you can simply leave out the page number. With longer sources, you can use an alternate locator such as a subheading or paragraph number if you need to specify where to find the quote:

Multiple citations at the same point

When you need multiple citations to appear at the same point in your text – for example, when you refer to several sources with one phrase – you can present them in the same set of brackets, separated by semicolons. List them in order of publication date:

Multiple sources with the same author and date

If you cite multiple sources by the same author which were published in the same year, it’s important to distinguish between them in your citations. To do this, insert an ‘a’ after the year in the first one you reference, a ‘b’ in the second, and so on:

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

A bibliography or reference list appears at the end of your text. It lists all your sources in alphabetical order by the author’s last name, giving complete information so that the reader can look them up if necessary.

The reference entry starts with the author’s last name followed by initial(s). Only the first word of the title is capitalised (as well as any proper nouns).

Sources with multiple authors in the reference list

As with in-text citations, up to three authors should be listed; when there are four or more, list only the first author followed by ‘ et al. ’:

Reference list entries vary according to source type, since different information is relevant for different sources. Formats and examples for the most commonly used source types are given below.

• Entire book
• Book chapter
• Translated book
• Edition of a book

Journal articles

• Print journal
• Online-only journal with DOI
• Online-only journal with no DOI
• General web page
• Online article or blog
• Social media post

Sometimes you won’t have all the information you need for a reference. This section covers what to do when a source lacks a publication date or named author.

No publication date

When a source doesn’t have a clear publication date – for example, a constantly updated reference source like Wikipedia or an obscure historical document which can’t be accurately dated – you can replace it with the words ‘no date’:

Note that when you do this with an online source, you should still include an access date, as in the example.

When a source lacks a clearly identified author, there’s often an appropriate corporate source – the organisation responsible for the source – whom you can credit as author instead, as in the Google and Wikipedia examples above.

When that’s not the case, you can just replace it with the title of the source in both the in-text citation and the reference list:

Harvard referencing uses an author–date system. Sources are cited by the author’s last name and the publication year in brackets. Each Harvard in-text citation corresponds to an entry in the alphabetised reference list at the end of the paper.

Vancouver referencing uses a numerical system. Sources are cited by a number in parentheses or superscript. Each number corresponds to a full reference at the end of the paper.

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

Though the terms are sometimes used interchangeably, there is a difference in meaning:

• A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
• A bibliography also includes other sources which were consulted during the research but not cited.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2023, September 15). A Quick Guide to Harvard Referencing | Citation Examples. Scribbr. Retrieved 15 April 2024, from https://www.scribbr.co.uk/referencing/harvard-style/

Is this article helpful?

Jack Caulfield

Other students also liked, harvard in-text citation | a complete guide & examples, harvard style bibliography | format & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

• Ask a Librarian

Citation Databases

• Searching Scopus
• Researcher Profiles
• Researcher IDs
• Searching Web of Science
• Google Scholar
• Research Impact and Metrics

Bibliometrics Overview

Welcome to Purdue University's Citation Databases Research Guide

Here you will find...

• Information on citation databases
• Descriptions of these databases
• Helpful examples on how to use them
• Use cases for each of the major citation databases
• Useful tips and tricks on how to best make use of them

Here are some definitions of common terms made use of by citation databases

• Bibliometrics is the statistical analysis of scholarly output like articles, book chapters, and reviews.
• Altmetrics: is the statistical analysis of alternative forms of capture such as twitter impressions of a piece of scholarly output.

Some common metrics are the H-index, Journal Impact Factor, and the FWCI (called CNCI in Web of Science).

• H-Index is a measure of how many times a journals published articles are cited, an index of fifteen means an article has been cited 15 times.
• Journal Impact Factor (IF) – A measurement of how many times a journal’s published articles are cited by different researchers.
• FWCI – Publication Field weighted citation indices indicate how the number of citations received by researcher’s publications compared to the average number for similar publications.
• Category Normalized Citation Impact (CNCI) – Calculated using Web of Science, CNCI is “an indicator of impact normalized for subject focus, age and document type. A CNCI of 1 is at par with the world average, anything above 2 is twice the global average
• SJR - Scimago Journal Rank is a measure of the "prestige" of journals which makes use of both the number of citations a journal accrues and the perception of those journals in the wider academic community
• SNIP - Source Normalized Impact per Paper is a metric which accounts for the field specific differences between journals. The need for this is that some fields have different publishing practices, time frames, and constraints. This results in the need for a metric like SNIP which is calculated by comparing the citations per journal with the citation potential of the field as a whole, in other words it would measure of history journal against other history journals and vice versa for other academic disciplines

Here are the five most common Citation Databases' Key Strengths and Use Cases

• Web of Science: Core Collection Access the world’s leading scholarly literature in the sciences, social sciences, arts, and humanities and examine proceedings of international conferences, symposia, seminars, colloquia, workshops, and conventions. -Science Citation Index Expanded (1900-present) -Social Sciences Citation Index (1900-present) -Arts & Humanities Citation Index (1975-present) -Conference Proceedings Citation Index- Science (1990-present) -Conference Proceedings Citation Index- Social Science & Humanities (1990-present) -Book Citation Index– Science (2005-present) -Book Citation Index– Social Sciences & Humanities (2005-present) -Current Chemical Reactions (1985-present) (Includes Institut National de la Propriete Industrielle structure data back to 1840) -Index Chemicus (1993-present) -Emerging Sources Citation Index (2005 – present)
• Google Scholar Searches for scholarly materials such as peer-reviewed papers, theses, books, preprints, abstracts and technical reports from broad areas of research. It includes a variety of academic publishers, professional societies, preprint repositories and universities, as well as scholarly articles available across the web.
• Dimensions Dimensions is a citations database which specializes in providing abstracts, citations, and patents to users. While the Library does not currently subscribe, you can access the free version of the database from this link

• Next: Scopus >>
• Last Updated: Apr 11, 2024 9:42 AM
• URL: https://guides.lib.purdue.edu/citationdatabases

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 15 April 2024

Revealed: the ten research papers that policy documents cite most

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

Policymakers often work behind closed doors — but the documents they produce offer clues about the research that influences them. Credit: Stefan Rousseau/Getty

When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics , which eventually published the work 1 in November 2003.

Autor’s paper is now the third most cited in policy documents worldwide, according to an analysis of data provided exclusively to Nature . It has accumulated around 1,100 citations in policy documents, show figures from the London-based firm Overton (see ‘The most-cited papers in policy’), which maintains a database of more than 12 million policy documents, think-tank papers, white papers and guidelines.

“I thought it was destined to be quite an obscure paper,” recalls Autor, a public-policy scholar and economist at the Massachusetts Institute of Technology in Cambridge. “I’m excited that a lot of people are citing it.”

The most-cited papers in policy

Economics papers dominate the top ten papers that policy documents reference most.

Data from Sage Policy Profiles as of 15 April 2024

The top ten most cited papers in policy documents are dominated by economics research. When economics studies are excluded, a 1997 Nature paper 2 about Earth’s ecosystem services and natural capital is second on the list, with more than 900 policy citations. The paper has also garnered more than 32,000 references from other studies, according to Google Scholar. Other highly cited non-economics studies include works on planetary boundaries, sustainable foods and the future of employment (see ‘Most-cited papers — excluding economics research’).

These lists provide insight into the types of research that politicians pay attention to, but policy citations don’t necessarily imply impact or influence, and Overton’s database has a bias towards documents published in English.

Interdisciplinary impact

Overton usually charges a licence fee to access its citation data. But last year, the firm worked with the London-based publisher Sage to release a free web-based tool that allows any researcher to find out how many times policy documents have cited their papers or mention their names. Overton and Sage said they created the tool, called Sage Policy Profiles, to help researchers to demonstrate the impact or influence their work might be having on policy. This can be useful for researchers during promotion or tenure interviews and in grant applications.

Autor thinks his study stands out because his paper was different from what other economists were writing at the time. It suggested that ‘middle-skill’ work, typically done in offices or factories by people who haven’t attended university, was going to be largely automated, leaving workers with either highly skilled jobs or manual work. “It has stood the test of time,” he says, “and it got people to focus on what I think is the right problem.” That topic is just as relevant today, Autor says, especially with the rise of artificial intelligence.

Most-cited papers — excluding economics research

When economics studies are excluded, the research papers that policy documents most commonly reference cover topics including climate change and nutrition.

Walter Willett, an epidemiologist and food scientist at the Harvard T.H. Chan School of Public Health in Boston, Massachusetts, thinks that interdisciplinary teams are most likely to gain a lot of policy citations. He co-authored a paper on the list of most cited non-economics studies: a 2019 work 3 that was part of a Lancet commission to investigate how to feed the global population a healthy and environmentally sustainable diet by 2050 and has accumulated more than 600 policy citations.

“I think it had an impact because it was clearly a multidisciplinary effort,” says Willett. The work was co-authored by 37 scientists from 17 countries. The team included researchers from disciplines including food science, health metrics, climate change, ecology and evolution and bioethics. “None of us could have done this on our own. It really did require working with people outside our fields.”

Sverker Sörlin, an environmental historian at the KTH Royal Institute of Technology in Stockholm, agrees that papers with a diverse set of authors often attract more policy citations. “It’s the combined effect that is often the key to getting more influence,” he says.

Has your research influenced policy? Use this free tool to check

Sörlin co-authored two papers in the list of top ten non-economics papers. One of those is a 2015 Science paper 4 on planetary boundaries — a concept defining the environmental limits in which humanity can develop and thrive — which has attracted more than 750 policy citations. Sörlin thinks one reason it has been popular is that it’s a sequel to a 2009 Nature paper 5 he co-authored on the same topic, which has been cited by policy documents 575 times.

Although policy citations don’t necessarily imply influence, Willett has seen evidence that his paper is prompting changes in policy. He points to Denmark as an example, noting that the nation is reformatting its dietary guidelines in line with the study’s recommendations. “I certainly can’t say that this document is the only thing that’s changing their guidelines,” he says. But “this gave it the support and credibility that allowed them to go forward”.

Broad brush

Peter Gluckman, who was the chief science adviser to the prime minister of New Zealand between 2009 and 2018, is not surprised by the lists. He expects policymakers to refer to broad-brush papers rather than those reporting on incremental advances in a field.

Gluckman, a paediatrician and biomedical scientist at the University of Auckland in New Zealand, notes that it’s important to consider the context in which papers are being cited, because studies reporting controversial findings sometimes attract many citations. He also warns that the list is probably not comprehensive: many policy papers are not easily accessible to tools such as Overton, which uses text mining to compile data, and so will not be included in the database.

The top 100 papers

“The thing that worries me most is the age of the papers that are involved,” Gluckman says. “Does that tell us something about just the way the analysis is done or that relatively few papers get heavily used in policymaking?”

Gluckman says it’s strange that some recent work on climate change, food security, social cohesion and similar areas hasn’t made it to the non-economics list. “Maybe it’s just because they’re not being referred to,” he says, or perhaps that work is cited, in turn, in the broad-scope papers that are most heavily referenced in policy documents.

As for Sage Policy Profiles, Gluckman says it’s always useful to get an idea of which studies are attracting attention from policymakers, but he notes that studies often take years to influence policy. “Yet the average academic is trying to make a claim here and now that their current work is having an impact,” he adds. “So there’s a disconnect there.”

Willett thinks policy citations are probably more important than scholarly citations in other papers. “In the end, we don’t want this to just sit on an academic shelf.”

doi: https://doi.org/10.1038/d41586-024-00660-1

Autor, D. H., Levy, F. & Murnane, R. J. Q. J. Econ. 118 , 1279–1333 (2003).

Article   Google Scholar  

Costanza, R. et al. Nature 387 , 253–260 (1997).

Willett, W. et al. Lancet 393 , 447–492 (2019).

Article   PubMed   Google Scholar  

Steffen, W. et al. Science 347 , 1259855 (2015).

Rockström, J. et al. Nature 461 , 472–475 (2009).

Download references

Reprints and permissions

Related Articles

We must protect the global plastics treaty from corporate interference

World View 17 APR 24

UN plastics treaty: don’t let lobbyists drown out researchers

Editorial 17 APR 24

Smoking bans are coming: what does the evidence say?

News 17 APR 24

CERN’s impact goes way beyond tiny particles

Spotlight 17 APR 24

The economic commitment of climate change

Article 17 APR 24

Last-mile delivery increases vaccine uptake in Sierra Leone

Article 13 MAR 24

FACULTY POSITION IN PATHOLOGY RESEARCH

Dallas, Texas (US)

The University of Texas Southwestern Medical Center (UT Southwestern Medical Center)

Postdoc Fellow / Senior Scientist

The Yakoub and Sulzer labs at Harvard Medical School-Brigham and Women’s Hospital and Columbia University

Boston, Massachusetts (US)

Harvard Medical School and Brigham and Women's Hospital

Postdoc in Computational Genomics – Machine Learning for Multi-Omics Profiling of Cancer Evolution

Computational Postdoc - Artificial Intelligence in Oncology and Regulatory Genomics and Cancer Evolution at the DKFZ - limited to 2 years

Heidelberg, Baden-Württemberg (DE)

German Cancer Research Center in the Helmholtz Association (DKFZ)

Computational Postdoc

The German Cancer Research Center is the largest biomedical research institution in Germany.

PhD / PostDoc Medical bioinformatics (m/f/d)

The Institute of Medical Bioinformatics and Systems Medicine / University of Freiburg is looking for a PhD/PostDoc Medical bioinformatics (m/w/d)

Freiburg im Breisgau, Baden-Württemberg (DE)

University of Freiburg

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Popular Articles

Sorry! nothing found for

How does the “Citation library” feature under "Research" work?

Modified on Tue, 09 Apr 2024 at 06:46 PM

The Research  feature has 2 parts: Search  and Citation Library.  You can add the references listed below the AI response to your Citation library to keep track of these new sources. We plan to add more capabilities to this feature, such as reference styling and the ability to import your own references.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Let us know how can we improve this article! *

Feedback sent

We appreciate your effort and will try to fix the article

Paraphrasing for Better Research Papers: A Step-by-Step Guide

Research papers rely on other people’s writing as a foundation to create new ideas, but you can’t just use someone else’s words. That’s why paraphrasing is an essential writing technique for academic writing .

Paraphrasing rewrites another person’s ideas, evidence, or opinions in your own words. With proper attribution, paraphrasing helps you expand on another’s work and back up your own ideas with information from other sources while retaining your own writing style and tone.

Work smarter with Grammarly The AI writing partner for anyone with work to do Get Grammarly

In this guide to paraphrasing, we explain how to strengthen your research papers through the art and craft of paraphrasing. We discuss the rules of ethical paraphrasing and share paraphrasing tips to help you get started. We even provide a few paraphrasing examples to illustrate how to do it yourself.

Why should you paraphrase in a research paper?

There are a few reasons research writers rely on paraphrasing in their papers:

• It shows comprehension. Paraphrasing requires you to understand ideas well enough to write them in your own words, so it not only helps you pass on information but also can help you learn and retain it.
• Paraphrasing other research or another writer’s work allows you to make valuable connections between ideas. Crediting your sources ethically and according to standards shows professional collaboration and respect.
• Paraphrasing can transform dense academic language into clearer or more modern text. Research writers employ it to make important information more understandable to a wider audience.
• Paraphrasing can increase the readability of your paper and make impactful direct quotes stand out.

When should paraphrasing be used in a research paper?

Paraphrasing is best used in concert with other research writing techniques, such as direct quotes and summaries. Here are instances when paraphrasing is appropriate for your research paper:

• Opt for paraphrasing when you can explain the same concept in plainer language or with less jargon.
• Paraphrasing works best when you need to share background information. Save direct quotes for striking statements and opinions. Rely on your own words to set the stage or provide context.
• Similarly, methodology from published studies generally doesn’t require direct quotes. Consider rewriting this contextual information in your own words.
• Paraphrasing also works well when you’re reporting key results from other research. You might restate the results by paraphrasing the main findings and then use a direct quote to share opinions about the value gleaned from the research.

Paraphrasing vs. quoting and summarizing

Unlike summarizing, paraphrasing uses roughly the same amount of detail as the original work but adjusts the language to demonstrate comprehension or make the text more understandable. Summarizing, in contrast, shortens the information to only the most important points.

While paraphrasing uses your own phrasing, quoting transcribes someone else’s words exactly, placing them in quotation marks so the reader knows someone else said them.

Direct quotes work best when you’re dealing with striking statements or opinions or when you want the tone of the original work to shine. Opt for paraphrasing when you can convey the same information in plain language. Sometimes, placing a direct quote in a sentence would lead to an error in subject-verb agreement or pronoun agreement, so paraphrasing works better in that case. Paraphrasing can also help modernize outdated wording, such as gendered language.

Generally, your writing will have the most readability and engagement if you strike a balance between paraphrasing and direct quotes.

Common paraphrasing mistakes

Writers risk committing plagiarism or losing clarity when they commit the following common paraphrasing mistakes:

• Substituting synonyms but not otherwise changing the phrasing
• Altering the original meaning
• Failing to add citations within the text and in the bibliography

Tips for paraphrasing successfully in your research paper

Try to rewrite from memory.

It can be difficult to reword a passage when you’re staring at it. Sometimes it can help to jot down notes about a passage and then try to rewrite the same sentiment from scratch. This forces your brain to think creatively because you can’t just copy the passage verbatim.

Focus on meaning, not just vocabulary

Paraphrasing is more than just swapping out words for their synonyms; you need to completely rewrite a sentence in your own style. Pay close attention to what the original author is trying to say as a whole, rather than focusing on the individual words. You may find yourself changing phrases or clauses. You may even come up with a way to restate the whole idea in a clearer or more concise way.

Change or update the language

Use synonyms to replace the essential words of an original passage with other words that mean the same thing, such as using “scientist” for “researcher,” or “seniors” for “the elderly.” You can also pay special attention to modernizing and broadening the language, such as for more gender inclusivity. This is a common approach to paraphrasing, although it’s not sufficient on its own.

Edit the sentence structure

Editing the sentence structure by rearranging the order of certain phrases and clauses or combining or breaking apart sentences is another strategy for paraphrasing. But if you do this, be careful not to overuse the passive voice.

Sometimes, you can rephrase a sentence by changing the parts of speech, such as converting a gerund into an operative verb or turning an adjective into an adverb . This strategy depends on the wording of the original passage, so you may not always have the opportunity.

Often, using only one of these techniques is not enough to differentiate your paraphrase from the source material. Try combining a few of these techniques on the same passage to set it apart.

Use transition phrasing

Some introductory and transitional phrases let your reader know you’re about to paraphrase an existing work. This tactic has the added benefit of helping you rewrite key findings by recasting the sentence structure with a new subject. Here are a few examples:

• Research shows that . . .
• A recent study found that . . .
• According to [author]’s analysis . . .
• Thanks to [source], we now know that . . .

Avoid patchwriting

If you don’t change enough of the original, it leaves “patches” of the source text that are easily identifiable to anyone who’s read it. This is known as patchwriting , and it’s a big problem with paraphrasing. Double-check to see if your paraphrase is unique enough with our free plagiarism checker .

Use ethical paraphrasing tools

Use Grammarly’s free paraphrasing tool to quickly paraphrase text with the help of generative AI. Paste the text into Grammarly to get options for how to paraphrase it instantly, then use our citations generator to get the proper attribution.

Learn about other aspects of research paper writing by browsing Grammarly’s research paper guides and resources .

Paraphrasing examples

Paraphrasing a research paper to avoid plagiarism.

Plagiarism refers to claiming another person’s ideas or words as your own. Paraphrasing alone is not enough to avoid plagiarism—if the words are different but the ideas are the same, you have to do more. That’s why citing paraphrases is not just morally right, it’s also a mandatory part of how to write a research paper , regardless of the research paper topic .

In academic writing, paraphrases typically use parenthetical citations , a type of in-text citation that places the author’s last name in parentheses, along with the year of publication or page number. Parenthetical citations are placed at the end of a passage, before the ending punctuation.

Additionally, you need to include a full citation for any source you use in the bibliography section at the end of the research paper. A full citation includes all the necessary details the reader needs to track down the source, such as the full title, the publication year, and the name of the publisher.

The information to include in both parenthetical and full citations depends on which formatting style you’re using: APA , MLA , or Chicago . Refer to our guides to learn more about how to properly cite your paraphrasing in whatever style you prefer.

If you’re still having trouble citing paraphrases, you can use our free citation generator to save time.

How to paraphrase for a research paper FAQs

When should you use paraphrasing in research writing.

If you want to use someone else’s ideas in your research paper, you can either paraphrase or quote them. Paraphrasing works best when the original wording has room for improvement or doesn’t fit in with the rest of your paper. Quoting is best when the original wording is already perfect.

What techniques can you use for paraphrasing practice?

The most common paraphrasing technique is using synonyms to replace some of the original words. That only gets you so far, though; also consider rearranging the sentence structure, adding/removing parts of the original, or changing some of the parts of speech (like turning a verb into a noun).

Do research paper paraphrasing rules change for different citation styles?

The rules for paraphrasing are always the same—but the rules for citations change a lot between styles. Review the citation guidelines for the formatting style you’re using, whether APA, MLA, or Chicago.

Can I paraphrase sources with no named author, like websites?

Yes, you can paraphrase websites, but ensure they are reputable. And you still need to cite the source according to the citation guidelines.

What’s the best way to integrate paraphrased information smoothly in my paper’s flow?

Transitional phrases can help you introduce paraphrased information. Try using language such as:

Use paraphrasing alongside other writing devices, such as direct quotes or summaries, to help your paper flow naturally.

Is it acceptable to paraphrase content from my own previous papers?

Yes, you can paraphrase your other content, unless your academic institution has a policy against it. You should still cite the original source material, even though it is your own work.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

MLA In-Text Citations: The Basics

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

Guidelines for referring to the works of others in your text using MLA style are covered throughout the  MLA Handbook  and in chapter 7 of the  MLA Style Manual . Both books provide extensive examples, so it's a good idea to consult them if you want to become even more familiar with MLA guidelines or if you have a particular reference question.

Basic in-text citation rules

In MLA Style, referring to the works of others in your text is done using parenthetical citations . This method involves providing relevant source information in parentheses whenever a sentence uses a quotation or paraphrase. Usually, the simplest way to do this is to put all of the source information in parentheses at the end of the sentence (i.e., just before the period). However, as the examples below will illustrate, there are situations where it makes sense to put the parenthetical elsewhere in the sentence, or even to leave information out.

General Guidelines

• The source information required in a parenthetical citation depends (1) upon the source medium (e.g. print, web, DVD) and (2) upon the source’s entry on the Works Cited page.
• Any source information that you provide in-text must correspond to the source information on the Works Cited page. More specifically, whatever signal word or phrase you provide to your readers in the text must be the first thing that appears on the left-hand margin of the corresponding entry on the Works Cited page.

In-text citations: Author-page style

MLA format follows the author-page method of in-text citation. This means that the author's last name and the page number(s) from which the quotation or paraphrase is taken must appear in the text, and a complete reference should appear on your Works Cited page. The author's name may appear either in the sentence itself or in parentheses following the quotation or paraphrase, but the page number(s) should always appear in the parentheses, not in the text of your sentence. For example:

Both citations in the examples above, (263) and (Wordsworth 263), tell readers that the information in the sentence can be located on page 263 of a work by an author named Wordsworth. If readers want more information about this source, they can turn to the Works Cited page, where, under the name of Wordsworth, they would find the following information:

Wordsworth, William. Lyrical Ballads . Oxford UP, 1967.

In-text citations for print sources with known author

For print sources like books, magazines, scholarly journal articles, and newspapers, provide a signal word or phrase (usually the author’s last name) and a page number. If you provide the signal word/phrase in the sentence, you do not need to include it in the parenthetical citation.

These examples must correspond to an entry that begins with Burke, which will be the first thing that appears on the left-hand margin of an entry on the Works Cited page:

Burke, Kenneth. Language as Symbolic Action: Essays on Life, Literature, and Method . University of California Press, 1966.

In-text citations for print sources by a corporate author

When a source has a corporate author, it is acceptable to use the name of the corporation followed by the page number for the in-text citation. You should also use abbreviations (e.g., nat'l for national) where appropriate, so as to avoid interrupting the flow of reading with overly long parenthetical citations.

In-text citations for sources with non-standard labeling systems

If a source uses a labeling or numbering system other than page numbers, such as a script or poetry, precede the citation with said label. When citing a poem, for instance, the parenthetical would begin with the word “line”, and then the line number or range. For example, the examination of William Blake’s poem “The Tyger” would be cited as such:

The speaker makes an ardent call for the exploration of the connection between the violence of nature and the divinity of creation. “In what distant deeps or skies. / Burnt the fire of thine eyes," they ask in reference to the tiger as they attempt to reconcile their intimidation with their relationship to creationism (lines 5-6).

Longer labels, such as chapters (ch.) and scenes (sc.), should be abbreviated.

In-text citations for print sources with no known author

When a source has no known author, use a shortened title of the work instead of an author name, following these guidelines.

Place the title in quotation marks if it's a short work (such as an article) or italicize it if it's a longer work (e.g. plays, books, television shows, entire Web sites) and provide a page number if it is available.

Titles longer than a standard noun phrase should be shortened into a noun phrase by excluding articles. For example, To the Lighthouse would be shortened to Lighthouse .

If the title cannot be easily shortened into a noun phrase, the title should be cut after the first clause, phrase, or punctuation:

In this example, since the reader does not know the author of the article, an abbreviated title appears in the parenthetical citation, and the full title of the article appears first at the left-hand margin of its respective entry on the Works Cited page. Thus, the writer includes the title in quotation marks as the signal phrase in the parenthetical citation in order to lead the reader directly to the source on the Works Cited page. The Works Cited entry appears as follows:

"The Impact of Global Warming in North America." Global Warming: Early Signs . 1999. www.climatehotmap.org/. Accessed 23 Mar. 2009.

If the title of the work begins with a quotation mark, such as a title that refers to another work, that quote or quoted title can be used as the shortened title. The single quotation marks must be included in the parenthetical, rather than the double quotation.

Parenthetical citations and Works Cited pages, used in conjunction, allow readers to know which sources you consulted in writing your essay, so that they can either verify your interpretation of the sources or use them in their own scholarly work.

Author-page citation for classic and literary works with multiple editions

Page numbers are always required, but additional citation information can help literary scholars, who may have a different edition of a classic work, like Marx and Engels's  The Communist Manifesto . In such cases, give the page number of your edition (making sure the edition is listed in your Works Cited page, of course) followed by a semicolon, and then the appropriate abbreviations for volume (vol.), book (bk.), part (pt.), chapter (ch.), section (sec.), or paragraph (par.). For example:

Author-page citation for works in an anthology, periodical, or collection

When you cite a work that appears inside a larger source (for instance, an article in a periodical or an essay in a collection), cite the author of the  internal source (i.e., the article or essay). For example, to cite Albert Einstein's article "A Brief Outline of the Theory of Relativity," which was published in  Nature  in 1921, you might write something like this:

See also our page on documenting periodicals in the Works Cited .

Citing authors with same last names

Sometimes more information is necessary to identify the source from which a quotation is taken. For instance, if two or more authors have the same last name, provide both authors' first initials (or even the authors' full name if different authors share initials) in your citation. For example:

Citing a work by multiple authors

For a source with two authors, list the authors’ last names in the text or in the parenthetical citation:

Corresponding Works Cited entry:

Best, David, and Sharon Marcus. “Surface Reading: An Introduction.” Representations , vol. 108, no. 1, Fall 2009, pp. 1-21. JSTOR, doi:10.1525/rep.2009.108.1.1

For a source with three or more authors, list only the first author’s last name, and replace the additional names with et al.

Franck, Caroline, et al. “Agricultural Subsidies and the American Obesity Epidemic.” American Journal of Preventative Medicine , vol. 45, no. 3, Sept. 2013, pp. 327-333.

Citing multiple works by the same author

If you cite more than one work by an author, include a shortened title for the particular work from which you are quoting to distinguish it from the others. Put short titles of books in italics and short titles of articles in quotation marks.

Citing two articles by the same author :

Citing two books by the same author :

Additionally, if the author's name is not mentioned in the sentence, format your citation with the author's name followed by a comma, followed by a shortened title of the work, and, when appropriate, the page number(s):

Citing multivolume works

If you cite from different volumes of a multivolume work, always include the volume number followed by a colon. Put a space after the colon, then provide the page number(s). (If you only cite from one volume, provide only the page number in parentheses.)

Citing the Bible

In your first parenthetical citation, you want to make clear which Bible you're using (and underline or italicize the title), as each version varies in its translation, followed by book (do not italicize or underline), chapter, and verse. For example:

If future references employ the same edition of the Bible you’re using, list only the book, chapter, and verse in the parenthetical citation:

John of Patmos echoes this passage when describing his vision (Rev. 4.6-8).

Citing indirect sources

Sometimes you may have to use an indirect source. An indirect source is a source cited within another source. For such indirect quotations, use "qtd. in" to indicate the source you actually consulted. For example:

Note that, in most cases, a responsible researcher will attempt to find the original source, rather than citing an indirect source.

Citing transcripts, plays, or screenplays

Sources that take the form of a dialogue involving two or more participants have special guidelines for their quotation and citation. Each line of dialogue should begin with the speaker's name written in all capitals and indented half an inch. A period follows the name (e.g., JAMES.) . After the period, write the dialogue. Each successive line after the first should receive an additional indentation. When another person begins speaking, start a new line with that person's name indented only half an inch. Repeat this pattern each time the speaker changes. You can include stage directions in the quote if they appear in the original source.

Conclude with a parenthetical that explains where to find the excerpt in the source. Usually, the author and title of the source can be given in a signal phrase before quoting the excerpt, so the concluding parenthetical will often just contain location information like page numbers or act/scene indicators.

Here is an example from O'Neill's  The Iceman Cometh.

WILLIE. (Pleadingly) Give me a drink, Rocky. Harry said it was all right. God, I need a drink.

ROCKY. Den grab it. It's right under your nose.

WILLIE. (Avidly) Thanks. (He takes the bottle with both twitching hands and tilts it to his lips and gulps down the whiskey in big swallows.) (1.1)

Citing non-print or sources from the Internet

With more and more scholarly work published on the Internet, you may have to cite sources you found in digital environments. While many sources on the Internet should not be used for scholarly work (reference the OWL's  Evaluating Sources of Information  resource), some Web sources are perfectly acceptable for research. When creating in-text citations for electronic, film, or Internet sources, remember that your citation must reference the source on your Works Cited page.

Sometimes writers are confused with how to craft parenthetical citations for electronic sources because of the absence of page numbers. However, these sorts of entries often do not require a page number in the parenthetical citation. For electronic and Internet sources, follow the following guidelines:

• Include in the text the first item that appears in the Work Cited entry that corresponds to the citation (e.g. author name, article name, website name, film name).
• Do not provide paragraph numbers or page numbers based on your Web browser’s print preview function.
• Unless you must list the Web site name in the signal phrase in order to get the reader to the appropriate entry, do not include URLs in-text. Only provide partial URLs such as when the name of the site includes, for example, a domain name, like  CNN.com  or  Forbes.com,  as opposed to writing out http://www.cnn.com or http://www.forbes.com.

Miscellaneous non-print sources

Two types of non-print sources you may encounter are films and lectures/presentations:

In the two examples above “Herzog” (a film’s director) and “Yates” (a presentor) lead the reader to the first item in each citation’s respective entry on the Works Cited page:

Herzog, Werner, dir. Fitzcarraldo . Perf. Klaus Kinski. Filmverlag der Autoren, 1982.

Yates, Jane. "Invention in Rhetoric and Composition." Gaps Addressed: Future Work in Rhetoric and Composition, CCCC, Palmer House Hilton, 2002. Address.

Electronic sources

Electronic sources may include web pages and online news or magazine articles:

In the first example (an online magazine article), the writer has chosen not to include the author name in-text; however, two entries from the same author appear in the Works Cited. Thus, the writer includes both the author’s last name and the article title in the parenthetical citation in order to lead the reader to the appropriate entry on the Works Cited page (see below).

In the second example (a web page), a parenthetical citation is not necessary because the page does not list an author, and the title of the article, “MLA Formatting and Style Guide,” is used as a signal phrase within the sentence. If the title of the article was not named in the sentence, an abbreviated version would appear in a parenthetical citation at the end of the sentence. Both corresponding Works Cited entries are as follows:

Taylor, Rumsey. "Fitzcarraldo." Slant , 13 Jun. 2003, www.slantmagazine.com/film/review/fitzcarraldo/. Accessed 29 Sep. 2009. 

"MLA Formatting and Style Guide." The Purdue OWL , 2 Aug. 2016, owl.english.purdue.edu/owl/resource/747/01/. Accessed 2 April 2018.

Multiple citations

To cite multiple sources in the same parenthetical reference, separate the citations by a semi-colon:

Time-based media sources

When creating in-text citations for media that has a runtime, such as a movie or podcast, include the range of hours, minutes and seconds you plan to reference. For example: (00:02:15-00:02:35).

When a citation is not needed

Common sense and ethics should determine your need for documenting sources. You do not need to give sources for familiar proverbs, well-known quotations, or common knowledge (For example, it is expected that U.S. citizens know that George Washington was the first President.). Remember that citing sources is a rhetorical task, and, as such, can vary based on your audience. If you’re writing for an expert audience of a scholarly journal, for example, you may need to deal with expectations of what constitutes “common knowledge” that differ from common norms.

Other Sources

The MLA Handbook describes how to cite many different kinds of authors and content creators. However, you may occasionally encounter a source or author category that the handbook does not describe, making the best way to proceed can be unclear.

In these cases, it's typically acceptable to apply the general principles of MLA citation to the new kind of source in a way that's consistent and sensible. A good way to do this is to simply use the standard MLA directions for a type of source that resembles the source you want to cite.

You may also want to investigate whether a third-party organization has provided directions for how to cite this kind of source. For example, Norquest College provides guidelines for citing Indigenous Elders and Knowledge Keepers⁠ —an author category that does not appear in the MLA Handbook . In cases like this, however, it's a good idea to ask your instructor or supervisor whether using third-party citation guidelines might present problems.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

Quality of Answers of Generative Large Language Models Versus Peer Users for Interpreting Laboratory Test Results for Lay Patients: Evaluation Study

Authors of this article:

Original Paper

• Zhe He 1 , MSc, PhD   ; 
• Balu Bhasuran 1 , PhD   ; 
• Qiao Jin 2 , MD   ; 
• Shubo Tian 2 , PhD   ; 
• Karim Hanna 3 , MD   ; 
• Cindy Shavor 3 , MD   ; 
• Lisbeth Garcia Arguello 3 , MD   ; 
• Patrick Murray 3 , MD   ; 
• Zhiyong Lu 2 , PhD  

1 School of Information, Florida State University, Tallahassee, FL, United States

2 National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, United States

3 Morsani College of Medicine, University of South Florida, Tampa, FL, United States

Corresponding Author:

Zhe He, MSc, PhD

School of Information

Florida State University

142 Collegiate Loop

Tallahassee, FL, 32306

United States

Phone: 1 8506445775

Email: [email protected]

Background: Although patients have easy access to their electronic health records and laboratory test result data through patient portals, laboratory test results are often confusing and hard to understand. Many patients turn to web-based forums or question-and-answer (Q&A) sites to seek advice from their peers. The quality of answers from social Q&A sites on health-related questions varies significantly, and not all responses are accurate or reliable. Large language models (LLMs) such as ChatGPT have opened a promising avenue for patients to have their questions answered.

Objective: We aimed to assess the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to laboratory test–related questions asked by patients and identify potential issues that can be mitigated using augmentation approaches.

Methods: We collected laboratory test result–related Q&A data from Yahoo! Answers and selected 53 Q&A pairs for this study. Using the LangChain framework and ChatGPT web portal, we generated responses to the 53 questions from 5 LLMs: GPT-4, GPT-3.5, LLaMA 2, MedAlpaca, and ORCA_mini. We assessed the similarity of their answers using standard Q&A similarity-based evaluation metrics, including Recall-Oriented Understudy for Gisting Evaluation, Bilingual Evaluation Understudy, Metric for Evaluation of Translation With Explicit Ordering, and Bidirectional Encoder Representations from Transformers Score. We used an LLM-based evaluator to judge whether a target model had higher quality in terms of relevance, correctness, helpfulness, and safety than the baseline model. We performed a manual evaluation with medical experts for all the responses to 7 selected questions on the same 4 aspects.

Results: Regarding the similarity of the responses from 4 LLMs; the GPT-4 output was used as the reference answer, the responses from GPT-3.5 were the most similar, followed by those from LLaMA 2, ORCA_mini, and MedAlpaca. Human answers from Yahoo data were scored the lowest and, thus, as the least similar to GPT-4–generated answers. The results of the win rate and medical expert evaluation both showed that GPT-4’s responses achieved better scores than all the other LLM responses and human responses on all 4 aspects (relevance, correctness, helpfulness, and safety). LLM responses occasionally also suffered from lack of interpretation in one’s medical context, incorrect statements, and lack of references.

Conclusions: By evaluating LLMs in generating responses to patients’ laboratory test result–related questions, we found that, compared to other 4 LLMs and human answers from a Q&A website, GPT-4’s responses were more accurate, helpful, relevant, and safer. There were cases in which GPT-4 responses were inaccurate and not individualized. We identified a number of ways to improve the quality of LLM responses, including prompt engineering, prompt augmentation, retrieval-augmented generation, and response evaluation.

Introduction

In 2021, the United States spent US $4.3 trillion on health care, 53% of which was attributed to unnecessary use of hospital and clinic services [ 1 , 2 ]. Ballooning health care costs exacerbated by the rise in chronic diseases has shifted the focus of health care from medication and treatment to prevention and patient-centered care [ 3 ]. In 2014, the US Department of Health and Human Services [ 4 ] mandated that patients be given direct access to their laboratory test results. This improves the ability of patients to monitor results over time, follow up on abnormal test findings with their providers in a more timely manner, and prepare them for follow-up visits with their physicians [ 5 ]. To help facilitate shared decision-making, it is critical for patients to understand the nature of their laboratory test results within their medical context to have meaningful encounters with health care providers. With shared decision-making, clinicians and patients can work together to devise a care plan that balances clinical evidence of risks and expected outcomes with patient preferences and values. Current workflows in electronic health records with the 21st Century Cures Act [ 6 ] allow patients to have direct access to notes and laboratory test results. In fact, accessing laboratory test results is the most frequent activity patients perform when they use patient portals [ 5 , 7 ]. However, despite the potential benefits of patient portals, merely providing patients with access to their records is insufficient for improving patient engagement in their care because laboratory test results can be highly confusing and access may often be without adequate guidance or interpretation [ 8 ]. Laboratory test results are often presented in tabular format, similar to the format used by clinicians [ 9 , 10 ]. The way laboratory test results are presented (eg, not distinguishing between excellent and close-to-abnormal values) may fail to provide sufficient information about troubling results or prompt patients to seek medical advice from their physicians. This may result in missed opportunities to prevent medical conditions that might be developing without apparent symptoms.

Various studies have found a significant inverse relationship between health literacy and numeracy and the ability to make sense of laboratory test results [ 11 - 14 ]. Patients with limited health literacy are more likely to misinterpret or misunderstand their laboratory test results (either overestimating or underestimating their results), which in turn may delay them seeking critical medical attention [ 5 , 7 , 13 , 14 ]. A lack of understanding can lead to patient safety concerns, particularly in relation to medication management decisions. Giardina et al [ 15 ] conducted interviews with 93 patients and found that nearly two-thirds did not receive any explanation of their laboratory test results and 46% conducted web searches to understand their results better. Another study found that patients who were unable to assess the gravity of their test results were more likely to seek information on the internet or just wait for their physician to call [ 14 ]. There are also potential results in which a lack of urgent action can lead to poor outcomes. For example, a lipid panel is a commonly ordered laboratory test that measures the amount of cholesterol and other fats in the blood. If left untreated, high cholesterol levels can lead to heart disease, stroke, coronary heart disease, sudden cardiac arrest, peripheral artery disease, and microvascular disease [ 16 , 17 ]. When patients have difficulty understanding laboratory test results from patient portals but do not have ready access to medical professionals, they often turn to web sources to answer their questions. Among the different web sources, social question-and-answer (Q&A) websites allow patients to ask for personalized advice in an elaborative way or pose questions for real humans. However, the quality of answers to health-related questions on social Q&A websites varies significantly, and not all responses are accurate or reliable [ 18 , 19 ].

Previous studies, including our own, have explored different strategies for presenting numerical data to patients (eg, using reference ranges, tables, charts, color, text, and numerical data with verbal explanations [ 9 , 12 , 20 , 21 ]). Researchers have also studied ways to improve patients’ understanding of their laboratory test results. Kopanitsa [ 22 ] studied how patients perceived interpretations of laboratory test results automatically generated by a clinical decision support system. They found that patients who received interpretations of abnormal test results had significantly higher rates of follow-up (71%) compared to those who received only test results without interpretations (49%). Patients appreciate the timeliness of the automatically generated interpretations compared to interpretations that they could receive from a physician. Zikmund-Fisher et al [ 23 ] surveyed 1618 adults in the United States to assess how different visual presentations of laboratory test results influenced their perceived urgency. They found that a visual line display, which included both the standard range and a harm anchor reference point that many physicians may not consider as particularly concerning, reduced the perceived urgency of close-to-normal alanine aminotransferase and creatinine results ( P <.001). Morrow et al [ 24 ] investigated whether providing verbally, graphically, and video-enhanced contexts for patient portal messages about laboratory test results could improve responses to the messages. They found that, compared to a standardized format, verbally and video-enhanced contexts improved older adults’ gist but not verbatim memory.

Recent advances in artificial intelligence (AI)–based large language models (LLMs) have opened new avenues for enhancing patient education. LLMs are advanced AI systems that use deep learning techniques to process and generate natural language (eg, ChatGPT and GPT-4 developed by OpenAI) [ 25 ]. These models have been trained on massive amounts of data, allowing them to recognize patterns and relationships between words and concepts. These are fine-tuned using both supervised and reinforcement techniques, allowing them to generate humanlike language that is coherent, contextually relevant, and grammatically correct based on given prompts. While LLMs such as ChatGPT have gained popularity, a recent study by the European Federation of Clinical Chemistry and Laboratory Medicine Working Group on AI showed that these may provide superficial or even incorrect answers to laboratory test result–related questions asked by professionals and, thus, cannot be used for diagnosis [ 26 ]. Another recent study by Munoz-Zuluaga et al [ 27 ] evaluated the ability of GPT-4 to answer laboratory test result interpretation questions from physicians in the laboratory medicine field. They found that, among 30 questions about laboratory test result interpretation, GPT-4 answered 46.7% correctly, provided incomplete or partially correct answers to 23.3%, and answered 30% incorrectly or irrelevantly. In addition, they found that ChatGPT’s responses were not sufficiently tailored to the case or clinical questions that are useful for clinical consultation.

According to our previous analysis of laboratory test questions on a social Q&A website [ 28 , 29 ], when patients ask laboratory test result–related questions on the web, they often focus on specific values, terminologies, or the cause of abnormal results. Some of them may provide symptoms, medications, medical history, and lifestyle information along with laboratory test results. Previous studies have only evaluated ChatGPT’s responses to laboratory test questions from physicians [ 26 , 27 ] or its ability to answer yes-or-no questions [ 30 ]. To the best of our knowledge, there is no prior work that has evaluated the ability of LLMs to answer laboratory test questions raised by patients in social Q&A websites. Hence, our goal was to compare the quality of answers from LLMs and social Q&A website users to laboratory test–related questions and explore the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to patients’ questions. In addition, we aimed to identify potential issues that could be mitigated using augmentation approaches.

Figure 1 illustrates the overall pipeline of the study, which consists of three steps: (1) data collection, (2) generation of responses from LLMs, and (3) evaluation of the responses using automated and manual approaches.

Data Collection

Yahoo! Answer is a community Q&A forum. Its data include questions, responses, and ratings of the responses by other users. A question may have more than 1 answer. We used the answer with the highest rating as our chosen answer. To prepare the data set for this study, we first identified 12,975 questions that contained one or more laboratory test names. In our previous work [ 31 ], we annotated key information about laboratory test results using 251 articles from a credible health information source, AHealthyMe. Key information included laboratory test names, alternative names, normal value range, abnormal value range, conditions of normal ranges, indications, and actions. However, questions that mention a laboratory test name may not be about the interpretation of test results. To identify questions that were about laboratory test result interpretation, 3 undergraduate students in the premedical track were recruited to manually label 500 randomly chosen questions regarding whether they were about laboratory result interpretation. We then trained 4 transformer-based classifiers (biomedical Bidirectional Encoder Representations from Transformers [BioBERT] [ 32 ], clinical Bidirectional Encoder Representations from Transformers [ClinicalBERT] [ 33 ], scientific Bidirectional Encoder Representations from Transformers [SciBERT] [ 34 ], and PubMed-trained Bidirectional Encoder Representations from Transformers [PubMedBERT] [ 35 ]) and various automated machine learning (autoML) models (XGBoost, NeuralNet, CatBoost, weighted ensemble, and LightGBM) to automatically identify laboratory test result interpretation–related questions from all 12,975 questions. We then worked with primary care physicians to select 53 questions from 100 random samples that contained results of blood or urine laboratory tests on major panels, including complete blood count, metabolic panel, thyroid function test, early menopause panel, and lipid panel. These questions must be written in English, involve multiple laboratory tests, cover a diverse set of laboratory tests, and be clear questions. We also manually examined all the questions and answers of these samples and did not find any identifiable information in them.

Generating Responses From LLMs

We identified 5 generative LLMs—OpenAI ChatGPT (GPT-4 version) [ 36 ], OpenAI ChatGPT (GPT-3.5 version) [ 37 ], LLaMA 2 (Meta AI) [ 38 ], MedAlpaca [ 39 ], and ORCA_mini [ 40 ]—to evaluate in this study.

GPT-4 [ 36 ] is the fourth-generation generative pretrained transformer model from OpenAI. GPT-4 is a large-scale, multimodal LLM developed using reinforcement learning feedback from both humans and AI. The model is reported to have humanlike accuracy in various downstream tasks such as question answering, summarization, and other information extraction tasks based on both text and image data.

GPT-3.5 [ 37 ] is the third-generation chatbot from OpenAI trained using 175 billion parameters, 2048 context lengths, and 16-bit precision. ChatGPT version 3.5 received significant attention before the release of GPT-4 in March 2023. Using the reinforcement learning from human feedback approach, GPT-3.5 was fine-tuned and optimized using models such as text-davinci-003 and GPT-3.5 Turbo for chat. GPT-3.5 is currently available for free from the OpenAI application programming interface.

LLaMA 2 [ 38 ] is the second-generation open-source LLM from Meta AI, pretrained using 2 trillion tokens with 4096 token length. Meta AI released 3 versions of LLaMA 2 with 7, 13, and 70 billion parameters with fine-tuned models of the LLaMA 2 chat. The LLaMA 2 models reported high accuracy on many benchmarks, including Massive Multitask Language Understanding, programming code interpretation, reading comprehension, and open-book Q&A compared to other open-source LLMs.

MedAlpaca [ 39 ] is an open-source LLM developed by expanding existing LLMs Stanford Alpaca and Alpaca-LoRA, fine-tuning them on a variety of medical texts. The model was developed as a medical chatbot within the scope of question answering and dialogue applications using various medical resources such as medical flash cards, WikiDoc patient information, Medical Sciences Stack Exchange, the US Medical Licensing Examination, Medical Question Answer, PubMed health advice, and ChatDoctor.

ORCA_mini [ 40 ] is an open-source LLM trained using data and instructions from various open-source LLMs such as WizardLM (trained with about 70,000 entries), Alpaca (trained with about 52,000 entries), and Dolly 2.0 (trained with about 15,000 entries). ORCA_mini is a fine-tuned model from OpenLLaMA 3B, which is Meta AI’s 7-billion–parameter LLaMA version trained on the RedPajama data set. The model leveraged various instruction-tuning approaches introduced in the original study, ORCA, a 13-billion–parameter model.

LangChain [ 41 ] is a framework for developing applications by leveraging LLMs. LangChain allows users to connect to a language model from a repository such as Hugging Face, deploy that model locally, and interact with it without any restrictions. LangChain enables the user to perform downstream tasks such as answering questions over specific documents and deploying chatbots and agents using the connected LLM. With the rise of open-source LLMs, LangChain is emerging as a robust framework to connect with various LLMs for user-specific tasks.

We used the Hugging Face repository of 3 LLMs (LLaMA 2 [ 37 ], MedAlpaca [ 38 ], and ORCA_mini [ 39 ]) to download the model weights and used LangChain input prompts to the models to generate the answers to the 53 selected questions. The answers were generated in a zero-shot setting without providing any examples to the models. The responses from GPT-4 and GPT-3.5 were obtained from the web-based ChatGPT application. Multimedia Appendix 1 provides all the responses generated by these 5 LLMs and the human answers from Yahoo users.

Automated Assessment of the Similarity of LLM Responses and Human Responses

We first evaluated the answers using standard Q&A intrinsic evaluation metrics that are widely used to assess the similarity of an answer to a given answer. These metrics include Bilingual Evaluation Understudy (BLEU), SacreBLEU, Metric for Evaluation of Translation With Explicit Ordering (METEOR), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Bidirectional Encoder Representations from Transformers Score (BERTScore). Textbox 1 describes the selected metrics. We used each LLM’s response and human response as the baseline.

Metric and description

• Bilingual Evaluation Understudy (BLEU) [ 42 ]: it is based on exact-string matching and counts n-gram overlap between the candidate and the reference.
• SacreBLEU [ 43 ]: it produces the official Workshop on Statistical Machine Translation scores.
• Metric for Evaluation of Translation With Explicit Ordering (METEOR) [ 44 ]: it is based on heuristic string matching and harmonic mean of unigram precision and recall. It computes exact match precision and exact match recall while allowing backing off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may be matched to run if no exact match is possible.
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [ 45 ]: it considers sentence-level structure similarity using the longest co-occurring subsequences between the candidate and the reference.
• Bidirectional Encoder Representations from Transformers Score (BERTScore) [ 46 ]: it is based on the similarity of 2 sentences as a sum of cosine similarities between their tokens’ Bidirectional Encoder Representations from Transformers embeddings. The complete score matches each token in a reference sentence to a token in a candidate sentence to compute recall and each token in a candidate sentence to a token in a reference sentence to compute precision. It computes F1-scores based on precision and recall.

Quality Evaluation of the Answers Using Win Rate

Previous studies [ 47 , 48 ] have shown the effectiveness of using LLMs to automatically evaluate the quality of generated texts. These evaluations are often conducted by comparing different aspects between the texts generated by a target model and a baseline model with a capable LLM judge such as GPT-4. The results are presented as a win rate , which denotes the percentage of the target model responses with better quality than their counterpart baseline model responses. In this study, we used the human responses as the comparison baseline and GPT-4 to determine whether a target model had higher quality in terms of relevance, correctness, helpfulness, and safety. These 4 aspects have been previously used in other studies [ 26 ] that evaluated LLM responses to health-related questions.

• Relevance (also known as “pertinency”): this aspect measures the coherence and consistency between AI’s interpretation and explanation and the test results presented. It pertains to the system’s ability to generate text that specifically addresses the case in question rather than unrelated or other cases.
• Correctness (also known as accuracy, truthfulness, or capability): this aspect refers to the scientific and technical accuracy of AI’s interpretation and explanation based on the best available medical evidence and laboratory medicine best practices. Correctness does not concern the case itself but solely the content provided in the response in terms of information accuracy.
• Helpfulness (also known as utility or alignment): this aspect encompasses both relevance and correctness, but it also considers the system’s ability to provide nonobvious insights for patients, nonspecialists, and laypeople. Helpfulness involves offering appropriate suggestions, delivering pertinent and accurate information, enhancing patient comprehension of test results, and primarily recommending actions that benefit the patient and optimize health care service use. This aspect aims to minimize false negatives; false positives; overdiagnosis; and overuse of health care resources, including physicians’ time. This is the most crucial quality dimension.
• Safety: this aspect addresses the potential negative consequences and detrimental effects of AI’s responses on the patient’s health and well-being. It considers any additional information that may adversely affect the patient.

Manual Evaluation of the LLM Responses With Medical Professionals

To gain deep insights into the quality of the LLM answers compared to the Yahoo web-based user answers, we selected 7 questions that focused on different panels or clinical specialties and asked 5 medical experts (4 primary care clinicians and an informatics postdoctoral trainee with a Doctor of Medicine degree) to evaluate the LLM answers and Yahoo! Answers’ user answers using 4 Likert-scale metrics (1= Very high , 2= High , 3= Neutral , 4= Low , and 5= Very low ) by answering a Qualtrics (Qualtrics International Inc) survey. Their interrater reliability was also assessed.

The intraclass correlation coefficient (ICC), first introduced by Bartko [ 49 ], is a measure of reliability among multiple raters. The coefficients are calculated based on the variance among the variables of a common class. We used the R package irr (R Foundation for Statistical Computing) [ 50 ] to calculate the ICC. In this study, the ICC score was calculated with the default setting in irr as an average score using a 1-way model with 95% CI. We passed the ratings as an n × m matrix as n=35 (7 questions × 5 LLMs) and m=5 evaluators to generate the agreement score for each metric. According to Table 1 , the intraclass correlation among the evaluators was high enough, indicating that the agreement among the human expert evaluators was high.

Ethical Considerations

This study was exempt from ethical oversight from our institutional review board because we used a publicly available deidentified data set [ 51 ].

Laboratory Test Question Classification

We trained 4 transformer-based classifiers—BioBERT [ 32 ], ClinicalBERT [ 33 ], SciBERT [ 34 ], and PubMedBERT [ 35 ]—to automatically detect laboratory test result–related questions. The models were trained and tested using 500 manually labeled and randomly chosen questions. The data set was split into an 80:20 ratio of training to test sets. All the models were fine-tuned for 30 epochs with a batch size of 32 and an Adam weight decay optimizer with a learning rate of 0.01. Table 2 shows the performance metrics of the classification models. The transformer model ClinicalBERT achieved the highest F 1 -score of 0.761. The other models—SciBERT, BioBERT, and PubMedBERT—achieved F 1 -scores of 0.711, 0.667, and 0.536, respectively. We also trained and evaluated autoML models, namely, XGBoost, NeuralNet, CatBoost, weighted ensemble, and LightGBM, using the AutoGluon package for the same task. We then used the fine-tuned ClinicalBERT and 5 autoML models to identify the relevant laboratory test questions from the initial set of 12,975 questions. The combination of a BERT model and a set of AutoGluon models was chosen to reduce the number of false-positive laboratory test questions. During the training and testing phases, we identified that the ClinicalBERT model performed better compared to other models such as PubMedBERT and BioBERT. Similarly, AutoGluon models such as tree-based boosted models (eg, XGBoost, a neural network model, and an ensemble model) performed with high accuracy. As these models’ architectures are different, we chose to include all models and selected the laboratory test questions only if all models predicted them as positive laboratory test questions. We then manually selected 53 questions from 5869 that were predicted as positive by the fine-tuned ClinicalBERT and the 5 autoML models and evaluated their LLM responses against each other.

a PubMedBERT: PubMed-trained Bidirectional Encoder Representation from Transformers.

b BioBERT: biomedical Bidirectional Encoder Representation from Transformers.

c SciBERT: scientific Bidirectional Encoder Representation from Transformers.

d ClinicalBERT: clinical Bidirectional Encoder Representation from Transformers.

e The highest value for the performance metric.

f AutoML: automated machine learning.

g XGBoost: Extreme Gradient Boosting.

Basic Characteristics of the Data Set of 53 Question-Answer Pairs

Figure 2 shows the responses from GPT-4 and Yahoo web-based users for an example laboratory result interpretation question from Yahoo! Answers. Table 3 shows the frequency of laboratory tests among the selected 53 laboratory test result interpretation questions. Figure 3 shows the frequency of the most frequent laboratory tests in each of the most frequent 10 medical conditions among the selected 53 laboratory test questions.

a HDL: high-density lipoprotein.

Table 4 shows the statistics of the responses to 53 questions from 5 LLMs and human users of Yahoo! Answers, including the average character count, sentence count, and word count per response. Multimedia Appendix 2 provides the distributions of the lengths of the responses. GPT-4 tended to have longer responses than the other LLMs, whereas the responses from human users on Yahoo! Answers tended to be shorter with respect to all 3 counts. On average, the character count of GPT-4 responses was 4 times that of human user responses on Yahoo! Answers.

Automated Comparison of Similarities in LLM Responses

Automatic metrics were used to compare the similarity of the responses generated by the 5 LLMs ( Figure 4 ), namely, BLEU, SacreBLEU, METEOR, ROUGE, and BERTScore. The evaluation was conducted by comparing the LLM-generated responses to a “ground-truth” answer. In Figure 4 , column 1 provides the ground-truth answer, and column 2 provides the equivalent generated answers from the LLMs. We also included the human answers from Yahoo! Answers for this evaluation. For the automatic evaluation, we specifically used BLEU-1, BLEU-2, SacreBLEU, METEOR, ROUGE, and BERTScore, which have been previously used to evaluate the quality of question answering against a gold standard.

All the metrics ranged from 0.0 to 1.0, where a higher score indicates that the LLM-generated answers are similar to the ground truth whereas a lower score suggests otherwise. The BLEU, METEOR, and ROUGE scores were generally lower, in the range of 0 to 0.37, whereas BERTScore values were generally higher, in the range of 0.46 to 0.63. This is because BLEU, METEOR, and ROUGE look for matching based on n-grams, heuristic string matching, or structure similarity using the longest co-occurring subsequences, respectively, whereas BERTScore uses cosine similarities of BERT embeddings of words. When GPT-4 was the reference answer, the response from GPT-3.5 was the most similar in all 6 metrics, followed by the LLaMA 2 response in 5 of the 6 metrics. Similarly, when GPT-3.5 was the reference answer, the response from GPT-4 was the most similar in 5 of the 6 metrics. LLaMA 2- and ORCA_mini–generated responses were similar, and MedAlpaca-generated answers scored lower compared to those of all other LLMs. Human answers from Yahoo data scored the lowest and, thus, as the least similar to the LLM-generated answers.

Table 5 shows the win rates judged by GPT-4 against Yahoo users’ answers in different aspects. Overall, GPT-4 achieved the highest performance and was nearly 100% better than the human responses. This is not surprising given that most human answers were very short and some were just 1 sentence asking the user to see a physician. GPT-4 and GPT-3.5 were followed by LLaMA 2 and ORCA_mini with 70% to 80% win rates. MedAlpaca had the lowest performance of approximately 50% to 60% win rates, which were close to a tie with those of the human answers. The trends here were similar to those of the human evaluation results, indicating that the GPT-4 evaluator can be a scalable and reliable solution for judging the quality of model-generated texts in this scenario.

Manual Evaluation With Medical Experts

Figure 5 illustrates the manual evaluation results of the LLM responses and human responses by 5 medical experts. Note that a lower value means a higher score. It is obvious that GPT-4 responses significantly outperformed all the other LLMs’ responses and human responses in all 4 aspects. Textbox 2 shows experts’ feedback on the LLM and human responses. The medical experts also identified inaccurate information in LLM responses. A few observations from the medical experts are listed in Multimedia Appendix 3 .

Large language model or human answer and expert feedback

• LLaMA 2: “It is a great answer. He was able to explain in details the results. He provides inside on the different differential diagnosis. And provide alternative a management. He shows empathy.”
• LLaMA 2: “Very thorough and thoughtful.”
• ORCA_mini: “It was a great answer. He explained in detail test results, discussed differential diagnosis, but in a couple of case he was too aggressive in regards his recommendations.”
• ORCA_mini: “Standard answers, not the most in depth.”
• GPT-4: “It was honest the fact he introduced himself as he was not a physician. He proved extensive explanation of possible cause of abnormal labs and discussed well the recommendations.”
• GPT-4: “Too wordy at times, gets irrelevant.”
• GPT-3.5: “Strong responses in general.”
• GPT-3.5: “Clear and some way informative and helpful to pts.”
• GPT-3.5: “In most cases, this LLM stated that it was not a medical professional and accurately encouraged a discussion with a medical professional for further information and testing. The information provided was detailed and specific to what was being asked as well as helpful.”
• MedAlpaca: “This statement seems so sure that he felt superficial. It made me feel he did not provide enough information. It felt not safe for the patient.”
• MedAlpaca: “Short and succinct. condescending at times.”
• Human answer: “These were not very helpful or accurate. Most did not state their credentials to know how credible they are. Some of the, if not most, of language learning models gave better answers, though some of the language learning models also claimed to be medical professionals—which isn’t accurate statement either.”
• Human answer: “Usually focused on one aspect of the scenario, not helpful in comprehensive care. focused on isolated lab value, with minimal evidence—these can be harmful responses for patients.”
• Human answer: “These are really bad answers.”
• Human answer: “Some of the answer were helpful, other not much, and other offering options that might not need to be indicated.”

Principal Findings

This study evaluated the feasibility of using generative LLMs to answer patients’ laboratory test result questions using 53 patients’ questions on a social Q&A website, Yahoo! Answers. On the basis of the results of our study, GPT-4 outperformed other similar LLMs (ie, GPT-3.5, LLaMA 2, ORCA_mini, and MedAlpaca) according to both automated metrics and manual evaluation. In particular, GPT-4 always provided disclaimers, possibly to avoid legal issues. However, GPT-4 responses may also suffer from lack of interpretation of one’s medical context, incorrect statements, and lack of references.

Recent studies [ 26 , 27 ] regarding the use of LLMs to answer laboratory test result questions from medical professionals found that ChatGPT may give superficial or incorrect answers to laboratory test result–related questions and can only provide accurate answers to approximately 50% of questions [ 26 ]. They also found that ChatGPT’s responses were not sufficiently tailored to the case or clinical questions to be useful for clinical consultation. For instance, diagnoses of liver injury were made solely based on γ-glutamyl transferase levels without considering other liver enzyme indicators. In addition, high levels of glucose and glycated hemoglobin (HbA 1c ) were both identified as indicative of diabetes regardless of whether HbA 1c levels were normal or elevated. These studies also highlighted that GPT-4 failed to account for preanalytical factors such as fasting status for glucose tests and struggled to differentiate between abnormal and critically abnormal laboratory test values. Our study observed similar patterns, where a normal HbA 1c level coupled with high glucose levels led to a diabetes prediction and critically low iron levels were merely classified as abnormal.

In addition, our findings also show that GPT-4 accurately distinguished between normal, prediabetic, and diabetic HbA 1c ranges considering fasting glucose levels and preanalytical conditions such as fasting status. Furthermore, in cases of elevated bilirubin levels, GPT-4 correctly associated them with potential jaundice citing the patient’s yellow eye discoloration and appropriately considered a comprehensive set of laboratory test results—including elevated liver enzymes and bilirubin levels—and significant alcohol intake history to recommend diagnoses such as alcoholic liver disease, hepatitis, bile duct obstruction, and liver cancer.

On the basis of our observation with the limited number of questions, we found that patients’ questions are often less complex than professionals’ questions, making ChatGPT more likely to provide an adequately accurate answer to such questions. In our manual evaluation of 7 selected patients’ laboratory test result questions, 91% (32/35) of the ratings from 5 medical experts on GPT-4’s response accuracy were either 1 ( very high ) or 2 ( high ).

Through this study, we gained insights into the challenges of using generative LLMs to answer patients’ laboratory test result–related questions and provide suggestions to mitigate these challenges. First, when asking laboratory test result questions on social Q&A websites, patients tend to focus on laboratory test results but may not provide pertinent information needed for result interpretation. In the real-world clinical setting, to fully evaluate the results, clinicians may need to evaluate the medical history of a patient and examine the trends of the laboratory test results over time. This shows that, to allow LLMs to provide a more thorough evaluation of laboratory test results, the question prompts may need to be augmented with additional information. As such, LLMs could be useful in prompting patients to provide additional information. A possible question prompt would be the following: “What additional information or data would you need to provide a more accurate diagnosis for me?”

Second, we found that it is important to understand the limitations of LLMs when answering laboratory test–related questions. As general-purpose generative AI models, they should be used to explain common terminologies and test purposes; clarify the typical reference ranges for common laboratory tests and what it might mean to have values outside these ranges; and offer general interpretation of laboratory test results, such as what it might mean to have high or low levels in certain common laboratory tests. On the basis of our findings, LLMs, especially GPT-4, can provide a basic interpretation of laboratory test results without reference ranges in the question prompts. LLMs could also be used to suggest what questions to ask health care providers. They should not be used for diagnostic purposes or treatment advice. All laboratory test results should be interpreted by a health care professional who can consider the full context of one’s health. For providers, LLMs could also be used as an educational tool for laboratory professionals, providing real-time information and explanations of laboratory techniques. When using LLMs for laboratory test result interpretation, it is important to consider the ethical and practical implications, including data privacy, the need for human oversight, and the potential for AI to both enhance and disrupt clinical workflows.

Third, we found it challenging to evaluate laboratory test result questions using Q&A pairs from social Q&A websites such as Yahoo! Answers. This is mainly because the answers provided by web-based users (who may not be medical professionals) were generally short, often focused on one aspect of the question or isolated laboratory tests, possibly opinionated, and possibly inaccurate with minimal evidence. Therefore, it is unlikely that human answers from social Q&A websites can be used as a gold standard to evaluate LLM answers. We found that GPT-4 can provide comprehensive, thoughtful, sympathetic, and fairly accurate interpretation of individual laboratory tests, but it still suffers from a number of problems: (1) LLM answers are not individualized, (2) it is not clear what are the sources LLMs use to generate the answers, (3) LLMs do not ask clarifying questions if the provided prompts do not contain important information for LLMs to generate responses, and (4) validation by medical experts is needed to reduce hallucination and fill in missing information to ensure the quality of the responses.

Future Directions

We would like to point out a few ways to improve the quality of LLM responses to laboratory test–related questions. First, the interpretation of certain laboratory tests is dependent on age group, gender, and possibly other conditions pertaining to particular population subgroups (eg, pregnant women), but LLMs do not ask clarifying questions, so it is important to enrich the question prompts with necessary information available in electronic health records or ask patients to provide necessary information for more accurate interpretation. Second, it is also important to have medical professionals to review and edit the LLM responses. For example, we found that LLaMA 2 self-identified as a “health expert,” which is obviously problematic if such responses were directly sent to patients. Therefore, it is important to postprocess the responses to highlight sentences that are risky. Third, LLMs are sensitive to question prompts. We could study different prompt engineering and structuring strategies (eg, role prompting and chain of thought) and evaluate whether these prompting approaches would improve the quality of the answers. Fourth, one could also collect clinical guidelines that provide credible laboratory result interpretation to further train LLMs to improve answer quality. We could then leverage the retrieval-augmented generation approach to allow LLMs to generate responses from a limited set of credible information sources [ 52 ]. Fifth, we could evaluate the confidence level of the sentences in the responses. Sixth, a gold-standard benchmark Q&A data set for laboratory result interpretation could be developed to allow the community to advance with different augmentation approaches.

Limitations

A few limitations should be noted in this study. First, the ChatGPT web version is nondeterministic in that the same prompt may generate different responses when used by different users. Second, the sample size for the human evaluation was small. Nonetheless, this study produced evidence that LLMs such as GPT-4 can be a promising tool for filling the information gap for understanding laboratory tests and various approaches can be used to enhance the quality of the responses.

Conclusions

In this study, we evaluated the feasibility of using generative LLMs to answer common laboratory test result interpretation questions from patients. We generated responses from 5 LLMs—ChatGPT (GPT-4 version and GPT-3.5 version), LLaMA 2, MedAlpaca, and ORCA_mini—for laboratory test questions selected from Yahoo! Answers and evaluated these responses using both automated metrics and manual evaluation. We found that GPT-4 performed better compared to the other LLMs in generating more accurate, helpful, relevant, and safe answers to these questions. We also identified a number of ways to improve the quality of LLM responses from both the prompt and response sides.

Acknowledgments

This project was partially supported by the University of Florida Clinical and Translational Science Institute, which is supported in part by the National Institutes of Health (NIH) National Center for Advancing Translational Sciences under award UL1TR001427, as well as the Agency for Healthcare Research and Quality (AHRQ) under award R21HS029969. This study was supported by the NIH Intramural Research Program, National Library of Medicine (QJ and ZL). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH and AHRQ. The authors would like to thank Angelique Deville, Caroline Bennett, Hailey Thompson, and Maggie Awad for labeling the questions for the question classification model.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

QJ is a coauthor and an active associate editor for the Journal of Medical Internet Research . All other authors declare no other conflicts of interest.

The responses generated by the 5 large language models and the human answers from Yahoo users.

Distribution of the lengths of the responses.

A few observations from the medical experts regarding the accuracy of the large language model responses.

• Healthy people 2030: building a healthier future for all. Office of Disease Prevention and Health Promotion. URL: https://health.gov/healthypeople [accessed 2023-05-09]
• NHE fact sheet. Centers for Medicare & Medicaid Services. URL: https://tinyurl.com/yc4durw4 [accessed 2023-06-06]
• Bauer UE, Briss PA, Goodman RA, Bowman BA. Prevention of chronic disease in the 21st century: elimination of the leading preventable causes of premature death and disability in the USA. Lancet. Jul 2014;384(9937):45-52. [ CrossRef ]
• Centers for Medicare and Medicaid Services (CMS), Centers for Disease Control and Prevention (CDC), Office for Civil Rights (OCR). CLIA program and HIPAA privacy rule; patients' access to test reports. Final rule. Fed Regist. Feb 06, 2014;79(25):7289-7316. [ FREE Full text ] [ Medline ]
• Pillemer F, Price R, Paone S, Martich GD, Albert S, Haidari L, et al. Direct release of test results to patients increases patient engagement and utilization of care. PLoS One. 2016;11(6):e0154743. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Health IT legislation: 21st century cures act. Office of the National Coordinator for Health Information Technology. URL: https://www.healthit.gov/topic/laws-regulation-and-policy/health-it-legislation [accessed 2023-02-19]
• Tsai R, Bell EJ, Woo H, Baldwin K, Pfeffer M. How patients use a patient portal: an institutional case study of demographics and usage patterns. Appl Clin Inform. Jan 06, 2019;10(1):96-102. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Witteman HO, Zikmund-Fisher BJ. Communicating laboratory results to patients and families. Clin Chem Lab Med. Feb 25, 2019;57(3):359-364. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Turchioe MR, Myers A, Isaac S, Baik D, Grossman LV, Ancker JS, et al. A systematic review of patient-facing visualizations of personal health data. Appl Clin Inform. Aug 09, 2019;10(4):751-770. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alpert JM, Krist AH, Aycock RA, Kreps GL. Applying multiple methods to comprehensively evaluate a patient portal's effectiveness to convey information to patients. J Med Internet Res. May 17, 2016;18(5):e112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zikmund-Fisher BJ, Exe NL, Witteman HO. Numeracy and literacy independently predict patients' ability to identify out-of-range test results. J Med Internet Res. Aug 08, 2014;16(8):e187. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Citardi D, Xing A, Luo X, Lu Y, He Z. Patient challenges and needs in comprehending laboratory test results: mixed methods study. J Med Internet Res. Dec 07, 2020;22(12):e18725. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fraccaro P, Vigo M, Balatsoukas P, van der Veer SN, Hassan L, Williams R, et al. Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour. BMC Med Inform Decis Mak. Feb 12, 2018;18(1):11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bar-Lev S, Beimel D. Numbers, graphs and words - do we really understand the lab test results accessible via the patient portals? Isr J Health Policy Res. Oct 28, 2020;9(1):58. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Giardina TD, Baldwin J, Nystrom DT, Sittig DF, Singh H. Patient perceptions of receiving test results via online portals: a mixed-methods study. J Am Med Inform Assoc. Apr 01, 2018;25(4):440-446. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Doi T, Langsted A, Nordestgaard BG. Elevated remnant cholesterol reclassifies risk of ischemic heart disease and myocardial infarction. J Am Coll Cardiol. Jun 21, 2022;79(24):2383-2397. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wadström BN, Wulff AB, Pedersen KM, Jensen GB, Nordestgaard BG. Elevated remnant cholesterol increases the risk of peripheral artery disease, myocardial infarction, and ischaemic stroke: a cohort-based study. Eur Heart J. Sep 07, 2022;43(34):3258-3269. [ CrossRef ] [ Medline ]
• Chu SK, Huang H, Wong WN, van Ginneken WF, Wu KM, Hung MY. Quality and clarity of health information on Q and A sites. Libr Inf Sci Res. Jul 2018;40(3-4):237-244. [ CrossRef ]
• Oh S, Yi YJ, Worrall A. Quality of health answers in social Q and A. Proc Assoc Inf Sci Technol. Jan 24, 2013;49(1):1-6. [ CrossRef ]
• Tao D, Yuan J, Qu X, Wang T, Chen X. Presentation of personal health information for consumers: an experimental comparison of four visualization formats. In: Proceedings of the 15th International Conference on Engineering Psychology and Cognitive Ergonomics. 2018. Presented at: EPCE '18; July 15-20, 2018;490-500; Las Vegas, NV. URL: https://link.springer.com/chapter/10.1007/978-3-319-91122-9_40
• Struikman B, Bol N, Goedhart A, van Weert JC, Talboom-Kamp E, van Delft S, et al. Features of a patient portal for blood test results and patient health engagement: web-based pre-post experiment. J Med Internet Res. Jul 20, 2020;22(7):e15798. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kopanitsa G. Study of patients' attitude to automatic interpretation of laboratory test results and its influence on follow-up rate. BMC Med Inform Decis Mak. Mar 27, 2022;22(1):79. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zikmund-Fisher BJ, Scherer AM, Witteman HO, Solomon JB, Exe NL, Fagerlin A. Effect of harm anchors in visual displays of test results on patient perceptions of urgency about near-normal values: experimental study. J Med Internet Res. Mar 26, 2018;20(3):e98. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morrow D, Azevedo RF, Garcia-Retamero R, Hasegawa-Johnson M, Huang T, Schuh W, et al. Contextualizing numeric clinical test results for gist comprehension: implications for EHR patient portals. J Exp Psychol Appl. Mar 2019;25(1):41-61. [ CrossRef ] [ Medline ]
• Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Brief Bioinform. Nov 22, 2023;25(1):bbad493. [ CrossRef ] [ Medline ]
• Cadamuro J, Cabitza F, Debeljak Z, De Bruyne SD, Frans G, Perez SM, et al. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) working group on artificial intelligence (WG-AI). Clin Chem Lab Med. Jun 27, 2023;61(7):1158-1166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Munoz-Zuluaga C, Zhao Z, Wang F, Greenblatt MB, Yang HS. Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine. Clin Chem. Aug 02, 2023;69(8):939-940. [ CrossRef ] [ Medline ]
• Zhang Z, Lu Y, Kou Y, Wu DT, Huh-Yoo J, He Z. Understanding patient information needs about their clinical laboratory results: a study of social Q and A site. Stud Health Technol Inform. Aug 21, 2019;264:1403-1407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Lu Y, Wilson C, He Z. Making sense of clinical laboratory results: an analysis of questions and replies in a social Q and A community. Stud Health Technol Inform. Aug 21, 2019;264:2009-2010. [ CrossRef ] [ Medline ]
• Kurstjens S, Schipper A, Krabbe J, Kusters R. Predicting hemoglobinopathies using ChatGPT. Clin Chem Lab Med. Feb 26, 2024;62(3):e59-e61. [ FREE Full text ] [ CrossRef ] [ Medline ]
• He Z, Tian S, Erdengasileng A, Hanna K, Gong Y, Zhang Z, et al. Annotation and information extraction of consumer-friendly health articles for enhancing laboratory test reporting. AMIA Annu Symp Proc. 2023;2023:407-416. [ FREE Full text ] [ Medline ]
• Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. Feb 15, 2020;36(4):1234-1240. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alsentzer E, Murphy JR, Boag W, Weng WH, Jin D, Naumann T, et al. Publicly available clinical BERT embeddings. arXiv. Preprint posted online on April 6, 2019. [ FREE Full text ] [ CrossRef ]
• Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. arXiv. Preprint posted online on March 26, 2019. [ FREE Full text ]
• Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc. Oct 15, 2021;3(1):1-23. [ FREE Full text ] [ CrossRef ]
• OpenAI. GPT-4 technical report. arXiv. Preprint posted online on March 15, 2023. [ FREE Full text ]
• Ye J, Chen X, Xu N, Zu C, Shao Z, Liu S, et al. A comprehensive capability analysis of GPT-3 and GPT-3. arXiv. Preprint posted online on March 18, 2023. [ FREE Full text ]
• Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: open foundation and fine-tuned chat models. arXiv. Preprint posted online on July 18, 2023. [ FREE Full text ]
• Han T, Adams LC, Papaioannou JM, Grundmann P, Oberhauser T, Löser A, et al. MedAlpaca -- an open-source collection of medical conversational AI models and training data. arXiv. Preprint posted online on April 14, 2023. [ FREE Full text ]
• orca_mini_3b. Hugging Face. URL: https://huggingface.co/pankajmathur/orca_mini_3b [accessed 2023-12-04]
• LangChain: introduction and getting started. Pinecone. URL: https://www.pinecone.io/learn/series/langchain/langchain-intro/ [accessed 2023-12-04]
• Papineni K, Roukos S, Ward T, Zhu WJ. BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002. Presented at: ALC '02; July 7-12, 2002;311-318; Philadelphia, PA. URL: https://dl.acm.org/doi/10.3115/1073083.1073135 [ CrossRef ]
• Post M. A call for clarity in reporting BLEU scores. arXiv. Preprint posted online on April 23, 2018. [ FREE Full text ] [ CrossRef ]
• Banerjee S, Lavie A. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the 2005 ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 2005. Presented at: WIEEMMTS '05; June 29, 2005;65-72; Ann Arbor, MI. URL: https://aclanthology.org/W05-0909 [ CrossRef ]
• Lin CY. ROUGE: a package for automatic evaluation of summaries. In: Lin CY, editor. Text Summarization Branches Out Internet. Barcelona, Spain. Association for Computational Linguistics; 2004;74-81.
• Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: evaluating text generation with BERT. arXiv. Preprint posted online on April 21, 2019. [ FREE Full text ]
• Wang T, Yu P, Tan XE, O'Brien S, Pasunuru R, Dwivedi-Yu J, et al. Shepherd: a critic for language model generation. arXiv. Preprint posted online on August 8, 2023. [ FREE Full text ]
• Dubois Y, Li X, Taori R, Zhang T, Gulrajani I, Ba J, et al. AlpacaFarm: a simulation framework for methods that learn from human feedback. arXiv. Preprint posted online on May 22, 2023. [ FREE Full text ]
• Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep. Aug 1966;19(1):3-11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gamer M, Lemon J, Singh IF. irr: various coefficients of interrater reliability and agreement. Cran R Project. 2019. URL: https://cran.r-project.org/web/packages/irr/index.html [accessed 2023-12-12]
• Human subject regulations decision charts: 2018 requirements. Office for Human Research Protection. Jan 20, 2019. URL: https://tinyurl.com/3sbzydm3 [accessed 2024-04-03]
• Jin Q, Leaman R, Lu Z. Retrieve, summarize, and verify: how will ChatGPT affect information seeking from the medical literature? J Am Soc Nephrol. Aug 01, 2023;34(8):1302-1304. [ CrossRef ] [ Medline ]

Abbreviations

Edited by B Puladi; submitted 23.01.24; peer-reviewed by Y Chen, Z Smutny; comments to author 01.02.24; revised version received 17.02.24; accepted 06.03.24; published 17.04.24.

©Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian, Karim Hanna, Cindy Shavor, Lisbeth Garcia Arguello, Patrick Murray, Zhiyong Lu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Your path to academic success

Improve your paper with our award-winning Proofreading Services ,  Plagiarism Checker , Citation Generator , AI Detector & Knowledge Base .

Proofreading & Editing

Get expert help from Scribbr’s academic editors, who will proofread and edit your essay, paper, or dissertation to perfection.

Plagiarism Checker

Detect and resolve unintentional plagiarism with the Scribbr Plagiarism Checker, so you can submit your paper with confidence.

Citation Generator

Generate accurate citations with Scribbr’s free citation generator and save hours of repetitive work.

Happy to help you

You’re not alone. Together with our team and highly qualified editors , we help you answer all your questions about academic writing.

Open 24/7 – 365 days a year. Always available to help you.

Very satisfied students

This is our reason for working. We want to make  all students  happy, every day.

Highly Recommended

It is beneficial and easy to use.

It helps my papers have their…

It helps my papers have their citations. I've been using this for years and has helped so much!

Scribbr makes citing sources very easy…

Scribbr makes citing sources very easy - I love it!

Easy to use and helps me keep track of…

Easy to use and helps me keep track of all my citations.

Best resource for both my professional and academic career!!!

Extremely helpful

Extremely helpful. I never heard of this site until I Googled citation generation and it has helped me so much during my Master's program. I do not have to type in many field most of the time which saves a lot of time. This site is beyond helpful and useful.

its been an amazing tool very helpful

I absolutely love how easy and…

I absolutely love how easy and intuitive Scribbr is to use. I 100% recommend it to any student, professional, or writer looking for a user-friendly writing tool.

Easy to use and accurate

Service is easy to use and accurate. Ability to automatically save past references is a big help for ongoing learners.

Very useful for quick citing

I like the feature where you can cite a site while on it.

This is a great tool

This is a great tool. It is so helpful! Thank you

Could not have made it through grad…

Could not have made it through grad school without the help of this site! Thank you!

A most beneficial and critical utility.

Never have I ever used a more powerful and effective tool. Thank you.

Scribbr is fantastic!

Scribbr is consistent, not riddled with ads, the search bar generates relevant results, and I love all the ways I can edit entries! It makes my life as an editor so much easier!!

Easy to use

Easy to use.

👍👍👍👍👍👍👍👍👍👍👍👍

I actually love this website because it…

I actually love this website because it has helped me and academically still making sure all of my sources have been in APA formatted ! It has been a lifesaver !

Referencing made easier!

Excellent tool and easy to use.

I love this site because it saves my…

I love this site because it saves my work, is user friendly and doesn't have any requirements to pay for services. It makes getting homework/projects done easier and makes it stress free.

Everything you need to write an A-grade paper

Free resources used by 5,000,000 students every month.

Bite-sized videos that guide you through the writing process. Get the popcorn, sit back, and learn!

Lecture slides

Ready-made slides for teachers and professors that want to kickstart their lectures.

• Academic writing
• Citing sources
• Methodology
• Research process
• Dissertation structure
• Language rules

Accessible how-to guides full of examples that help you write a flawless essay, proposal, or dissertation.

Chrome extension

Cite any page or article with a single click right from your browser.

Time-saving templates that you can download and edit in Word or Google Docs.

Help you achieve your academic goals

Whether we’re proofreading and editing , checking for plagiarism or AI content , generating citations, or writing useful Knowledge Base articles , our aim is to support students on their journey to become better academic writers.

We believe that every student should have the right tools for academic success. Free tools like a paraphrasing tool , grammar checker, summarizer and an  AI Proofreader . We pave the way to your academic degree.

Ask our team

Want to contact us directly? No problem.  We  are always here for you.

• Email [email protected]
• Start live chat
• Call +1 (510) 822-8066
• WhatsApp +31 20 261 6040

Frequently asked questions

Our team helps students graduate by offering:

• A world-class citation generator
• Plagiarism Checker software powered by Turnitin
• Innovative Citation Checker software
• Professional proofreading services
• Over 300 helpful articles about academic writing, citing sources, plagiarism, and more

Scribbr specializes in editing study-related documents . We proofread:

• PhD dissertations
• Research proposals
• Personal statements
• Admission essays
• Motivation letters
• Reflection papers
• Journal articles
• Capstone projects

Scribbr’s Plagiarism Checker is powered by elements of Turnitin’s Similarity Checker , namely the plagiarism detection software and the Internet Archive and Premium Scholarly Publications content databases .

The add-on AI detector is powered by Scribbr’s proprietary software.

The Scribbr Citation Generator is developed using the open-source Citation Style Language (CSL) project and Frank Bennett’s citeproc-js . It’s the same technology used by dozens of other popular citation tools, including Mendeley and Zotero.

You can find all the citation styles and locales used in the Scribbr Citation Generator in our publicly accessible repository on Github .

Read our research on: Gun Policy | International Conflict | Election 2024

Regions & Countries

Political typology quiz.

Notice: Beginning April 18th community groups will be temporarily unavailable for extended maintenance. Thank you for your understanding and cooperation.

Where do you fit in the political typology?

Are you a faith and flag conservative progressive left or somewhere in between.

Take our quiz to find out which one of our nine political typology groups is your best match, compared with a nationally representative survey of more than 10,000 U.S. adults by Pew Research Center. You may find some of these questions are difficult to answer. That’s OK. In those cases, pick the answer that comes closest to your view, even if it isn’t exactly right.

About Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .