Low-Scoring Books

What is the low-score threshold?

  • In the years that MWSA has used numerical scoring, we’ve set a threshold analogous to a failing grade (an “F”) in school.

  • We have different thresholds for tech and overall scores.

  • Thus, any book might have a numerical score below the applicable threshold in

    • the tech scoring only,

    • the overall score only, or

    • both both the tech and overall score.

Why have a low-score threshold?

  • Every year, authors submit books that are clearly “not ready for prime time.”

  • MWSA reviews are not normally critical.

  • If ALL books submitted were to receive a favorable review—even the poorly crafted ones—MWSA’s credibility would suffer.

Doesn’t this conflict with MWSA’s mission of helping our authors?

  • Yes it can; and that’s why we’ve created our system of “Caveated Reviews.”

The Caveated Review
Reviewer one's job in a low-score situation

Although comments are intended to be as critical as needed, MWSA will never “slam” our authors with brutal reviews—even when such a review might be brutally accurate. Instead, we use what we call “caveated reviews.”

Considering the credibility issue raised above, in a low-score situation, the number one’s review MUST mention some shortfalls discovered during the book’s evaluation. We have two ways of doing this:

  1. Specific remarks pointing out problem areas

  2. A boilerplate statement appended to the end of the review.

We strongly prefer option one because the comments can be targeted, specific, and diplomatic. These comments can also be softened by countervailing positive observations.

Put yourself in a low-scoring author’s shoes, and pick which of the below you’d prefer to read in your MWSA review:

  1. A few typos and run-on sentences sometimes detracted from this page-turning thriller.

  2. MWSA's evaluation of this book found technical problems—including some combination of misspellings, grammar, punctuation, or capitalization errors.

See our Writing a Review page for details about review writing and low-scoring books.

 

Low-Score Notification Email

Dear John Q Author:

Your three MWSA reviewers have completed their evaluation of your book, "Name of Book." Unfortunately, their combined average score fell below the minimum threshold required for both award qualification and that mentioned under the blue-colored REVIEW section of our Book Submission page:

https://www.mwsadispatches.com/awards/

Under that policy, our MWSA review would have to mention at least some of the problem areas noted during our evaluation. Since our goal is to help our MWSA member authors, we will not post the review without first getting your consent. We’ve included a draft of our review below.

Action required
We understand this is not the news you hoped to get from MWSA, but we need you to let us know which of the following options you choose:

1. Accept the review as written and have it posted on our website.
2. Not accept the review, remove your book from this season's review and awards page but keep your book listed in our online library.
3. Not accept the review, remove your book from both the current season page and MWSA’s online library.

Reviewer Comments
We're also forwarding your reviewers' comments in this email. Before reading these comments, a few caveats are in order:

- Although all three reviewers score each book, only one writes our MWSA review.
- Unlike your MWSA review, your reviewers' comments are never released to the public and serve two purposes: 1) to document their numerical scoring and 2) to provide feedback to authors.
- Some of the comments explaining the reviewer's individual scoring can be blunt, direct, and often in the form of unedited shorthand notes -- we do not edit our reviewers' comments before sharing them with authors.
- The level of detail can vary significantly among our volunteer reviewers.
- In some cases, you’ll see references to numbers in reviewers' comments; these let the Awards Directors know that the comment addresses a specific numbered evaluation criterion.
- You may use these reviewer comments for your feedback. Do not share any of them publicly. You might consider that some of the comments would look good on your website, but that would violate our terms of sharing.

We hope you find the comments below helpful to your writing efforts.

Again, please let us know how you’d like to proceed regarding your MWSA review (choosing one of the options above). Thank you for sharing your work with us, and we wish you luck in your future writing endeavors. ​

Sincerely,

________________________
MWSA Awards Directors
Betsy Beard
John Cathcart


------------- Draft MWSA Review -------------

***** Paste Review Here ***** [#1 review must include critical comments OR applicable low-score boilerplate]

------------------ Reviewers' Comments --------------------
[Reviewer 1 - 3]
Content:
Style:
Visual:
Technical:
Author Comments:

--------------------------------------------------------

Content:
Style:
Visual:
Technical:
Author Comments:

--------------------------------------------------------

Content:
Style:
Visual:
Technical:
Author Comments:

Reviewers' Membership Status and Address Changes/Updates

Since the season started, Betsy and I have been trying to update our reviewer information. In doing so, we discovered that several reviewers do not appear in the MWSA membership database. In the coming days, we'll upgrade any reviewer who isn't already a full MWSA member to an associate (free) member. We'll then send all 70 of you an individually tailored reviewer information email so you can crosscheck the information we have (address, format preference, participation in the reviewer recognition program, training dates, etc.).

 

Going through this data scrub--especially of addresses--will ensure that we have accurate information so that our authors always send review copies to the correct address. However, if you ever need to update your address, it's important to know that our review and awards tracking system is NOT connected to MWSA's membership database. If you log into your MWSA profile and update your address, we won't know you did this (unless you inform us via email) because the two systems don't "talk to each other." Sorry for any inconvenience and double-duty work this entails for any of you with new addresses. 

Important Reminder on Scoring MWSA Books

We appreciate your willingness to serve and to help other authors with constructive feedback. We receive expressions of gratitude each year from many authors when they get the comments (both positive and negative) we provide at the end of the season.  

Because it's so important to our Awards and Reviews program, we wanted to remind you of several things regarding scoring comments as we start this new season. Please consider printing this email and keeping it for reference when you score books. 

As you know, reviewer comments are required for Content, Style, Visual, and Tech scoring sections. Comments are NOT required in the Author Comments section. At the end of the season, authors receive ALL reviewer comments other than EYES ONLY, but not scores. What does this mean for what you should (or should NOT) write in your comments sections?

DO:

  • Do summarize your thoughts in each section. Write a few sentences that let the author know the weak points and strong points of the content, style, and visual aspects. 

  • Do be kind while pointing out negative aspects. 

  • Do note tech problems by listing misspelled/wrong/extra/missing words (with page numbers or Kindle locations). This helps us justify a low score. 

  • Do show examples of tech problems that are awkward sentences, consistent misuse of punctuation, capitalization issues, etc.

DO NOT:

  • DO NOT copy and paste or use the same language as the questions. (Be original and summarize without using the numbers from the questions.)

  • DO NOT regurgitate your same answers (copy and paste) from other sections into the author comments section. (Needless repetition is annoying, and you do not even need to add author comments if you explained things in the section comments. You may use author comments to encourage, thank for service, make generic suggestions.)

  • DO NOT comment on medalworthiness in ANY comment section except EYES ONLY.

  • DO NOT mention your opinion of the author's other works in ANY comment section except EYES ONLY.

If we all follow the above guidelines, it will mean less work for the awards directors and also provide meaningful feedback for our authors. If you have any questions about this, let us know.

Thank you so much for reading until the end. 

Betsy

An AI interpretation of our reviewer slides

As a test, I entered our training slides into an AI program to generate an interview style overview. Therer are a couple of glaring instances where the AI got things very wrong — especially regarding the visual component of our scoring — but this is pretty interesting.

Below is the unedited result.

  • 1:13: Although most of our books have a military theme, it’s not true that we only accept those types of books.

  • 3:15: Not an accurate accounting of our visual scoring criteria. What they describe DOES belong in the content and/or style sections. Our visual scoring is concerned mostly with layout (font, margins, etc.) and art (cover art and content, plus photos and illustrations, if applicable).

  • Sections covering reviewers, reviewer conferences, and reviews are pretty well done—although most of the comments are focused on fiction genres.

  • 13:00: Starts to muddle the distinction between reviews and feedback.

  • 14:58: The e-book section is a little convoluted:

    • There was no prior mention of the reviewer recognition program (Amazon gift cards)

    • “Secret Literary Exchange Program”? What?

How to find (and evaluate) your review book's back cover on a Kindle book

The Problem

You need to look at a book’s back cover to evaluate it. Unfortunately, Kindle formats almost never include a back cover!

The Solution

Even though you’re reading a Kindle version, go online and look at the PAPER version on Amazon to see its back cover. Of course, if the author does not HAVE a paper version, we’re out of luck!

Follow the steps below.

Click to enlarge

  1.  Go to the book’s Amazon listing (available by clicking on the book’s cover on our website).

  2. Select the hardcover or paper version of the book (arrow 1).

  3. If you can’t see those versions, click on “See all formats and editions” (arrow 2)

  4. Click on “Read sample” (arrow 3)

  5. The back cover should be on the last available page of the sample (i.e., you may have to scroll all the way to the end of the previewed pages).

Factual Errors in Fiction

Examples of Factual Errors in Fiction

  • For any fiction genre:

    • Using an incorrect date for an actual event, or an incorrect name for an actual person.

    • Incorrect nomenclature: calling an F-16 a cargo jet, referring to an M16 as a pistol, calling the 82 Airborne the Screaming Eagles (that’s the 101st), etc.

  • Historical fiction: To be credible, the story must fit into the period depicted by the author. So, referring to President Lincoln as if he were still alive in late 1865, or referring to a state before it existed would be a factual error.

  • SciFi: it’s credible to depict faster-than-light travel, but not to say that light travels at 186,000 miles per hour (it’s miles per second).

2025 Timeline

  • Our submission window closes June 1. 

  • We hope to have all books assigned to their three reviewers two weeks after the window closes (mid-June).

  •  This will give us plenty of time to finish all books by about July 20. 

    • We plan to announce the finalists via a recorded video presentation, so it'd be nice if I had a couple of days to finish the video after Betsy and I have completed any required reviewer conferences and all the books.

    • If we finish earlier than July 20, that's even better, as it gives our finalists more time to make their travel plans.

  • Medals will be presented at the awards banquet in Kansas City (tentatively scheduled on Saturday, September 20, 2025). 

Technical Scoring Area

So far this season, we've noticed an uptick in problematic scoring in the tech area involving:

1. Insufficient comments to justify tech markdowns

2. Scoring NON-technical problems in the technical area. 

A few points to keep in mind:

  • A book can score no higher than its technical score (the "tech torpedo").

  • To avoid the possibility of an unfair or inaccurate evaluation, it's VITAL that reviewers restrict their scoring in this area to the exact five issues/questions raised:

    • Technical 1. Spelling/Typos/Wrong Word punctuation errors

    • Technical 2. Punctuation/Hyphenation

    • Technical 3. Capitalization/Italics

    • Technical 4. Sentence Structure/Syntax

    • Technical 5. Verb Form and Noun-verb Agreement

  • Of the above questions, number 4 is probably the least cut-and-dried (due to our inclusion of "awkwardly worded sentences"). Generally speaking, we're looking at sentences that are incorrectly put together, NOT necessarily long sentences. 

  • The real "no-no" is adding stylistic problems to the tech area in your comments or scoring. Some examples of this year's TECH comments from reviewers that should have been included in other sections:

    • Lack of footnotes or footnotes incorrectly rendered--covered in the Content section

    • Reviewers deducted points when they thought the author could have used better or more appropriate phrasing or word choice (happy to glad)—this is covered in the Style section. And we are not editing the books, anyway, so we don't need to think of better adjectives for them. 

  • When in doubt, either ask an awards director or put your observation into another scoring area! 

  • You can see all the evaluation questions before you score a book; go to Scoring Forms (https://www.mwsadispatches.com/scoring-forms -- the link is also found on the left navigation bar on our reviewers' page). 

Our plan:

To ensure that we're being fair and consistent in our evaluations, if we find problems in your tech scoring, we'll ask you to either provide more justification for your scoring or raise your score. 

Again, you don't have to document ALL errors; just provide enough examples to justify your scoring.

End of Season Reviewer Zoom

November 18, 2023

END-OF-SEASON ZOOM NOVEMBER 18, 2023

·  Watch the video on YouTube: https://youtu.be/cDLPft6cI-0

·  Past season questions regarding:

o historical accuracy/fact-checking: if something jumps out at you, look it up, but don't fact-check the entire book.

o reviewer comments: remember to be kind in the comment sections and use the Eyes-only section to vent your frustrations.

o spotting bad repetition and bad fragments: samples were shared.

·  New information for the 2024 season

    • Improved genre definitions: Thank you to our genre working group (Dawn Brotherton, Janette Stone, Pat Walkow, and Lillian Zelinski). See them here https://www.mwsadispatches.com/review-genres-subcategories

    • Improved scoring questions: Thank you to our scoring form re-write group (Nancy Arbuthnot, Dawn Brotherton, Annette Grunseth, Janette Stone, Pat Walkow, Lillian Zelinski, and Sue Rushford for proofing).

    • New timeline for next season is January 2 to June 1, 2024

    • New scale for scoring, using 100-point score instead of 70-point score. This also changes the medal thresholds and weighting factors.

    • New table for tech scoring using word count instead of page count.

    • New choices at end of the scoring form's Medal recommendation: 6 caveated review and 7 not appropriate for the website

      • Past: caveat review discussed during reviewer conference. Now it’s part of your initial evaluation.

      • 2 types of Caveats:

        • Not ready for prime time (rife with errors)

        • Warning label (i.e., not suitable for all audiences)

    • The new scoring forms incorporate expanded remarks and links, including genre definitions and where to find additional help. The links open a new tab, so you don't lose your place in the scoring form.

    • Streamlined reviewer conference.

    • Decision that we will now accept only paper submissions in several categories due to heavy reliance on artistic elements: Children’s Picture Book, Poetry, Graphic Novel/Comic Book, Pictorial/Coffee Table. We will continue to accept paper or E-book in all other categories.

 

REQUEST FOR VOLUNTEERS

We need volunteers to test the new scoring forms so we can ensure everything is working correctly before the new season, and you can see the new scoring questions. We have completed all fiction and nonfiction scoring forms at this time.

How to score a fake book:

  1. Pick a Book# between 1 and 10 and add a reviewer slot 1, 2, or 3. Your Book.Reviewer# will be something between 1.1 and 10.3.

  2. Choose a fiction or nonfiction category to fake-score a book by going to the reviewer page and scrolling down to the scoring section. PW is MWSA2023.
    https://www.mwsadispatches.com/reviewers-private

  3. Read the new questions, fill out the form, but use the comment section if you note any errors in the questions.

  4. Submit the score, and do as many as you want.

FINAL NOTE

If you have moved, PLEASE make sure we have your current address before January! If you haven’t moved, we have the right one. 

 

Thanks so much, 

Betsy

Reviewer Comments

Thank you to everyone for getting this season off to a good start. By now many of you have discovered that the books we receive can range in quality from exceptionally good to exceptionally bad. Because the comment sections are critical to explaining the difference, I wanted to send this separate email to clear up some misunderstandings.  

Purpose: The main function of commenting in each of the four sections (content, style, visual, and tech) is to give the author feedback as to what went right and what went wrong. It also justifies your scoring. 

Type of Information: Please use the comment space to point out your personal observations in a few sentences, without using the wording from the scoring form questions. For instance, in the fiction content section, the question about characters gives you an answer choice of "Mostly well developed, believable, and genre-appropriate." The answer here helps reviewers determine if the author met the standards for character development. But using this wording in your comment section won't help the author. It's vague and general. Instead, be specific about the characters. e.g. "The main male character was well developed with a believable back story, but the female characters were all beautiful, vapid, and clueless. Female readers likely won't appreciate this."  Again, don't simply say the plot was ridiculous, but give examples. e.g. "I found it hard to believe that the antagonist would spend ten years plotting revenge against someone who cut him off in traffic." Give the authors something concrete to help them develop their craft.

Amount of Information: Give enough information to help the author. It is not necessary to comment on each of the five questions, but go with your main observations. Also, please don't use the question numbers in the comment section to tell us which question your comment pertains to. This only confuses the author, whereas reviewers and directors will already understand which question you are addressing in more detail. A good rule of thumb is that if you lower a score, a comment should explain that, especially in the tech category. We are seeing a number of lowered tech scores without sufficient examples to back it up. While we are not editing the book and don't need to document/report every error, we need examples of what the author did wrong.

Tone: Despite some books being frustrating/boring/poorly edited/disgusting, we don't want to pound the author into the ground. Well, we might want to, but instead we need to recognize that authors spend a great deal of time and money on their creations. If you can find one good thing the author did, comment on that first. Try to couch negative comments objectively. Instead of "This book needs an editor," go with "I noted twelve misspelled words and many sentence fragments. Here are some examples of what I found..." What about positive comments? Please don't say you enjoyed the book. The goal is to determine if the book meets the standards, not if you liked it. Here's an example of a positive comment: "I appreciate how the author was able to incorporate military-speak and acronyms in a way that civilians could understand." Giving concrete examples takes the comment out of the realm of opinion and places it more in the realm of objectivity. 

Eyes-only: Use this space to say anything you don't want the author to read, because the author will be given ALL of the other comments. The eyes-only space is reserved for you to vent if needed ... or gush if you want to. This section is only seen by the awards directors and the book's two other reviewers, so feel free to make your case here if you feel passionate about the book (good or bad). Also use this space to let the awards directors know of other issues or comment on medal-worthiness.  e.g. "There really wasn't a question that fit my concerns about [fill in the blank] in content, so I lowered the score of question 2 to account for it." 

Author Comments: This is not a required field. You can leave it blank if you have made suitable comments in the other sections. Don't use this section to repeat what you already wrote in the other sections. Do use it for other thoughts. e.g. "Thank you for sharing your story," or "I live in Alabama, so the dialogue really rings true for me," or "This was a little-known facet of World War II, and it deserves attention."

No-No's: 

  • Please do not mention medal worthiness in any comment the author will see. Use the Eyes-Only section to convince other reviewers and let the averaged scores determine the medal-worthiness for the author. 

  • Don't use superlatives (best/worst) in comments the author will see. Use the Eyes-only Section. What if you tell the author, "This is the best book I've ever read," and then change your score based on the findings of the other two reviewers? 

Again, thank you for all your hard work so far this season. We truly appreciate our reviewers!

Betsy

(from 2/25/23 reviewer blast email)

Do I have to read the whole book?

Normally, the answer is yes.

However, MWSA occasionally gets books that are not “ready for primetime.” Rather than force reviewers to endure the task of reading an entire book that is chock full of errors, we offer the option to do an “abbreviated evaluation.”

Here are the details:

  • Read the first 30-50 pages (depending on the length of the book).

  • Document enough errors to show the book cannot win an award.

    • If you’re finding 2-3 errors on every page, you know it’s probably not going to win an award.

    • Under our old system (where we used page numbers to determine the allowable number of errors per score), ten errors in one category (spelling, punctuation, etc.) in the first 20 pages of a book would equate to 100 errors in a 200-page book, and the book would fail in the technical scoring area.

    • Under our new word-count-based scoring system, use basic math (divide your book’s word count by its total pages) to come up with a rough idea of your book’s number of words per page. Use this number to determine which part of the tech scoring table to use.

    • Along with a list of the errors, Include a note in your comments that you found the errors in the first XX pages and did not continue to mark errors in the rest of the book.

  • Continue scanning the remainder of the book.

    • Check every 10 to 20 pages to scrutinize a couple of pages to confirm that the book continues to include errors.

    • Try to get a sense of the story arc, if fiction.

  • Expect to go through a reviewer conference for the book since it’s likely to end up being low-scoring. At that point, you’ll provide your input regarding the type of review—keeping in mind that we require a “caveated” review in books that score lower than our threshold.

If you have any doubts or questions, contact the awards directors.

2/25/2024

Google Forms Autosave and MWSA reviewing

For the past several years, MWSA has used scoring forms (Google Forms) rather than Excel spreadsheets for our evaluations and tracking (reviewer preferences, review copy receipts, etc.). The system integrates with our tracking software and has been working smoothly for several years. However, recently (sometime in 2022?), Google implemented an autosave function in all its forms. This was Google's idea, not ours.

  • The good news is that Google records your scores and comments while you’re working on the form.

  • The bad news is that the form now “requires” a Google account email sign-in.

    • We used scare quotes around “requires” because you can actually use the forms without being signed into a Google account.

    • For more details, see the “Opting Out” section below

Form with a recognized Google account
Click for larger image

Filling out MWSA Forms

  • If you have a Google account, its associated email will automatically appear at the top of the form.

  • Have a look at the screenshot (right) and notice that on Betsy or John’s computer, the form automatically uses our MWSAAwardsDirectors @gmail.com email.

  • Whatever email appears in that part of the form is NOT transmitted to MWSA and does NOT need to be the email you use with MWSA.

  • According to Google: “Google Forms automatically saves your progress for 30 days when you’re signed in to your Google account so you can work across devices or take a break without losing a step.”

    • If you’ve ever lost several pages worth of comments due to a Google Glitch, you’ll appreciate how helpful this might be. :-)

  • Your awards directors strongly recommend that you accept this “big brother intrusion” and use all our forms without fear. However, if Google makes you uncomfortable, please continue reading the next section.

Form without a recognized Google account.

Click for larger image

Opting Out

  • If you use an “Incognito Window” on your browser to fill out your form, it won’t recognize your Google account.

  • In this case, the form looks like the example on the screenshot to the right.

  • Even though it says “Sign in to Google” and includes a red “*Required,” it is NOT required!

  • When you click on “Sign in to Google,” you’ll see a pop-up like the one below and to the right.

  • Notice that you now have the option to “Continue without signing in.”

  • Once you click on that link, you’ll return to the form and can fill out your MWSA form.





Current as of February 2023

Converting scoresheet answers to scores (MWSA Rosetta Stone)

Over the years, reviewers have wondered how our scoring system works. They’re often interested in finding out how 20 multiple-choice questions convert to a max score of 70. What follows is a little history, a little math, and more than a little lingering confusion.

Normal Scoring

  • Our scoring used to be done via Excel spreadsheets. And that basic architecture carried over when we integrated reviewers' scoring and our tracking via scoring forms.

  • Possible RAW scores for every question range from zero to 1 in quarter-point increments.

    • On our scoring forms, a 5 (worst possible score) is converted to 0, a 4 = 0.25, 3 to .5, etc., topping out where a 1 = 1

    • Although each of our 20 multiple-choice questions starts with a min of 0 and a max of 1, the TOP combined score is not 20, nor is it 100. It's 70.

  • Depending on the genre, there are different weighting factors applied to each section.

    • Content questions are weighted heavier than visual, for example, EXCEPT for children's picture books (for obvious reasons)

    • For most books, the order of weighting in descending importance is: content, tech, style, visual

    • The actual weighting factor numbers by genre are buried inside our spreadsheets. It all happens automatically.

Bottom line: you'll never be able to know exactly how each change in scoring will impact the final score -- none of us do. That's why we first ask reviewers to "score blind" (answering each question as objectively as possible, without knowing how their answers translate to scores or medals).

Reviewer Conference Scoring

Your second chance at scoring only happens after a reviewer conference. At this point, rather than requiring all reviewers to do a complete do-over score, we ask them to make changes using the scoring scales found in that reviewer conference form. Although any reviewer can opt to do another scoring form, since it's easier to complete the reviewer conference form, most choose that less time-consuming option.

Reviewer Comments--One More Friendly Reminder

Please keep in mind that your reviewer comments serve two purposes: 1) document/justify your scoring, and 2) provide feedback to the author. In my opinion (and hopefully, yours), providing authors with our feedback is the most important of the two.

With the exception of the Eyes-Only section, ALL your comments are shared (anonymously) with our authors on request.  Other than the #1's written review, the comments you submit with your scoring form are the only feedback our authors receive.  Therefore, please keep in mind that if you only provide "Eyes-Only" comments, unless you're the #1, the author will receive NO feedback from you! 

If all three reviewers were to do this, and the book in question was also scored below the "low-score threshold," we might have to refuse to do a review, but could not tell the author why.

Thanks!

Book.Reviewer#--A Reminder

As a reminder, each reviewer has a unique Book.Reviewer# for each of his/her review books.  The number is a combination of the book number and the reviewer's slot number, separated by a period.  Your Book.Reviewer#(s) can always be found in your reviewer summary and its accurate use is what allows us to automate a large portion of our tracking work (i.e., keeping our system up-to-date and accurate).  

The number is used whenever you...

  • report receipt of each of your review copies via our online form

  • submit your scores and comments (plus the written review if you're the #1) for each book

  • need to submit your reviewer conference input (if necessary)

If you're ever in doubt or can't remember where to find that number, please don't guess and submit any form with a possibly bad number!  Just ask, we'd be happy to send you another reviewer summary.  Sending that reminder email is MUCH easier than trying to figure out what you did and then fix it manually (by searching through hundreds of spreadsheet lines and columns, deleting the erroneous entry, and entering the corrected data). :-)

Originally posted Jan 2019

Ready To Join MWSA or Learn More?