Labels and scores

Labels

• SpamScore 0.8 and above

  The message reads like spam. Drop it, or file it away.

• CleanScore below 0.8

  Deliver it as usual, or hold it for a person if its score is high enough for your own review queue.

Act on the label by default: if (label === "spam"). The label is Botpoison's recommended cut on the score, spam from 0.8 and clean below it, so the two never disagree. The dashboard counts and charts labels, so your form and your dashboard agree.

Scores

The score is a number from 0 to 1 that ranks how spam-like the submission is. It is not a probability. Each value stands for a false positive rate: the share of good form submissions in Botpoison's reference set, counted together, that score at or above it.

The table stops at 0.8, where the spam label starts. Every score from 0.8 up is spam, and a higher score still ranks one message above another.

Every model release is fitted to this table, and a release ships only after a check on a separate set of good messages keeps its rates within twice the table at 0.3, 0.5, 0.65 and 0.8. So if (score >= 0.7) means about the same rate after a model moves to a newer release, and the models table states the points of each one.

The rates count every good message in the reference set together, so sites with a lot of traffic weigh the most. Taken site by site, the share of good messages that reach a score is usually several times the table's rate, and higher still on a new site. The rates are measured on whole forms. A plain string sent as content is scored on the same scale, but its rates vary more around the table.

Choosing a threshold

Start with the label. Move to a number of your own only for a reason, and log the score beside what you did with the message while you tune it.

• Default: 0.8, which is the spam label, about 1 in 1,000. Drop these, or file them away.
• Review queue: from 0.5 (about 1 in 100) or 0.65 (about 1 in 280). Hold these for a person rather than discarding them.

const { label, score } = await botpoison.classify(req.body, {
  model: "spam-standard",
});
// Keep the score beside your own outcome while you tune a threshold
log.info({ label, score });
if (label === "spam") {
  // 0.8 and above: about 1 in 1,000 good messages score this high
  return res.status(202).end();
}
if (score >= 0.65) {
  // Your own review queue: about 1 in 280 good messages score this high
  return holdForReview(req.body);
}
// Deliver it as usual

Forms with nothing to read

Some forms carry no free text at all, such as an email-only signup. The model has nothing to read in them, so Classify answers clean with a score of 0, unless the model is very confident: a score of 0.8 or more stays spam with its score. A review queue at 0.5 never holds a signup for having nothing in it.

A form has free text when it holds any of these:

• A message or subject field, or a field whose name says nothing and whose value runs to five words or more.
• Any word in a name, company, address or other field.
• A link in any field other than email and website.
• An email field that holds anything but one address, or a phone field that holds anything but a phone number.

A string sent as content is a message, so this never applies to it.