A project to build with System One models

A labelling assistant that asks a person only when the model is unsure

Pre-label a dataset with a decision model and its probability, accept the confident labels after a spot check, and send only the uncertain rows to people, so labelling takes a fraction of the time.

Documents and dataIntermediate5 stepsSuggested by @biplov

The decision

Your labelling task as a typed question: a choice over the label set, a score for ordered labels such as severity, or a noul for yes/no labels. The probability decides who labels each row: the model when it is sure, a person when it is not.

How to build it

  1. Write each label with a one-line definition: the same guide you would give a person doing the labelling.
  2. Run the model over every row, and store its label and probability.
  3. Have a person label 200 random rows without seeing the model's answers, and find the probability above which the model agrees with them 98% of the time.
  4. Accept the model's labels above that line. Load the rest into your labelling tool, such as Label Studio or Argilla, with the model's answer shown as a suggestion.
  5. Spot check 50 accepted rows in every batch.

Make it better

Fine-tune on the labels people gave in System One Studio (systemone run studio), and run the remaining rows again: each round, fewer of them need a person. Publish the agreement rate with the dataset.

More projects to build

All projects

Documents and dataBeginner

Sort app store and product reviews into themes

Tag every new review with the themes it mentions, such as crashes, pricing, login or a missing feature, score its sentiment, and send the product team a weekly count instead of a spreadsheet to read.

4 stepswith gliner2-5-decide, julia-1 +1