Skip to main content

Improve the accuracy of our genre & topic system (plus add themes, tropes, and moods).

— Ben Fox

Improve the accuracy of our genre & topic system (plus add themes, tropes, and moods).

What does this feature do?

Here is how our current system works:

  • Genre - We pull genre from the data publishers provide our API partner. Unfortunately, this is not always accurate, and it is limited by an older system the industry uses called BISAC.
    • For the genre system, I give a B+ in terms of accuracy. Publishers make mistakes, and sometimes, they misuse the categories because they think it will help them sell more books.
  • Topic - We pull the topic from the LOC when it is available; otherwise, we use a free API called Wikifier (https://wikifier.org/). We feed it some data, and it tries to tie that book to Wikidata topics.
    • For the topic system, I give it a B- for nonfiction books and a D for fiction books. It just isn't as accurate as I would like.

Here is a longer writeup on my build diary if you are curious...

What are we adding?

Switching to a newer AI system

We will switch the genre and topic system to a newer AI system. So far in testing, it has performed better than publisher data. This will allow us to improve accuracy and create the genres that readers use now (as BISAC can be dated and imprecise).

In the long term, I hope to open this up to authors and our librarians to fine-tune. Slowly but surely, we will get there 😀.

Add theme, mood, and trope.

We will also add theme, mood, and trope analysis for each book. I've wanted this for a long time, and I look forward to adding it.

Revamp the entire system on the backend.

A massive chunk of this work is rewriting how this system works based on what we have learned in the last three years.

Status

This feature is not yet started.

Why is this update important?

  • This system is Shepherd's heart and powers everything we do. Getting this ride is crucial in our mission for readers and authors.
  • This feature will allow us to expand from 3,000+ to 10,000+ bookshelves and help readers find what they are searching for. Right now, we are missing a lot of genres/topics they would like to see because of the limits of the old system.
  • This will improve the accuracy of our personalized book recommendation email, which we will then launch out of beta.