SERMO RATINGS, MY STORY

When I was hired at SERMO, I was brought in to handle a complex B2B survey creation and fielding product called RealTime Pro. So it came to me by surprise when I met with the CEO on my first day that I would be heading up the design on the largest initiative our company would be facing over the next 18 months, a consumer-facing product called Ratings. While I was initially upset about having the rug swept under me, I came to love working on a product that turned out to be far more complex than I could have ever imagined. You may think you know your users and you may think you know your industry... but when you combine the two, there is a lot of room to learn.

My First Day at SERMO

It was 4 PM on my first day at SERMO and we were sitting down to our weekly product review session. We had the CEO, CTO, director of product, and VP of Sales (remote) sitting in a room together and my colleague, the director of product, starting presenting what would come to be known as the ratings modal.

As I took notes during the meeting and was told it was going to be my job to make the modal on brand. I took notes on other comments made by the CTO and CEO throughout the meeting. There was one key question that stood out in my mind at the end of the meeting, what is the goal of the ratings modal? The CEO answered,

The goal of the modal is to allow a doctor to provide an accurate rating on a drug

The design presented of the Ratings modal presented on my first day

That goal would prove to take multiple iterations and refinements before we got to the final product of the modal. The first iteration I created focused on 3 things:

  1. Introducing a stronger relationship between Drug and Indication
  2. A clear focus on the star ratings and diminishing the focus on comments
  3. Introducing our brand into the modal

We introduced this first iteration to a 10% segment of our membership and received generally negative feedback.

The first iteration i created of the ratings modal

We had missed on a key knowledge point of our doctors. Our doctors are experts on generic names, not brand names. We had failed to give our doctors a key reference point to provide accurate ratings. We also wanted to correct another piece of negative feedback, the ability to switch to a specific drug they do know. The next iteration I focused on was to correct these negative pieces of feedback

  1. Made the generic name a static element instead of a hover state for the grey "i" icon, providing more information to doctors.
  2. Included the ability to change drugs and search for a specific drug through a link in the top right side of the modal.
  3. Lowered the number of indication pill rows to two, lowering the amount of additional information that is not always necessary

After having done a previous rollout, we felt more confident in this iteration. We tested this on 25% on our user base. We received much better feedback, but we still found that we needed to improve a couple of features.

The second iteration we launched to our users

We had delivered a ratings modal that was almost delivering on its original goal. But before I was ready to move on I wanted to imporve on the one piece of negative feedback that we had been hearing.

What does Accessibility mean?

Our doctors have been trained to question everything. So putting a term as vague as Accessibility in front of them could mean hundreds of things. Is it patient accessibility at the pharmacy or hospital accessibility? Is it cost or availability? Is it coverage? We needed to provide all the information to the doctor before they could confidently give us a rating. I advocated for a just-in-time explanatory text that would leave the static state of the modal clean, while explaining any axis as the doctor encounters them. I didn't want to risk the chance that as we explain the most complicated axis that another axis turns out to need an explanation as well.

I also worked with the team to create a stronger emphasis on the selected state of "Don't Know." I was worried that the purple selected state was not visually different enough from the blue link to signify to the user that we had registered their click. I decided on a pill-style for the on state to clearly indicate that the user had selected don't know.

We launched this last modal to 50% of our user base for 5 days. After 5 days, a couple thousand ratings and no feedback on issues, we launched to 100% of our user base with confidence.

The third iteration and current version on the platform

Success or Failure?

It took us a few tries, but we eventually completed our mission. We created a modal that allows a doctor to accurately rate a brand name drug by indication across 5 axis. We averaged just over 10 ratings per session per doctor on our initial launch. We have collected over 400,000 ratings at this point. We found a new issue shortly after launch. While the doctor accurately could complete the task, we had an issue of stickiness. Our conversion rates slowly went down, as doctors didn’t want to rate that often. It is an issue we tried to address on the Drug page by making the rating experience less invasive. We have seen success on that, converting 3x as many doctors to rate.

Future

I wasn't happy with how noisy the modal was and was concerned that we may be over-loading the doctor with unnecessary information. I redesigned the modal with a focus on:

  1. Give the doctor ONLY the information they need to start a rating by collapsing the indication list into a dropdown.
  2. Inclusion of a future request, to include the separation of Ages to the rating experience
  3. A solution higher on the modal experience for a doctor who encounters a drug they have no experience with.

The design has been approved to test but has not been prioritized.

A future design for the Modal experience