How can we effectively measure user retention?

Question

I'd like to see an answer on the 2018 community survey about user retention, as it's clearly an important issue. Unfortunately, I'm unfamiliar with (or lack access to?) the various tools to put one together myself. I'm left with only my own vague impressions, which are a lot less useful than data. So instead, I'd like to ask:

How is the main site doing with user retention?

This is intended to cover both converting one-shot users to recurring ones, as well as retaining active yearlings. A good answer would bring statistics to bear on this, preferably broken down into some kind of demographic.

Answer 1

It's possible to get at this kind of information with StackExchange Data Explorer.

---

When users sign up, how many of them come back?

This query computes the proportion of users active in the last 365 days who logged back into the site at least 7 days after they created their account for the first time. It turns out that, as of this posting:

80.0% of users from the last year stuck around for at least a week.

To provide a bit of context for this number, I ran the same query on a few other StackExchange sites (I chose ones that are similar to rpg.SE in either size or subject matter, plus a couple of particularly high-visibility sites). Here are the results:

• StackOverflow - 85.9%
• codereview.SE - 82.9%
• worldbuilding.SE - 74.2%
• Arqade - 67.6%
• cooking.SE - 64.2%
• judaism.SE - 62.6%
• interpersonal.SE - 60.2%

...so it looks like we have better retention than most other similarly-sized sites and most other "hobbyist" sites, but slightly lower retention than the more professional/work-related sites. Not sure what exactly to make of this; perhaps people are more invested in getting answers to questions that they need for their jobs?

---

How many users remain active year-over-year, and how does this change over time?

This query computes the proportion of users who posted in one year who also posted in another year; I've used it to estimate year-over-year retention since the site started in 2010. Each row gives the percentage of the users who posted in the first year who also posted in the second year (so the first row is the proportion of 2010 posters who also posted in 2011):

• 2010 - 2011: 35.8%
• 2011 - 2012: 33.9%
• 2012 - 2013: 27.1%
• 2013 - 2014: 30.6%
• 2014 - 2015: 26.6%
• 2015 - 2016: 25.5%
• 2016 - 2017: 26.7%
• 2017 - 2018: 18.7%

I wouldn't take the exceptionally low final entry too seriously here (it's probably just an artifact of the fact that 2018 isn't even halfway over yet), but even ignoring that entry we see a slight downward trend over time.

This isn't too surprising to me. According to our Area 51 proposal, this site graduated six years ago (it's proving oddly difficult to find the exact date). I'm guessing those first two years represent high-buy-in beta users being more engaged with the site than the average user from after graduation. Ignoring those two years, year-over-year retention looks flat, rather than downward-trending.

A quick check of the 2016-2017 numbers found that we're similar to judaism.se (25.7%), and higher than scifi.se (16.0%) and codereview.se (18.1%). The query timed out on Stack Overflow; guess they have too many posts!

---

These are pretty simplistic queries (I don't know SQL at all, I just threw these together by frantically Googling documentation for a bit). I invite anyone more skilled to improve this answer (or submit their own)!

For those who don't know SQL, I'm also happy to take requests if you have specific questions you'd like investigated (with the caveat that I still don't really know SQL either, so I might not be answer everything).

Answer 2

Sure! So, we're not free to share the specifics of the diamond moderator version of the site analytics with the public. And beyond that, we don't have a specific "bounce rate" metric.

What I can say is that there's a healthy growth in new user rate, which is at an all time high over the last three months. There was a significant lull last Sept/Oct but since then it's grown at a significant rate month over month.

We also have a visitor and active user stat that is also showing growth. Users that visited at least one day in the month tracks the new user stat. Frequent visitors and occasional actors are maybe 1/3 of that, and then frequent actors is the lowest but showing steady growth. These numbers are also at their all time high over the last three months.

That's all the data we have at our disposal. What we don't really have is a "conversion rate" or a "bounce rate". New users don't become active users for a lot of reasons and we don't have the data to determine more than that really, the site analytics even mods get is super limited - not "Google Analytics," but "a dozen line graphs". But our numbers are at historical highs, so that's good.

This doesn't mean we shouldn't work on being welcoming and retention of course. But that's a different question.