13
\$\begingroup\$

I'd like to see an answer on the 2018 community survey about user retention, as it's clearly an important issue. Unfortunately, I'm unfamiliar with (or lack access to?) the various tools to put one together myself. I'm left with only my own vague impressions, which are a lot less useful than data. So instead, I'd like to ask:

How is the main site doing with user retention?

This is intended to cover both converting one-shot users to recurring ones, as well as retaining active yearlings. A good answer would bring statistics to bear on this, preferably broken down into some kind of demographic.

\$\endgroup\$
  • \$\begingroup\$ I'm much less familiar with tags on meta than main site, so suggestions are very welcome. \$\endgroup\$ – Joel Harmon May 2 '18 at 1:31
11
\$\begingroup\$

It's possible to get at this kind of information with StackExchange Data Explorer.


When users sign up, how many of them come back?

This query computes the proportion of users active in the last 365 days who logged back into the site at least 7 days after they created their account for the first time. It turns out that, as of this posting:

80.0% of users from the last year stuck around for at least a week.

To provide a bit of context for this number, I ran the same query on a few other StackExchange sites (I chose ones that are similar to rpg.SE in either size or subject matter, plus a couple of particularly high-visibility sites). Here are the results:

  • StackOverflow - 85.9%
  • codereview.SE - 82.9%
  • worldbuilding.SE - 74.2%
  • Arqade - 67.6%
  • cooking.SE - 64.2%
  • judaism.SE - 62.6%
  • interpersonal.SE - 60.2%

...so it looks like we have better retention than most other similarly-sized sites and most other "hobbyist" sites, but slightly lower retention than the more professional/work-related sites. Not sure what exactly to make of this; perhaps people are more invested in getting answers to questions that they need for their jobs?


How many users remain active year-over-year, and how does this change over time?

This query computes the proportion of users who posted in one year who also posted in another year; I've used it to estimate year-over-year retention since the site started in 2010. Each row gives the percentage of the users who posted in the first year who also posted in the second year (so the first row is the proportion of 2010 posters who also posted in 2011):

  • 2010 - 2011: 35.8%
  • 2011 - 2012: 33.9%
  • 2012 - 2013: 27.1%
  • 2013 - 2014: 30.6%
  • 2014 - 2015: 26.6%
  • 2015 - 2016: 25.5%
  • 2016 - 2017: 26.7%
  • 2017 - 2018: 18.7%

I wouldn't take the exceptionally low final entry too seriously here (it's probably just an artifact of the fact that 2018 isn't even halfway over yet), but even ignoring that entry we see a slight downward trend over time.

This isn't too surprising to me. According to our Area 51 proposal, this site graduated six years ago (it's proving oddly difficult to find the exact date). I'm guessing those first two years represent high-buy-in beta users being more engaged with the site than the average user from after graduation. Ignoring those two years, year-over-year retention looks flat, rather than downward-trending.

A quick check of the 2016-2017 numbers found that we're similar to judaism.se (25.7%), and higher than scifi.se (16.0%) and codereview.se (18.1%). The query timed out on Stack Overflow; guess they have too many posts!


These are pretty simplistic queries (I don't know SQL at all, I just threw these together by frantically Googling documentation for a bit). I invite anyone more skilled to improve this answer (or submit their own)!

For those who don't know SQL, I'm also happy to take requests if you have specific questions you'd like investigated (with the caveat that I still don't really know SQL either, so I might not be answer everything).

\$\endgroup\$
  • \$\begingroup\$ Is it possible to automatically compare this to other SE web-sites? \$\endgroup\$ – Baskakov_Dmitriy May 10 '18 at 22:49
  • 1
    \$\begingroup\$ @Baskakov_Dmitriy I don't know how to do it in an automated way (as far as I can tell, you have to select which SE site you're asking about when you run the query). Somebody with more expertise on data.SE might know better. But manually comparing us to a few sites wouldn't be hard; I'll try to get to it today. \$\endgroup\$ – A_S00 May 11 '18 at 16:49
  • \$\begingroup\$ @Baskakov_Dmitriy Updated with a few site-to-site comparisons. Lemme know if there are any other specific sites you're curious about. \$\endgroup\$ – A_S00 May 11 '18 at 22:53
  • \$\begingroup\$ Would be probably nice if we see the users being active year-over-year from Arcade and Worldbuilding. BTW, what does qualify for "posting in this year"? Writing a question or an answer? Does a comment count? Your job is awesome anyway, though! \$\endgroup\$ – Baskakov_Dmitriy May 12 '18 at 8:58
  • \$\begingroup\$ As far as I can tell RPG.SE graduated on 2012-Apr-11. \$\endgroup\$ – Rubiksmoose May 16 '18 at 1:36
6
\$\begingroup\$

Sure! So, we're not free to share the specifics of the diamond moderator version of the site analytics with the public. And beyond that, we don't have a specific "bounce rate" metric.

What I can say is that there's a healthy growth in new user rate, which is at an all time high over the last three months. There was a significant lull last Sept/Oct but since then it's grown at a significant rate month over month.

We also have a visitor and active user stat that is also showing growth. Users that visited at least one day in the month tracks the new user stat. Frequent visitors and occasional actors are maybe 1/3 of that, and then frequent actors is the lowest but showing steady growth. These numbers are also at their all time high over the last three months.

That's all the data we have at our disposal. What we don't really have is a "conversion rate" or a "bounce rate". New users don't become active users for a lot of reasons and we don't have the data to determine more than that really, the site analytics even mods get is super limited - not "Google Analytics," but "a dozen line graphs". But our numbers are at historical highs, so that's good.

This doesn't mean we shouldn't work on being welcoming and retention of course. But that's a different question.

\$\endgroup\$
  • 1
    \$\begingroup\$ "we're not free to share the specifics of the site analytics with the public." Is that a thing? At 20 K I got this message: "Since this is a restricted privilege, we'd prefer you not share the raw data. However, we do hope you will share your analysis of the data with other users on meta. For instance, mature Stack Exchange sites tend to get the majority of their traffic from search engines (and particularly Google). Sharing the current percentage of traffic from search might shed some light on your site's particular strengths or weaknesses." \$\endgroup\$ – Please stop being evil May 10 '18 at 4:19
  • \$\begingroup\$ It also says "please don't feel any obligation to share this data", but the general feel of the thing is very much 'you don't have to share if you don't want to and a data dump would be inappropriate' and not at all 'don't share this even if you have reasons'. \$\endgroup\$ – Please stop being evil May 10 '18 at 4:20
  • \$\begingroup\$ @thedarkwanderer The problem is, I guess, in the phrase "we're not free to share the specifics of the site analytics with the public". Of course, there is no obligation to share anything. \$\endgroup\$ – Baskakov_Dmitriy May 10 '18 at 22:53
  • 1
    \$\begingroup\$ @Baskakov_Dmitriy Yeah, the problem is that phrasing makes it sound like if I wanted to look some stuff up and present it that'd be against the rules, where actually it's just supposed to be not anyone's job to do that (in my understanding). \$\endgroup\$ – Please stop being evil May 11 '18 at 0:15
  • 1
    \$\begingroup\$ I don't know what y'all are talking about, but the moderator analytics section we have access to says prominently at the top, "analytics data is intended for moderators only; please don't share the specifics of this data in public". So we're not sharing it in public. Thanks. \$\endgroup\$ – mxyzplk says reinstate Monica May 11 '18 at 3:19
  • 1
    \$\begingroup\$ We're talking about two different things. If the Moderator Analytics page has anything relating to new users, then it's different from the 25k+ User Analytics page. The user one has very little information on it, and the help page for it has the text that @thedarkwanderer is quoting. \$\endgroup\$ – DuckTapeAl May 11 '18 at 23:59
  • \$\begingroup\$ Mods get "a dozen line graphs", high rep users get 3. \$\endgroup\$ – DuckTapeAl May 12 '18 at 0:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .