Saturday, August 9, 2008

Cache your sessions. Don't piss off your users

I hope you're all enjoying the 1.2.6 stable release of memcached. Don't want to hear no whining about it crashing!

One of the most common questions in memcached land is the ever obnoxious "how do I put my sessions in memcached?". The long standing answer is usually "you don't", or "carefully", but people often walk the dark path instead. Many libraries do this as well, although I've seen at least one which gets it.

This isn't as huge of a deal as people make it out to be. I've been asked about this over the mailing list, in IRC, in person, and even in job interviews. What people end up doing gives me the willies! Why! Why why why... Well, I know why.

So what is the deal with sessions? Why does everyone want to jettison them from mysql/postgres/disk/whatever? Well, a session is:

- Almost always larger than 250 bytes, and almost always smaller than 5 kilobytes.
- Read from datastore for every logged in (and often logged out) user for every dynamic page load.
- Written to the datastore for every dynamic page load.
- Eventually reaped from the database after N minutes of inactivity.

Ok well that sucks I guess. Every time a user loads a page we read a blob row from mysql, then write a blob row back. This is a lot slower than row without blobs. Alright, so I see it now. Memcached to the rescue!

Er, except maybe it's a little complicated to actually memcached these things, since we need a write for every read... Why not just use memcached for sessions!? It lines up perfectly! Check it out:

- Set a memcached expire time for the max inactivity for a session. Say 30 minutes...
- Read from memcached.
- Write to memcached.
- A miss from memcached means the user is logged out.

Voila! ZERO reads or writes to the database, fantastic! Fast. Except I really don't like the tradeoffs here. This is one example where I believe the experience of both your users and your operations team is cheapened. Users now get logged out when anything goes wrong with memcached! Operations has to dance on eggshells. Or needles. Painful.

- Evictions are serious business. Even if you disable them (-M), out of memory errors means no one can log into your site.
- Upgrading memcached, OS kernel, hardware, etc, now means kicking everyone off your site.
- Adding/removing memcached servers kicks people off your site. Even with consistent hashing, while the miss rate is low it's not going to be zero.

So now what? Well we have zero accesses on our database, so it's fast! But we can't ever touch memcached again in fear of ticking off users. Progress be damned! Before you all think I'm completely off my rocker, I will admit there are some legitimate reasons to do this. If the way your site works doesn't really impact users on loss of a session, or impacts few enough users, you can use this design pattern. How many people are actually affected if you get logged out of wikipedia.org? Well, the people writing revisions certainly mind, but the greater userbase is unaffected. They're a non profit, they understand the tradeoff, etc. So that's fine. It's not fine for a lot of the people I see suggesting it or doing it. As developers get more comfy with memcached the session issue will become more of an obvious bottleneck.

The memcached/mysql hybrid really isn't that bad at all. You can get rid of over 90% of the database reads, a lot of the writes, and leave your users logged in during rolling upgrades of memcached.

First, recap the components involved: The page session handler itself, and some batch job which reaps dead sessions. For small websites (like a vbulletin forum) these batch jobs are often run during page loads. For larger sites they will be crons and so forth. This batch job can also be used to save data about sessions for later analysis.

The pattern is simple. For reads fetch from memcached first, database second. For writes write to memcached, unless you haven't synced the session to the database in the last N seconds. So if a user is clicking around they will only write to the database once every 120 seconds, and write to memcached every time.

Now modify the batch job. Crawl all expired sessions, and check memcached for the latest data. If session is not really expired don't expire it then, if it is use the latest possible data from memcached. Write back to the database. Easy.

You take the tradeoff of sessions being mildly lossy for recent information, but you gain reliability back in your system. Reads against the database should be almost nonexistent, and write load should drop significantly, but not as much as reads.

So please, if you run some website I might eventually use, don't put memcached in a place where restarting individual servers might piss me off. Thanks :)

I'd like to also challenge maintainers of session libraries for all languages to turn this design pattern into tunable (note all the places where I wrote N) libraries folks can plug in and use.

The more standard this stuff is the more likely the next fancy startup is going to get it right. Reuse is a great thing. I can't say enough about how great efforts like [info]krow's libmemcached go for standardizing how we use memcached, but it's also a great help to ship libraries for common design patterns...


Complete Story

0 comments:

Sign up for PayPal and start accepting credit card payments instantly.
ILoveTux - howtos and news | About | Contact | TOS | Policy