Django

We support the pylibmc memcache client as it has great performance and Python 3 support. However, it can sometimes be difficult to install locally as it relies on the C libmemcached library. If you prefer, you can try a pure python client, python-binary-memcached. You’ll also need the django-bmemcached package.

Here we explain how you setup and install MemCachier with Django. Please see the Django caching guide for how you effectively use MemCachier. Django supports whole site caching, per-view caching and fragement caching.

MemCachier has been tested with the pylibmc memcache client. This is a great client, fully-featured, high-performance and Python 2 & 3 support. As of Version 1.11 Django has out-of-the-box support for pylibmc. Older Django versions require django-pylibmc to work with MemCachier. Please follow the instructions in this example if you wish to use an older version.

The pylibmc client relies on the C libmemcached library. This should be fairly straight-forward to install with your package manager on Linux or Windows. For Mac OSX users, homebrew provides and easy solution. We also have a blog post for Ubuntu users on how to do this.

Once libmemcached is installed, then install pylibmc:

$ pip install pylibmc

Be sure to update your requirements.txt file with these new requirements (note that your versions may differ than what’s below):

pylibmc==1.5.1

Next, configure your settings.py file the following way:

servers = os.environ['MEMCACHIER_SERVERS']
username = os.environ['MEMCACHIER_USERNAME']
password = os.environ['MEMCACHIER_PASSWORD']

CACHES = {
    'default': {
        # Use pylibmc
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',

        # TIMEOUT is not the connection timeout! It's the default expiration
        # timeout that should be applied to keys! Setting it to `None`
        # disables expiration.
        'TIMEOUT': None,

        'LOCATION': servers,

        'OPTIONS': {
            # Use binary memcache protocol (needed for authentication)
            'binary': True,
            'username': username,
            'password': password,
            'behaviors': {
                # Enable faster IO
                'no_block': True,
                'tcp_nodelay': True,

                # Keep connection alive
                'tcp_keepalive': True,

                # Timeout settings
                'connect_timeout': 2000, # ms
                'send_timeout': 750 * 1000, # us
                'receive_timeout': 750 * 1000, # us
                '_poll_timeout': 2000, # ms

                # Better failover
                'ketama': True,
                'remove_failed': 1,
                'retry_timeout': 2,
                'dead_timeout': 30,
            }
        }
    }
}

The values for MEMCACHIER_SERVERS, MEMCACHIER_USERNAME, and MEMCACHIER_PASSWORD are listed on your cache overview page. Make sure to add them to your environment.

After this, you can start writing cache code in your Django app:

from django.core.cache import cache
cache.set("foo", "bar")
print cache.get("foo")

You may also be interested in the django-heroku-memcacheify pip, which fully configures MemCachier with one line of code for any Django app the pip supports.

A confusing error message you may get from pylibmc is MemcachedError: error 37 from memcached_set: SYSTEM ERROR (Resource temporarily unavailable). This indicates that you are trying to store a value larger than 1MB. MemCachier has a hard limit of 1MB for the size of key-value pairs. To work around this, either consider sharding the data or using a different technology. The benefit of an in-memory key-value store diminishes at 1MB and higher.

Template fragment caching

Django allows you to cache rendered template fragments. To enable fragment caching, add {% load cache %} to the top of each template caching is used in. The control statement to cache a fragment has the form {% cache timeout key ... %} where all additional parameters after the key are just appended to the key. In practice this may look as follows:

{% load cache %}
<!-- ... -->

<!-- Fragment caching example -->
{% for item in list %}
  {% cache None 'item-fragment' item.id %}
  <div>
    <!-- fragment that does something with the item -->
  </div>
  {% endcache %}
{% endfor %}

Here the timeout is None, but it can also be a variable that contains a time or an integer denoting seconds.

The cached snippet from the above example can be invalidated (deleted) as follows:

from django.core.cache import cache
from django.core.cache.utils import make_template_fragment_key
key = make_template_fragment_key("item-fragment", vary_on=[str(item.id)])
cache.delete(key)

View caching

Django also provides a decorator to cache views:

from django.shortcuts import render_to_response
from django.views.decorators.cache import cache_page
# ...

timeout = 600 # 10 min

@cache_page(timeout)
def index(request):
  # ...
  return render_template('index.html', ...)

If a cached view ever has to be invalidated explicitly, the key to the view needs to be saved:

from django.shortcuts import render_to_response
from django.views.decorators.cache import cache_page
from django.utils.cache import learn_cache_key
# ...

timeout = None
view_keys = {}

@cache_page(timeout)
def index(request):
  # ...
  response = render_template('index.html', ...)
  view_keys['index'] = learn_cache_key(request, response)
  return response

Now the view can be invalidated with:

from django.core.cache import cache
cache.delete(view_keys['index'])

Session storage

Memcache works well for storing information for short-lived sessions that time out. However, because Memcache is a cache and therefore not persistent, long-lived sessions are better suited to permanent storage options, such as your database.

For short-lived sessions configure SESSION_ENGINE to use the cache backend in django_tasklist/settings.py:

SESSION_ENGINE = 'django.contrib.sessions.backends.cache'

For long-lived sessions Django allows you to use a write-through cache, backed by a database. This is the best option for performance while guaranteeing persistence. To use the write-through cache, configure the SESSION_ENGINE in django_tasklist/settings.py like so:

SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'

For more information on how to use sessions in Django, please see the Django Session Documentation