On Monday, January 9th 2012, that all changed…
We had been down for almost an hour before we were alerted to the situation. It quickly became clear that our current host was not going to be able to support our growing needs. We’d long grown tired of maintaining our fabric scripts and needed a host that would restart our servers instead of just killing them when we ran into problems. Moving to Heroku had become priority number 1. Over the next 5 days, we pulled 18 hour days to make the move to Heroku happen. It was no small effort, but in terms of long-term developer productivity, its the best thing we’ve done yet.
What follows is an overview of what we did and how we did it.
But first, a little about our stack.
Strategy: Minimize Moving Parts.
With any migration to a totally new set of infrastructure, there are going to be a lot of moving parts. Before we started making the switch the very first thing we did is create a new branch in git from the HEAD of our Master branch. Along with the migration, we agreed that there would be no new feature changes. Only changes to the infrastructure would be allowed. It got progressively more difficult to enforce this along the way but it made the whole process a lot easier.
Environment Setup on Heroku
Another point of interest is that most teams will have more than one environment. At Sendhub.com, we have the Production environment, One staging environment that is an exact replication of what is on production, and a “mini” staging environment for each engineer on the team. Getting this all set up was easy once we read Managing Multiple Environments for an app.
“Environment variables set with heroku config are persistent. They will remain in place across deploys and app restarts, so unless you need to change values you only need to set them once.”
$ heroku addons:add heroku-postgresql:ronin —app <APP>
Setting up postgress on your local system is just as easy on a Mac…
NOTE: Heroku automatically appends the db configuration to your settings.py file once you deploy your app. By running the promote command above, it will allow you to hook up the new db to your Heroku app once you are ready.
This turned out to be the hardest part of the whole process. Here is a step by step process on how we did it for the real migration.
1. Set our Site into maintenance mode, so that no one was allowed to write anymore data.
2. Next, we had to get a local copy of our production database. We did this by using SequelPro. Here are the steps for that:
a. Open Up SequelPro and Connect to your prod db, (the steps for this will vary based on who hosts your db). For us we used an SSH connection.
b. Once connected, use File → Export to export the db to a local sql file.
c. Create a new local database in SequelPro and import the file you just exported. There were some errors that occurred on import, which we just ignored as they were not going to affect us once we moved to Postgres.
b. SOLUTION: The solution to this was found by initializing our Django app against an empty PostgresSQL database. When we did this we noticed we weren’t getting any errors. When we did an SQL export of the working db, we found that our migrated data was missing sql statements that resembled the following…
- Create an amazon account and sign up for Amazon S3.
- Create a bucket for your production user files
- pip install django-storages
- Add “boto==2.1.1” and “django-storages==1.1.4” to your requirements.txt file
- To configure your django site to use Amazon S3 for the static and default file storage, Add the following lines to your settings.py file
STATICFILES_STORAGE = ‘storages.backends.s3boto.S3BotoStorage’
DEFAULT_FILE_STORAGE = ‘storages.backends.s3boto.S3BotoStorage’
AWS_ACCESS_KEY_ID = os.getenv(‘AWS_ACCESS_KEY_ID’);
AWS_SECRET_ACCESS_KEY = os.getenv(‘AWS_SECRET_ACCESS_KEY’);
AWS_STORAGE_BUCKET_NAME = os.getenv(‘AWS_STORAGE_BUCKET_NAME’);
- Set the appropriate keys in your heroku app’s configuration using “heroku config:add key=value” To test this locally you will also have to set these values in your .bash_profile
- Setup s3cmd. You will use this to upload the files to amazon s3.
- When the site was in maintenance mode, log into the production server and create an archive of all of the user uploaded files.
- We used CyberDuck to pull down the archive locally.
- Extract the archive on your local machine and cd to where those files are.
- $ s3cmd put —recursive user_files_dir s3://my_prod_bucket_name/files/
Why CloudFlare made this easy
Switching our domains over to Heroku might have caused some serious downtime, but thankfully we were already using CloudFlare to manage our DNS (and speed up the site). Before starting deployment, we added our domains to Heroku so our apps were ready. All we needed to do was edit the IP address of our A records and add the necessary CNAME records for our subdomains.
As all but one of our subdomains were used internally, we set those up to point at the relevant Heroku apps ahead of time. When it came time to switch over the production domain, we only had to edit 3 A records and the WWW subdomain. If you’ve ever used a domain registrar’s DNS tools you’ll know they say it can take up to 2 days for the changes to propogate; with CloudFlare it was almost instant as our domain nameservers remained with CloudFlare and they updated the modified A records and CNAME straight away.
Overall the switch was not completely smooth and there were certainly some moments of stress, especially with the database migration, but the results speak for themselves. If you’re looking to scale your site, we highly recommend moving to Heroku.