Skip to main content

How to hire freelance writers

You haven’t been receiving articles I’m proud of in recent months. The quality of my writing has gone down. I’m sorry.

But I’m working hard to fix it. One idea I’ve had is to hire a pro. Someone smarter and more talented than me to write articles for you. I’ve been nervous about trusting someone else to do it. So I wanted to learn from an someone who has been successful at letting another person write for them.

Many of you have told me that you’re in a similar position. You just don’t have the time. Or you’re struggling to think of what to write. A freelance writer may be the answer.

Creating quality content on a consistent basis is the key to generating sustainable growth through content marketing.

Ruben Gamez is a master at working with freelancers to write articles for the Bidsketch blog. In this short interview we talk about things like:

  • How to find great people to write for you
  • How much to pay
  • How to evaluate writers
  • How to manage freelance writers working for you

Ruben and Colin

If you’re struggling to create quality content on a regular basis (like I am), this deep dive will be a beacon of light. We’ll look at the strategies Ruben uses on his blog and how he works with freelance writers to get quality content on his blog multiple times per week and drives predictable growth to his business.

(transcript is after the break)

Colin: I’m here today with Ruben Gamez, the founder of Bidsketch. And so Ruben, you and I met at…I think the first time at LessConf and then we…we met again recently at Bacon Biz Conf. I’ve always enjoyed our conversations about fast businesses, about content marketing and all of that type of stuff. So, I was excited to get the opportunity to chat with you about working with freelancers to do content marketing, which is the subject of our chat today. But first up, for people who don’t know you and maybe don’t know Bidsketch, what is Bidsketch and what can you tell folks about that?

Ruben: Sure. Well first, just thanks for inviting me to do this. This is pretty cool. Bidsketch is proposal software basically for freelancers, consultants, and agencies. They just use it to create…they create online proposals for their clients but we do have a lot of customers that also print those out to PDF and send them as well.

Colin: Cool. And how do people find out about Bidsketch? How do they discover your business?
Ruben: So, right now the two biggest sources of like leads and customers are SEO and content marketing. So, in content marketing. Yeah, it’s really been working for us lately and I say lately because I looked at. Just a few days ago I was looking Google’s Analytics and I was looking at the stats for a year ago like you know versus last month, right? And about a year ago, we were getting about something like 17,000 uniques…

Colin: Mm-hmm (affirmative).

Ruben: …a month. In like last month, lately it’s been about 50,000 uniques around. So most of that traffic is blog traffic and that’s helped like over the last year, double our paid customers like every month that we get.

Colin: Fantastic. And so. So, you mentioned SEO. You mentioned content marketing. I mean those things feel kind of related. How does SEO influence the content marketing side of things and vice versa? Do you pick a keyword and write for that or do you write some interesting content and figure out where the keywords go in that?

Ruben: So, they…they…definitely are related and they work together, right? Like, the three biggest. The three main things that work together are the e-mail lists …

Colin: Right.

Ruben: SEO focusing on certain keywords and then you know the blog, right, the content on there.

Colin: Hmm.

Ruben: So in the early days, the strategy was different than what it is now when it comes to SEO and the blog. So in the early days, it was very much about doing the…using the Google keyword tool to see what terms our customers or potential customers might be searching for, right? So maybe they’re searching…in about…like…they want to know about writing a web design proposal, right?

Colin: Mm-hmm (affirmative).

Ruben: So they type in “how to write web design proposal” or they type in something like “how to charge for web design” or things like that. A lot of “how to” posts. And that was great for bringing in that early traffic because we didn’t have an audience. We didn’t have a list.

Colin: Hmm.

Ruben: So, that just made it really easy for us to get targeted traffic to this post and then get those people on the list.

Colin: Mm-hmm (affirmative).

Ruben: Nowadays, we…we’re not so focused on SEO that way. We’re a little bit more broad. We do focus on certain keywords. We’ll actually go back and change some of the existing content, so right now the content and the titles and everything like that…it’s completely focused on making it shareable content, right, so that people retreat it and all that stuff, right? And then we’ll actually go back and then kind of change the titles of the posts to focus on certain keywords.

Colin: Interesting. So, you publish it initially as shareable and then later tweak the titles?

Ruben: Yeah. Yeah. Not all of them. Some. On a few. Basically, we’ll go through and do a little bit of researching and see if there’s something key there. We also have a few posts. We have several posts that have done pretty well to where we did actually, from the beginning, focus on specific keywords, right? Like freelance marketing is one of them that does pretty well. That was originally written with that in mind and but it’s also highly shareable.

Colin: Cool. So how often do you write? How often is there a new post on the blog?

Ruben: So, generally…try to have a new post about once or twice a week.

Colin: Mm-hmm (affirmative).

Ruben: Twice a week is the target. Lately it’s been one time a week. Originally it was like one post every two weeks and slowly because every post gets e-mailed out to the list so I’ve not been very comfortable with e-mailing people more frequently, right? I had to get a little bit comfortable with like e-mailing them weekly and then even two times a week felt like a big…like a big move to me, right? So, I just. Well, okay let’s test this out, right? Instead of just assuming that people were going to hate it or it’s too much e-mail and testing it out meant that we got a lot more traffic, a lot more we actually. Every time I send out e-mails I get customers, right?

Colin: Mm-hmm (affirmative).

Ruben: So we got more customers, more traffic. I tried out three, even, a week a couple times and I didn’t see. I didn’t see the impact that I saw from going. From going from one to two that I did, you know, when I did from two to three.

Colin: Did you get resistance from people? Did people e-mail you saying, “Hey, you’re e-mailing too much,” or anything like that?

Ruben: No, I’ve never gotten an e-mail that…from somebody saying that I’m e-mailing too much.

Colin: Okay.

Ruben: The only e-mails that I get about that, you know, anything promoting content is basically that they really like the e-mails.

Colin: Cool.

Ruben: That they really like the content. That’s it.

Colin: And so you’re sending. You’re writing like one to…one to two times a week. How often are you, Ruben, writing those posts?

Ruben: Oh, I’m not. I’m not writing any of them. It’s…it’s just way too time consuming. I used to write the posts in the early days, but it would take so long and just. I’m a slow writer and I don’t like doing it, so I don’t write any of the content. I just hire…hire people to do that.

Colin: Gotcha. And so how many like. So, in the early days you’re talking like a year ago or so or two years…two years ago and when did you start kind of…

Ruben: No. It’s like 3 ½ years ago.

Colin: So, 3 ½ years ago is when you started writing and then realized that you…you’re not the right person to be doing it? Your time is better maybe spent elsewhere and…

Ruben: Right.

Colin: So, how many different freelancers have you worked with over…over that time?

Ruben: A lot. So, way back in the beginning I was writing a lot of the content but then there was a period of time where nothing really was going on in the blog, right?

Colin: Hmm.

Ruben: So it was like a year maybe a year and a half like maybe just updates about the product, but you know which nowadays just thinking about it makes me cringe but there was not much going on and then there was a point where I said, “Okay, we need to get more traffic and this was working relatively well. Let’s do it again, but this time I don’t want to do all of the writing myself,” right?

Colin: Yeah.

Ruben: So, that’s how I started that and I started looking for…looking for…for writers at that point. I worked with, I don’t know. Like right now, I work with about five writers regularly?

Colin: Mm-hmm (affirmative).

Ruben: And I’ve hired, I don’t, 15, 20 more than that been or time tested them out.

Colin: So, how did you go about finding those people?

Ruben: The two best ways that have worked for me have been to actually just look at blogs that have great content and the type of content that you know that I want that’s also focused on my industry, on my customers and see which writers I like the best and pitch them, basically, to mean I’ll see if they’ll write for the blog. That works and I’ve, like, found a couple of writers that I work with regularly that way but it’s so time consuming and you get a lot of “nos”, right? If they’re really good. If they’re really good, you’ll get a lot of “nos”. And then sometimes too, depending on the blog, you have to be careful because sometimes blogs like Copy Blogger or I don’t know Pro Blogger, they do a lot of editing, so you’re not going to get the type of post that you think you’ll be getting.

Colin: Interesting.

Ruben: Right? Because I don’t want to do any of it. So, I want the post to be you know great right from the beginning without me getting involved at all.

Colin: Right.

Ruben: So that’s one way. The other way is then to actually through…go through referrals.

Colin: Right.

Ruben: …asking people who know great writers. And a good example of this is, I hired a writer who runs a content marketing podcast to come up with a list of I think it was 20 recommend writers. So, I told them about my market like what I was looking for and all this and it’s a way of short cutting…short cutting, right?

Colin: Mm-hmm (affirmative).

Ruben: And he just came back with a list of 20 writers. I paid him for it. It’s not what he normally does but there are a lot of people like that, right? That know more about like certain industries basically especially when it comes to content marketing that maybe you can hire like that.

Colin: Right. And so let’s say you found one of these people through a creative way like that.

Ruben: Mm-hmm (affirmative).

Colin: What’s the process you go through with them before a post that they write for you at Bidsketch blog?

Ruben: So I have a…I forget what I call it…but it’s like a customer profile document.

Colin: Mm-hmm (affirmative).

Ruben: Basically, it just lists the target customers, what industries they tend to work for you know. It’s like a two-page document with a…and it’s not like very wordy or anything like that we’re talking bold lists, big fonts and stuff like that but really media important stuff, right? Like, what are their biggest fears, what do they usually struggle with—basically just a document that tells them all about the target customers and the blog readers…

Colin: Mm-hmm (affirmative).

Ruben: so that they know who they’re writing for and then there’s a list of recommended…recommended topics…

Colin: Mm-hmm (affirmative).

Ruben: Right? So we’re talking about like it’s pretty broad nowadays. It’s just sales and marketing for freelancers, consultants and agencies.

Colin: Mm-hmm (affirmative).

Ruben: And that’s how broad.

Colin: Mm-hmm (affirmative).

Ruben: So, that’s what they get before them. I say, “Ok, we want the post to be about, anywhere from…at the minimum over 1,000 words…generally about 1,500 words or so.”

Colin: Hmm.

Ruben: So, longer media posts with lots of examples. I’ll give a couple of example posts.

Colin: Yeah. How…how did you arrive at that…at that length because that’s…I mean it…a 1,000-1,500 words is…is…you know that’s a longer post. You see a lot of like very short, short posts out there.

Ruben: Right. Right.

Colin: Is that? You found that the longer media posts get the best results for you guys?

Ruben: Yeah, try to also to different posts and it’s interesting when I try and hire a lot of writers like going on the pro blogger words and stuff like that and having them submit you know their post to me. I’ve never had a lot of luck that way but a lot of those writers are used to charging and writing around like let’s say 500 words or something like that, right? 500 or 800.

Colin: Mm-hmm (affirmative).

Ruben: Under 1,000. And that just hasn’t really done very well on our blog. Plus, I’ve looked at a lot of the other blogs that are…that really like…they’re doing really great on content marketing like the Kiss Matrix blog

Colin: Mm-hmm (affirmative).

Ruben: …or smashing…smashing magazine blog and they have just really meaty write posts. So, it’s just looking at what sort of content our potential customers or blog readers really like and it just tends to be super meaty stuff.

Colin: Hmm…that’s really…that’s really interesting. Cool. And so the topics that people are writing about or like the you know…do you usually write headlines for people or do you give people ideas when they’re writing that post for you?

Ruben: No. No. They have to. So, the writers that I hire are very experienced. If you go to their blog, they you know every single post gets a ton of retweets, a ton of comments and stuff like that. So, they have an audience. They know how to write great headlines and stuff like that. So, that’s the point. I don’t want to get involved in any of that, right? Because it takes time. It really takes a lot of time.

Colin: Mm-hmm (affirmative).

Ruben: Especially writing a great you know post title or headline. In the…before them I went through a period where I was hiring…I was just experimenting and hiring writers where this is… this is… really…a great writer and yes they might have been a good writer but they just didn’t write in that…the style that I needed which meant that I would have to go in there and I had to tweak the title, edit the post and every single time. It was just too much work.

Colin: Mm-hmm (affirmative). Okay. Cool. And so if you’ve got this post…you want…you’re trialing out a new person and…how do you…presumably you read through it the first time before publishing…

Ruben: Mm-hmm (affirmative).

Colin: that…that first post and let’s say…so let’s say it’s okay and you publish it. How do you kind of evaluate them after that? Is it like they satisfied Ruben so that’s good enough to go on the blog or do you kind of look at…look at their metrics versus other peoples’ metrics…are you going to that level?

Ruben: Yeah. I’m basically looking at the amount of like comments and shares that a post gets but it’s tough in the beginning because you’ll always…like even if you have somebody who’s great…can consistently can deliver posts that get a lot of comments and shares, they’ll have posts at that just don’t do as well, right?

Colin: Mm-hmm (affirmative).

Ruben: And that happens so you need…I need a few posts before I can tell whether or not, you know, we should continue working together. And yeah generally…they’ll…if I find somebody that is already doing it on their blog; they’re going to be good enough. They’ll deliver what I need.

Colin: And…and when you’re talking about paying for them to do articles for you, how do you…how do you do it? Do you do per article…per hour? What has worked best for you?

Ruben: So, I actually pay you mostly per post but I pay per hour. I still pay per hour for one of the writers so I’m open to paying any which way. It doesn’t really matter to me but most writers do seem to like to charge per post.

Colin: Gotcha. And if you’re working with good writers which sounds like you guys really are. How much should someone expect to pay per post for the blog?

Ruben: So, generally I like to stay around the $200 a post area. That’s like I was experimenting in earlier days with people that like I was paying $50 or $80 a post and then it wasn’t really until I got to the hundred you know like to the 100-150 that I got better quality and longer posts but that’s a different type of blogger that I’m getting better than what I’m working with now which is like the $200 you know 250…

Colin: Gotcha.

Ruben: …$250 range.

Colin: When I’m thinking to myself about how long it takes me to write an article, if I could get someone to write an article for $200 that’s a huge time savings for me.

Ruben: Oh yeah!

Colin: And I could devote that time to other things. So…

Ruben: Right, right. Like if it took me 8 hours or how many hours to write a post, right?

Colin: Wow.

Ruben: That’s a lot of time saved.

Colin: Yeah.

Ruben: Plus, like the amount of money that I’m spending a month on these posts, I make back within that first month, right? And since the product is a fast product means that every other month after that those new customers is just profit.

Colin: Gotcha. Yeah, just thinking about the economics, it sounds like a pretty good investment. If you’re going to get let’s say 1 or 2 people signing up per post you’re going to make that back.

Ruben: Yeah, yeah. The only thing is that it’s working this way now because I’m using a larger list as well so the strategy like is different when you have a larger list versus you don’t. So, if I had the exact writers and it was way back in the early days where I didn’t have a large list, I would not be making my money back so quickly, right? I’d be spending more money on. I’d probably be doing less posts on the blog…

Colin: Mm-hmm (affirmative).

Ruben: …like 2 or 3 a month and I’d be spending more on having people do guest posting for me.

Colin: Gotcha.

Ruben: to get new traffic. So, I do that now but I do less of that.

Colin: Interesting. Yeah and it sounds like the having that list where you can publish your content, get it out on the blog and then let people know about it, it’s a pretty key…

Ruben: Right.

Colin: … key part of the strategy.

Ruben: Right. Yeah. Basically.

Colin: Awesome. Well, this has been tremendously interesting. Do you…can you share any advice for maybe people who are writing or struggling to get their content out of the door right now and they’re considering hiring a freelancer to help. Any other advice for those folks?

Ruben: Sure. Yeah, I mean. I definitely would hire a freelancer. I know that it takes time. It takes practice. So, at the beginning you might not be so good at it. It doesn’t mean it doesn’t work you know so like you should quit you know it happens to a lot of people when they’re outsourcing anything whether it’s government writing or whatever.

Ruben: It takes a little time to get good at but also try and find people, like I mentioned, that are already doing what you need.

Ruben: Right?

Colin: Right.

Ruben: And don’t just depend on them. So, if they’re that good ask them for advice, right? Don’t try to micro-manage them, try to make them fit because you know I [inaudible 00:20:52] whatever that you’re seeing but you know give them a lot of freedom and ask them for advice on how to grow your blog.

Colin: Cool. Well thanks so much Ruben. Really, really appreciate this talk on how to hire a freelancer to do your content marketing so thank so much for taking the time to chat with me.

Ruben: No problem. It was fun. Thanks.

Message Composing

Customer Profile: Rocketlease

A few weeks ago, Eric Liu, the founder of Chicago-based RocketLease reached out to me with this really fantastic result:

16.2% of my customers respond to the emails I send through Customer.io

Eric was excited by the results. I was excited by his results. So I interviewed him to to learn more about Eric and his business.

If you have a business or are about start one, there are some fantastic lessons from the way Eric has built RocketLease.

Onwards to the interview:

What does RocketLease do?

Eric:: RocketLease is an app that lets landlords and property managers screen their applicants using an online form. It’s an alternative to a paper application process and a manual credit check process. RocketLease streamlines the process for landlords to get a new tenant into a property.

What does Rocketlease replace?

Eric: Right now, the majority of landlords are still using paper applications. Leasing agents and property managers will typically have a huge filing cabinet full of applications for the various people they work with. Every property manager might have a different application form. They’ll have stacks of manila folders with 20 or 30 applications, and then once they start running out they Xerox a bunch more and put it back in that folder.

When an applicant is interested in a property, they’ll give them a paper form and ask them to fill it out and then attach copies of their W-2, pay stubs, whatever information the landlord is going to require. Then they’ll take that resulting information on the paper document and type it into a system online for getting background checks or credit reports. That stuff is typically acquired from third party vendors and manually procured by the landlord or property manager.

What’s the competition like?

Eric: Real estate is a competitive space, but rental applications in particular are a really small niche. A lot of the companies that do rental applications aren’t specifically targeting rental applications.

For example, you have bigger companies that are trying to do full, like snout to tail, all property management. So applications to online rent payments, tenant management, repair tickets and that kind of stuff. They have a really different charge structure, and generally they’re ranking for stuff like property management software or something like that.

This is super niche and super focused. For example, leasing agents and landlords often don’t want a full solution because they just want to get someone in the door. Like they don’t actually manage rent payments and that kind of stuff necessarily. It’s a lot more niche and it’s more focused on that specific kind of landlord. I think it’s been a lot easier to rank for that than I would have initially guessed.

Why did you start Rocketlease?

Eric: Rocket Lease is largely borne of a personal need. A couple of years ago, I started investing in real estate. Me and a couple of buddies started buying multi-unit houses and renting them out in Chicago. I still remember the first time I got an actual applicant, and I was afraid. I didn’t want to get a deadbeat tenant. What I really want to do is make sure that I had an iron-clad guarantee that he was going to pay his rent. Of course that’s not really possible, but as a first time landlord that’s what you’re trying to get.

I talked to a friend of mine who is a leasing agent. He gave me his company’s rental application form. It was this super faded, barely readable document. It asked for anything you can possibly want about a person, and once I had my applicant fill that out, I was supposed to take that document and fax it to a company in Elmhurst. The only way I knew about this company was through him, and the only piece of information I had about the company was their fax number. I at that time didn’t even have a fax machine. I distinctly remember I had to go to Kinko’s to fax it. I’d write down my office number’s fax number so that I can receive the fax back.

The deal with this company in Elmhurst, I don’t even know where Elmhurst is. I know it’s like within a couple of hours of Chicago, but it’s like some suburb in Illinois. The deal was you would fax it to them and then within 48 hours they would fax back a credit report, background check, whatever they could about this person. I remember thinking like this paper process, faxing from Kinkos and then going back to pick up the credit report, was just ridiculous and archaic with a 48-hour turnaround time.

It kind of sat in the back of my head. It didn’t really turn into anything until about a year and a half ago. I remember when I was trying to figure out what the next step was. It’s like, “Well, this is a real problem. It’s a problem that I understand. It’s a problem that other people I’ve talked to understand as well.” It’s slowly turned into RocketLease.

Do you focus on search engine optimization (SEO) for Rocketlease?

Eric: Yeah. I’ve done a lot of work around trying to rank for what I think are available keywords. SEO is always an ongoing strategy for us. We also try to rank for terms like online rental applications. A friend of mine, Kevin and I scour all these real estate forums for landlords. We look at the kinds of questions that people are asking and we take each of those questions that a landlord might ask. We basically create some of the blog posts around that and we organize them into this repository on the Rocket Lease blog.

That’s kind of one of the channels. Generally terms like what’s the security deposit law in Texas aren’t going to be really competitive terms. It is easier to rank quickly for that, but the tradeoff is that you’re writing this article for a keyword that doesn’t have a ton of volume. It’s time intensive to create enough surface area to capture enough incoming links from those kinds search terms.

Have you tried any search engine marketing (SEM) like google adwords?

Eric: I’ve experimented a lot with SEMs. I think, for example, like Google AdWords is pretty close to being a valuable channel for me, but it’s really time intensive. If I pay other people to manage it for me then it’s not really a good ROI anymore. If I manage adwords myself, it just sucks out all my time and I think it’s not the highest value use of my time yet. Either I need to get a lot better at generating long term value relative to the cost so that I can spend on it, or I need to just burn it. Right now, I’m not doing anything with it.

Let’s talk a bit about your 16.2% response rate to your emails.

Eric: I remember being shocked at the time. I was like, “Okay, so basically one in six people who receive an e-mail from me is actually responding.” Like that was shocking to me and it was really important to me because I realized that I think that’s why a lot of my customers are staying onboard. Every time somebody e-mails me, I try to really, really understand their problems and make a personal connection with what they’re trying to accomplish with the site and help guide them through it and help them figure out the best way for them to use the product.

At the time, and even now, it’s resulted in a really heavy customer support load. What I find is that so my Customer.io e-mails are going out like right as a new landlord is onboarding.

What I want to do to push them through this activation funnel.

Eric: After they sign up I want them to:

  • Enter some information about a rental property
  • Try creating an application
  • Send out an application to an applicant
  • Then, after they sent one, I want them to obviously receive a completed application.

At each of those break points, I am sending out Customer.io e-mails to say things like, “Hey, I noticed you haven’t yet entered information about your rental property. Are you having problems? If so, call me here,” and I actually put my personal cellphone number and all that stuff. I say, “Call me here. I’ll walk you through it anytime.” Or I’ll say like if they have just signed up and they haven’t done anything, I’ll send them a note saying, “Hey, I just want to remind you, you signed up for this, but you haven’t done anything yet. This is why a lot of people do like it. If you have any questions, hit me up.”

At each of those levels, I do get responses from people who have fallen off. Once I get them through the activation for the first time, which is … so it’s really time-intensive for me to get into the first time, but after the first time I think it makes them your customers for life. After that, they require almost no support because they figured out the process. To me it’s really valuable investment to get this landlord in the funnel and he’s going to just keep using Rocket Lease in perpetuity.

Any parting words or things to plug?

Eric: Well, if you’re a landlord, use Rocket Lease. More importantly, 35% of Americans are actually renting property. You might not be a landlord, but you for sure know somebody who is a landlord or you know other people who are renters who know landlords. Word of mouth has been really, really big for Rocket Lease. Let people know that they should be using RocketLease instead of paper applications.

Thanks Eric Liu for taking the time to chat with all of us!

Four email A/B tests to improve results in five minutes today

This article is about A/B testing. Promise.

Imagine receiving a frantic call from your mother while at work: “Why didn’t you tell me you were dying?”.

This is the call my friend’s coworker received a few weeks ago. His mother had just gotten off the phone with the health insurance company. They had called to say that the test results were in and it was bad news. The guy left the office that day in a state of confusion and terror. He had gone to the doctor recently. They had run some tests. Was he really dying?

No. It was a horrible mistake. Turns out that the health insurance company had mixed up records. Then they had called his mother and disclosed private (not allowed) and incorrect medical information. Can you believe it?

This story popped in to my head as I debated titles for this article like: “Your test results are in” or “Has your copy gotten tested recently?”.

I couldn’t do it! So I went with “Four email A/B tests to improve results in five minutes today”

Here they are. And I hope you never have an experience like my friend’s coworker!

Test 1: Try sending emails from your CEO or Marketing Manager

We recommend that your from address is something like: name from company. For example: Colin from Customer.io.

Customers have told us (and our experience is) that people respond better when an email comes from a real person. (You could always test that vs just your company name)

Try testing some variations of your from address. What if an email comes from the CEO? Or the marketing manager? Or your community manager?

How does that affect your open rates and the number of responses?

Test 2: What if your subject lines are longer, shorter, funny or serious

Subject lines are always fun to test. You can get your creative juices flowing.

Get out a piece of paper and scribble down stream of consciousness subject lines for one of your emails.

Here are a few that hit my inbox recently:

Do your customers love you more than they did 6 months ago?

Why I said no to a million bucks

Can you pick a winner?

Do you know if your audience likes a long subject line or a short subject line?

Test 3: Personalization in your email

Will using the recipient’s first name in the subject increase opens?

What about sprinkling it throughout the content of the email you’re writing?

Light personalization

Too much personalization can seem cheesy or forced, but a little bit sprinkled in your email can make people feel more comfortable with your business.

Name is an obvious one. Another idea might be if you have other billing information for the person like city or country, you can use that to personalize content. Like: “How’s the weather in Oslo today?”.

Test 4: Find out which content drives more clicks & conversions

Once you’re happy with how many people are opening the emails you’re sending, you want to start looking at content testing.

You should wait because to get meaningful results for content, people first have to open your emails! So get those open rates up before trying things like:

  • Content Length (long vs. short?)
  • Lots of Images vs. Text Only (here’s one test)
  • Text link? Or large button?
  • One option to click through or lots of links to the same place?

Where you take your email testing is really up to your imagination.

What are some email A/B tests you’ve run (or want to run)?

Shout your successes from the hilltops in the comments below.

Happy Emailing,
Colin

What email looks like in 5 years

Show images to see this i love email button

Hello from San Francisco!

Litmus hosted the Email Design Conference in San Francisco on Monday and Tuesday and it was fantastic to be around so many smart people in email. If you missed it, it will be in London and Boston very soon.

I gave a talk entitled “How customer-centricity is changing email”. The ideas in the talk are mostly from conversations with you about what your big pains are in your business that relate to email and data.

After letting feedback over the past 2 years digest, I distilled the ideas down in to three areas:

Multi-channel aggregation

People interact with your business on twitter, facebook, phone, support email, mobile app, website.

Wouldn’t it be great to pull all that data in and build a unified view of a customer?

Multi-channel image

That’s what CRM has promised but fails to deliver on. CRM today is largely manual data entry. Nobody likes manual data entry.

We’re going to see a unified view of the customer that’s available in all the places you need it. Places like customer support, your email platform, your sales tool.

This is a big deal, and hard to do. I’m excited to see what people like Trak.io end up doing on this front.

Real-time targeting

Once you have a unified view of a customer and all of the great data coming in, you want to be able to act on that data.

A lot of companies today (especially larger ones) have a week turn-around time (or more) to send a targeted email. That’s painful for you, the people responsible for sending emails. It also leads to a lot of poorly targeted emails with poor open, click and conversion rates.

Here’s an early design from our designer Steve that shows where we’re going with our thinking about the interface to manage these things:

Convert more people

Combine real-time targeting with multi-channel aggregation and you’ll be able to easily do things like email all the people who tweeted about us in the past 10 days and emailed support. Customer.io today does part of that, but the more data we have the more powerful it is.

Real-time targeting makes sure that content goes to the right people at the right time.

Recommendation engines

The last piece of the puzzle is content recommendations. You’ll be able to use all of the great data you’ve collected to give people better content in their emails.

Companies like Netflix, Amazon, Twitter and LinkedIn all use recommendations to great success, but they’ve built them in-house. For Netflix, 75% of what people watch is the result of a recommendation.

Netflix thinks I like dictators

For you to build something in-house you’re looking at spending $500,000 or more. I think we’ll see more companies focusing on content recommendations.

For example, in the Shopify app store, you can use Directed Edge to do recommendations for your Shopify store. In the next 5 years, I’m looking forward to easier recommendations both on-site and in email.

These 3 things combined are going to decrease email volume per person, but increase the quality of email people receive and how much value you’re going to be able to give them.

That’s a short summary of what was covered in the talk. What do you think? Agree? Disagree? Is there something critical that’s missing from this suite of tools for you?

Let everyone know in the comments.

Sincerely,
Colin

P.S. I know this is late notice, but if you’re in San Francisco, I’m hosting a happy hour for our customers tonight from 6 – 8 at Mars Bar. I’d love to meet you.

Designing for Email

Yesterday I spoke with Justine Jordan. Justine is the Marketing Director at Litmus.com, a product that allows you to test email designs across all the clients you could imagine as well as providing spam testing and a whole host of other services.

The interview covers some topics I hope you’ll find valuable like:

  • How should a new company approach setting up their email program?
  • Some insight Litmus’ marketing strategy.
  • How to think about responsive designs and mobile when designing your emails.

You’re really going to enjoy all the wisdom Justine has to share.

Watch the full interview below:

Show Notes

Full transcript

Colin Nederkoorn: Hey everybody, I am here with Justine Jordan. She’s the Director of Marketing at Litmus.com. Justine, I’d love to hear a little bit about what Litmus is.

Justine Jordan: Sure. Litmus is a small software company based in Cambridge, Massachusetts. We build software to help email marketers make their emails better. We do some rendering testing. We take screenshots of what your emails look like in a variety of different email clients and mobile devices. We also offer a spam filter test, so you can see if you’re going to hit any spam filters before you send, and then, we also offer an analytics service that lets you know where people are opening your emails, how long people read them, and if they printed and forwarded, so some cool behavioral data around email too.

Colin: Cool. Your role there, you’re the Director of Marketing. What does that entail? How many people are on the team with you? Tell us a little bit about that.

Justine: Sure. I frequently make the joke that my job is whatever needs to get done that day. I’m sure a lot of you can relate; as a small business, a lot of us wear a lot of hats. I joined the Litmus team almost three years ago, and I was the first official marketing hire that the company ever had. For a while, probably about a year and a half, I was a one woman show. I did everything from writing blog posts to helping handle customer support issues, to jumping on demos when prospective customers asked for them, to writing and designing and coding emails; basically just like I said, [Indecipherable 1:45] , so I guess “marketing” is a really broad term in the Litmus universe.
Today, we have three people. We have myself. We have a Marketing Coordinator named Lauren, and then a brand new hire who I’m very, very excited about named Kevin, a Content Designer, that’s helped ease the design load here in the marketing team at Litmus.

Colin: Your background is actually more design than marketing, is that correct?

Justine: Absolutely. I have a BFA, a Bachelor of Fine Arts. I went to an art school in Indianapolis, Indiana. My passion, or my dream, was to become a brand manager one day. I spent my first year out of school designing brochures and other things that you print and physically hand to people, before I found a job doing mail design. It was there that I really fell in love with this idea of a designer being able to be so connected to their audience, and getting immediate feedback on their designs. I’ve never truly been that creative of a person, so I always felt like kind of a fake as a designer.
I really liked email marketing because it let me flex my analytical, rational, logical muscle, while also still being creative.

Colin: When you think about the marketing strategy for Litmus, what are the different pieces that you guys have in place? What are the different ways that you reach your audience?

Justine: Strangely enough, email isn’t one of the primary methods, although it is one of the ones that drives the most traffic. As all [Indecipherable 3:31] startups can probably relate, it’s really hard to make good email with a small team, so we really focus on quality over quantity, although we’re hoping to change that.
I think a lot of people are surprised when they hear that our marketing maybe isn’t as sophisticated as they assumed it was. [laughs] Again, I definitely think we take the quality over quantity approach. Email is definitely a huge driver of traffic to our website, which is a primary indicator of making sales and money, which is a good thing.
But Twitter is really big for us, and social media in general. I’m really surprised every day, looking at Google Analytics, the number of visitors that Facebook drives our way. We primarily spend a lot of our time blogging and creating content for the blog. All that content on the blog goes into our newsletter, and then to Tweets and Facebook posts, and all those things are intended to drive traffic back to our site.
It’s a pretty simple model, but it’s been working really well for us.

Colin: Hey folks, at this point we had a brief interruption in our Internet connection, so we jump back into the interview a few minutes later. Let me see, where were we?

Justine: I think we were talking about emails. [laughs]

Colin: Yeah. That’s a great transition into the next thing. Oh, I know what I was saying. Right as soon as we cut out, I was saying how much I love the emails that you guys put together. I was really curious about the process that you guys go through to build those emails. They’re really colorful. You have these different sections, and they look beautiful on my desktop, on my mobile phone, which is really what Litmus helps its customers to do.

Justine: Yeah. I’d say one of the biggest challenges of doing email marketing for Litmus is the whole “eating your own dog food” thing. Marketing to marketers, or especially doing email marketing to email marketers, is really interesting sometimes. Every time we send an email out, we get a ton of praise, but we also get a ton of really critical comments about, “You should have done this,” “You should have done that.” “You should be practicing what you preach,” yadda, yadda, ya.
I have too much to say about this topic. [laughs] Our emails weren’t responsive up until very recently, when Kevin joined us a month ago. Occasionally, our emails were responsive, but our monthly newsletters were not. If I did an email or a webinar on mobile email, I knew that if I didn’t do a follow up email that was responsive, I was going to hear about it later.
Those emails in the past have definitely been responsive. Others like our newsletter, the turnaround was too quick. They were too complicated, and frankly. our stats didn’t support us doing mobile optimization, which is the funniest thing of all.
If people really dug a little deeper or thought a little more, they would see that we were practicing what we preach, because although we talk about mobile optimization a lot, we also talk about how it’s important to know your audience and know if it’s the right fit for your company.
It’s fun when you get to use your own software; you can probably attest to this, Colin.

Colin: Yup. [laughs]

Justine: It’s always fun when you can use your own software to do your own company’s business, so we use our own email analytics to look and see how many people are opening our emails on mobile, and it’s never really gotten above 15 percent that’s one five whereas the average that we’re seeing overall is more like 45. We’re way below average in terms of our mobile opens, and we’re a really small team, as I mentioned before, and it just was never really worth the effort to mobile optimize.

Colin: Let’s hypothesize about that for a minute. You’re saying you get 15 percent mobile. Do you think that’s because you’re a tool for businesses, and people are reading your emails during the workday, while they’re in the office?

Justine: Absolutely. I think that we’re a B2B company, so the majority of our opens I want to say like 60 percent come from Outlook and Apple Mail, which are desktop clients. I’m assuming that the Outlook users are primarily the more corporate or the more marketing based of our users, whereas the Apple Mail folks are primarily our designer audience, or the creative audience, that tends towards Mac. It makes a lot of sense, when you really think about it. Yeah, we have very few mobile opens, and even fewer mobile website visits, which is why our website isn’t responsive yet, either. Being small, you have to be smart about where you use your resources, and it just hasn’t made sense for us to do the mobile bit, up until now.

Colin: Cool. Yeah, I think that’s a really good practice for any company with limited time, limited attention, that you really need to look at your numbers to decide where to focus. There’s no point to build a mobile site if nobody looks at your site on mobile. I know, for a long time, we didn’t have a mobile site, but I ended up using ZURB’s Foundation, and they just make it really easy.

Justine: Oh, nice! They really do, yeah. ZURB is great. Yeah, I say make decisions based on data, not on your gut. Marketers are famous for being distracted by the latest shiny object that everyone’s talking about, and mobile has been a hot topic for a really long time. I don’t blame people for wanting to embrace it, but I think you have to be smart about how you want to embrace it.

Colin: Thinking about new people starting to do email today; we’ve talked a little bit about making that decision, about should you be mobile responsive. What are the other decisions and things to think about, that someone building their email program in 2013 need to think about, with regards to the design?

Justine: Oh, gosh. If you’re brand new to email, like if you’re starting a new company and you’re trying to send email from the ground up, I would say definitely start with a plan. Before you even think about mobile, just start with a plan. You produce excellent content about that type of thing. If you jump straight into design I know you had a blog post about this recently if you jump straight into the design process and just start making emails, but you don’t have any content to go into them, that’s probably a big red flag.
I think, first and foremost, you don’t need to have some giant strategy mapped out but you definitely need to have some sort of strategic goal in mind. That’s one of the best things about email, is that you can measure its performance really, really easily.
If you don’t have a goal in the first place or you don’t even know if your email’s going to work after you send it… [laughs] But in terms of design, I would say that companies that are in the travel industry or have really urgent communications or are communicating with consumers are more than likely going to have to think about mobile, before someone that’s B2B or serving an older audience, or some of those older things.
Again, your gut sometimes is really wrong, and your instincts can be surprising. I would say do what you can in the beginning. Focus on great content, simple designs that are going to be I would say mobile first. Make sure that they’re readable everywhere before you go embrace a full on responsive design because it can be a lot more effort, and sometimes not worth it, frankly, in the end.

Colin: Yeah. One of the things that I’ve seen is people want their emails to kind of look like their website in many ways. Then, a lot of times that prevents it from being responsive because they haven’t gone the extra step. It also prevents someone on a mobile device from reading the content.

Justine: Absolutely.

Colin: Probably the first step for people is, make it possible for everyone, everywhere, to read the content. Then focus on getting the design and the branding the way you want it, after.

Justine: There’s just one blog post. If you’re going to read anything about that topic that you just mentioned; the whole “Make it readable first,” Luke Wroblewski has a blog post called “Mobile First.” It’s literally a couple paragraphs. It will convince you. Go read it.

Justine: It just talks about that idea of making sure that the mobile user’s needs are considered first and foremost, because if you can read it on a mobile device you’re going to be able to read it on the desktop, is basically the argument, as far as that goes.

Colin: Got you. Once you evolve a little bit and you’re in a position to start making beautiful emails like Litmus does, what does your process look like? How do you guys go about it? Do you start in Photoshop? What do you do?

Justine: Yeah. The design that you mentioned a little earlier, how we have those blocks of content that are each a different color, has actually been in place for quite some time. The emails look different every time we send them. It’s actually kind of a really easy trick to make it look like you went and started from scratch every time in Photoshop, without having to. We just have a template that has these 100 percent width columns set up, and then a smaller container width nested inside. You just swap out the background color on each of those bars. Then we do go into Photoshop for the imagery, but that’s really about it. We don’t do a full layout.
We might do parts of it in a full layout every time if we’re doing something tricky or new, but for the most part you can just build each of those images in Photoshop, and then the whole rest is HTML and CSS.
We try to send email newsletters monthly, so probably about two weeks before we’re going to send; I have a Marketing Coordinator. We say, “What blog posts have gone live in the last month, since the last newsletter?” or “What do we want to make sure hasn’t gone live yet, but does go live for the newsletter?”
We kind of build our content strategy, frankly, around the newsletter because that’s what drives a fairly significant chunk of traffic to our website.
We’ll plan blog and blog content around the newsletter. Then, now that we have Kevin, the Content Designer it used to be me that did some of the design work for the emails; it used to be our UI designer that did some content and design for the emails but now Kevin owns the whole thing.
We’ll put together, in a Google Doc that everyone can edit and share, we’ll put all that content in a Google Doc and share it with Kevin, and he’ll start doing the basic layout for the newsletter. It’s really collaborative.
We’re small enough and we’re lucky enough that if we don’t like the way that text is breaking on a line we’ll go in and we’ll edit the headline, and that kind of thing. It’s not possible with big organizations that have to use templates and that kind of thing, but it’s really flexible and streamlined.
Say we want to send the email on Wednesday, but we did something quirky and ran into a rendering issue and we had to test it, it’s no big deal. We just push it off till Thursday. It’s really, like I said, collaborative, “agile,” to a certain extent. We just ship it when it’s ready. [laughs]

Colin: Has adding more people to the marketing team freed you up to explore new types of content? I noticed you doing some videos recently, and one that was particularly awesome was talking about the different pre processors in web mail clients, and the different things that happen when they pre process the emails that they display.

Justine: Yeah. Having a bigger marketing team, it has let us do some bigger, more ambitious projects. The conference is a really great example. We’re hosting this email design conference this fall.

Colin: Three cities, right?

Justine: Three cities, two days each. [laughs]

Colin: Yeah, it’s San Francisco, London, and Boston, right?

Justine: Yes, and you’ll be speaking there. We’re really looking forward to that.

Colin: I will. I’m very excited.

Justine: It’s hard enough doing that with three people, so I can’t imagine having done it just by myself, or without the help of Lauren and Kevin. The videos are an ongoing initiative. We hired a part time in house videographer, probably close to a year ago now. He’s here a day a week, and we have some ongoing initiatives like the understanding webmail rendering with the pre processors and stuff, that you mentioned that’s part of a three part series. We’re going to do desktop, mobile, and webmail.
Things like that definitely make it a little bit easier to have a bigger team. More resources mean that you can tackle more complex and more specialized things, but I think there’s a never ending wish list of things that we still want to tackle, too, no matter how many people we have.

Colin: Yeah. Well, I think you guys are doing an awesome job. Like us, like many other companies, you’ve focused on education and creating content that educates people, and I think, really helps people do their job better. I think that’s a really awesome way to grow and nurture your audience. Yeah, I love reading your content and watching your videos.

Justine: Thank you. Likewise.

Colin: Yeah. We’re about 20 minutes in now. You mentioned the conference; do you want to wrap up and tell people about anything else, you guys.

Justine: A shameless plug for the conference?

Colin: Yeah, a shameless plug.

Justine: Yeah, absolutely. I will welcome it. We’re hosting three email design conferences this fall. We’re coming into San Francisco at the end of September. We’re coming to London at the end of October, and then, to Boston at the end of November. You can go to Litmus.com/conference and check those out.
Colin’s going to talk about customer centricity and how that focus on awesome customer data can help the email experience. We’re going to have how data informs design decisions for email, a panel on how to improve workflow and process, and a whole bunch of other really awesome stuff. We’re really looking forward to it.

Colin: Yeah. I’m super excited to hear everyone else talk there. I think it’s going to be really, really great.

Justine: It’s going to be like the world’s largest meet up of email geeks.

Colin: Yup. [laughter]

Colin: Yeah, thanks so much for taking the time to talk today. I’m going to add a bunch of links to great content that you guys produce, and that Luke Wroblewski article that you mentioned in the blog post that accompanies this video. Thanks again, Justine, and I look forward to talking to you soon.

Justine: Yeah, thanks.

Colin: Bye.

Justine: Bye.

Message Composing

How to use emails to improve activation

The first two weeks in your relationship with a new customer are critical. During those first weeks people are deciding if they’re going to pick you, pick a competitor, or pick no one at all.

In web and mobile apps, after sign up, there are almost always more things someone needs to do to be successful. This is commonly called an “Activation Funnel”.

Most activation funnels make people confused

The most common thing you’ll see is a funnel that is not really a funnel at all. Instead of a clear path, people are offered a jumbled mess of things to do.

Jumbled Mess - Show images

When faced with a barrage of choices, most people will turn off their brain and make no choice. Without a clear path forward, people will abandon you completely.

  • “It’s too complicated”
  • “I didn’t know where to start”
  • “I didn’t feel comfortable sharing that information”

Your buffet of options was well intentioned. Of course it was. It serves experts well, giving them the array of choices. Most people are not experts and benefit greatly from some guidance. Especially when they’re in unfamiliar territory.

You should be opinionated about what people do first… like video games

Video games do a great job of helping people start. Most video games have a “tutorial” mode where they introducing you to the mechanics of the game.

Video game tutorial

  • Here’s how you jump… GREAT!
  • Here’s how you slide under an object…. GREAT
  • Here’s how you slide, then jump…. GREAT

They teach you the basics. Then they teach you how to string those basics together so you can do more advanced things in the game.

What about in your product or with your business?

What is the path someone takes to “Activate”?

Do you have a linear path from 1 to 2 to 3? Or do people have to make up their own minds about where to start?

Take a look at the funnel below:

Actual Funnel - Show images

Some of you may have noticed that it looks like we’re getting a bad conversion from step 2 to step 3 in our funnel above. People “Create a Project” but many people drop off before they “Invite a Collaborator”.

That’s a great place to use an email.

Send emails when people get stuck in your funnel

Since you’ve got your funnel in a nice linear flow, it makes it easier to measure how you’re doing.

How many people make it to each step in the funnel? What happens when we add perfectly timed emails to help people get to the next step?

Take a look at how we might add some emails to improve our funnel for our project management app:

Emails in your funnel

Delay your emails to give people time to complete the step.

Each of those emails goes out on a time delay to give someone time to complete the step. But if they don’t…. there’s an email when someone fails to get from step 1 to 2. And more emails if someone doesn’t make it from step 2 to 3.

Increase relevance by matching the message to the step.

The way you write the email can be perfectly tailored to the things someone has done in your app. This helps you avoid sounding generic. As you’re writing you can now visualize the person you’re writing to and how they’ve engaged with you so far.

What does your activation funnel look like?

In Customer.io it’s something like:

Customer.io activation funnel

If you sign up but don’t send data there are 3 emails that get sent. If you send data, but don’t pick a paid plan, there are emails for that too.

Your activation funnel is a great opportunity to start building that relationship with someone new.

This week we focused on thinking about activation funnels. What does your activation funnel look like? Can you draw it out on a piece of paper? Do you have different funnels for different types of users?

Customer.io enables you to create targeted, triggered email campaigns to send to your users when they get stuck in your funnel. Get rescuing with our free trial!

The Email Design Conference

You’ve probably heard me mention Litmus a couple of times. We use their great product when we need to test our email designs in every email client.

I’ve got some exciting news to share with you:

Show images

Litmus is putting on a conference, The Email Design Conference. It will be in three different cities:

  • San Francisco – September 30 – October 1
  • London – October 28 – 29
  • Boston – November 21 – 22

You’ll find a fantastic line-up of presenters. Ros Hodgkiss from Campaign Monitor, Chris Savage from Wistia and so many other email experts. I’m humbled to say that I’m also going to be presenting.

If you or someone you work with spends a lot of their time working on email for your company, get to one of these conferences. If you’re planning to attend, please let me know. I’d love to meet.

-> Get early bird pricing on the Litmus conference <-

Hope to see you soon,
Colin

P.S. We’re away on our company retreat this week. If you haven’t done a retreat, I’d highly recommend it. It’s a great way to clear your head and focus on bigger picture things with the company. Take a look at a picture from our hike today:

Show images

Brand Strategy

FDR’s fireside chats

Franklin Delano Roosevelt had a Presidency filled with adversity. He became President during the height of the Great Depression. In his second term, the US entered World War 2.

On Sunday I went to Hyde Park, New York to see FDR’s childhood home and the Presidential Library that sits next to it.

Before this weekend I knew very little about FDR. However, I left with a strong admiration for him.

What does FDR have to do with content marketing and email?


Walking around the exhibit in the Presidential Library, I read this blurb and excitedly took a picture of it for you. Take a look at how people describe FDR’s fireside chats:

FDR

FDR was speaking to people over radio in a relatable way. He was connecting with people at scale the way you do when you write conversationally in email.

This level of intimacy with politics made people feel as if they too were part of the administrations decision-making process and many soon felt that they knew Roosevelt personally and most importantly, they grew to trust him.
Wikipedia

So, back in the 1930s, FDR got it. He knew how to connect with people. What’s amazing is how many companies and people have forgotten how to do this since the 1930s. Most still go with a formal, un-relatable tone.

The difference between connecting and not connecting with people is meaningful for a business or a President.

Stats

After the fireside chats started, White House inbound mail went from 5,000 letters to 50,000 a week. FDR was reaching people in a unique and personal way over radio.

I’d encourage you to imagine yourself sitting in your (probably rural) living room, turning on the radio and listening to the President speak to you:

He is speaking directly to you in language that is direct, to the point, and personal.

People built trust and confidence through FDRs fireside chats

Here’s an excerpt from a response FDR received after the broadcast above:

I want to thank you for your visit at 10 o’clock Sunday night. I can see you seated in the big armchair in my living room, pipe in mouth and talking on the crisis that confronts us all, telling me in words that I could understand what you had done and the reasons for your action.
Letter from Simon Miller

I was truly inspired by my visit to FDR’s home and Presidential Library. I was inspired by the adversity the man overcame (he was paralyzed from the waist down when he ran for President). I was inspired by his resolve during the difficult times of his Presidency. I was inspired by the way he used the fireside chats and plain language to connect with people.

I hope you are too.

Happy emailing,
Colin

P.S. Who do you think does a good job of connecting with people at scale? Share that in the comments.

Do you start with the design or the copy?

When’s the last time you heard someone get excited by process? Yesterday I was on the phone with a copywriter about helping us with our content strategy and to rewrite our homepage.

We started chatting about process and I said something to the effect of:

I want to start with the copy and once we’re happy with that, hand it off to our designer to make it look awesome.

I could hear the excitement on the other end of the line.

I’m so glad you said that she replied.

Her current client started with a design full of lorem ipsum text. Then they brought her in to write the copy. After trying to force the copy in to the design, they ended up making A LOT of time-consuming and costly changes to the design.

I’ve done redesigns and new designs the wrong way in the past too. I learned the hard way that while it’s satisfying to see a pretty picture, doing design first is backwards.

Copy informs design, not the other way around

Solid copy works much better than a nice coat of paint.

That said, a great design supports content and makes the words even more powerful. But remember, it’s much cheaper and easier to change words before they’re pixels in photoshop or styles in CSS. That’s why the content we do starts in plain text / markdown.

Do you have stories of projects gone wild when you did things the wrong way around? Or do you have a process you follow now for writing copy that works wonders?

Share your thoughts in the comments.

Sincerely,
Colin

P.S. Here’s my rough draft of our homepage re-written to focus on content rather than design. Feel free to check it out and suggest edits.

P.P.S. You should also check out this post by Justin Jackson: This is a web page

Unscheduled downtime – July 20, 2013

Customer.io’s management interface and inbound event processing experienced 4 hours 52 minutes of downtime on Saturday, July 20th at 5:22 eastern. Here are the events that occurred to cause and extend that outage.

As you may know, we have just moved to a new data-center. We’re now at OVH in Canada and have been pleased with the responsiveness of the staff and the increased performance of the network and servers.

On Saturday we had had our first experience with what happens when something goes wrong.

5:22 pm app server 1 (out of 2) has a hardware failure

OVH detected a hardware failure and automated systems booted the first of our two app servers in to recovery mode. This on it’s own is fine. We have two app servers. Then…

5:23 pm app server 2 (out of 2) is hard rebooted by OVH.

Ordinarily in a reboot scenario, services will restart. As we had just migrated the week before, we haven’t yet finalized all the scripts for the services, and the application server did not start on boot. At this point, one server was down and the other server was responding with a 500 error.

These servers run the Customer.io management interface. They also receive and process data from our geographically distributed collection servers (we call them “pickers”).

Almost immediately a customer got in touch saying the app was down.

At this time, John is on a plane

I had spoken to John a few minutes before the outage and he was in the airport. John’s responsible for our infrastructure and has been handling ops for Customer.io. He did the migration to this new data center the week before. Now he was on his way back to New York on a flight and out of contact. I first verified that the outage was limited and isolated. Rather than risking damaging anything, I didn’t attempt to do anything other than diagnostics on the servers.

So what went right during this outage?

Data collection stays up & no data is lost

We collect data on clusters of geographically distributed servers. It gets queued for processing. The first thing I checked was whether or not data collection was still working. It was.

Rather than being able to send the data on, the distributed servers had the data sitting in the queue waiting for our servers to become responsive again.

Emails continue to be sent

We also have a bunch of background workers that run to look for people who match to send emails to. Checking our logs, people were still matching your campaigns and receiving emails.

Where do we go from here?

Overall, I’m happy that critical pieces of our infrastructure stayed operating and there was no data loss. The management site was inoperable for several hours during a saturday evening, but most of you probably had no idea.

We’ll continue to work on infrastructure, making each piece of the infrastructure that delivers Customer.io more resilient — including having multiple people on the team able to respond to an outage.

I know we can do better than the 4 hours 52 minutes the site was down for you. We’ll be working to improve that. Thanks for your trust and confidence in us to deliver the Customer.io service to you.

Sincerely,
Colin

P.S. During this time, we were posting updates on status.customer.io syndicated to our twitter account. I’d encourage you to check those for updates if you ever see an outage.