See You At The Message Systems User Conference!

I’m looking forward to the upcoming Message Systems User Conference next month in San Francisco, not only for what looks like an excellent venue, but for the great set of quality sessions on the agenda.

There’s a number of sessions I’m looking forward to attending, but I’d like to invite you to attend the sessions I’ll be delivering next month (read to the end to save on conference admission):

What the Convergence of Data Security & Privacy Concerns Will Mean to Companies

The barrage of news stories about data breaches and privacy violations is taking a toll on consumer confidence.

What You’ll Learn:

  • Why data security and privacy issues are converging and how an erosion of consumer confidence can jeopardize data availability for communication and commerce.
  • How security and privacy are connected to Message Convergence and why they should now be of concern to all ecosystem players and at all levels, Marketing as well as IT.
  • What principles companies should embrace to address security and privacy in their own environments.
  • How companies can safeguard their customer data and messaging streams.

New Directions in Email Deliverability

Our panel of industry experts will explore the ongoing evolution of deliverability management and new technology advances, such as adaptive delivery, that will make it easier.

What You’ll Learn:

  • How deliverability is a tactic companion to Message Convergence – getting messages delivered, read and acted on.
  • How new advances in technology can improve deliverability management effectiveness and remove the hassles for all stakeholders.

Building Multi-Channel Apps

This session will introduce participants to the whys and wherefores of multi-channel messaging applications ­ how they deliver business value, and how to construct them. You¹ll gain both an understanding of the business strategy behind multi-channel apps, and a nuts-and-bolts working knowledge of the tools and techniques required to design, build and deploy them. Topics will include how to access multiple data sources on the fly and how to make routing determinations. For instance, once you¹ve made a judgment on content, context and preference, how to go about actually getting a message routed to its ultimate destination. We’ll go in depth on the subject of multi-channel message type (MCMT), a proprietary content container format that makes it possible to inject messages into the delivery stream with content alternatives dependent on the preferred message channel.

Target Audience:

Product and program managers, developers, line of business owners.

What you’ll learn:

  • How multi-channel messaging delivers business value across any number of industry verticals.
  • The messaging and data systems/architectures needed to deploy multi-channel messaging.
  • Introduction to MCMT.
  • How to configure Momentum, Mobile Momentum and Message Central for multi-channel apps.
  • Understanding and acting on customer preference data.

Advanced Momentum & Message Scope

This session will extend the sessions on “Introduction to Lua” and “Momentum Essentials and Message Scope” by taking participants through advanced, Lua-based message parsing APIs. Advanced policy scripts for database-driven binding assignment and DKIM signing will be demonstrated. Participants will see practical, but advanced remediation list usage with Message Scope and learn how to create custom remediation actions.

Target Audience: System administrators, operations and support personnel and developers.

What You’ll Learn:

  • Various parsing techniques using Lua API functionality.
  • Write Lua policy scripts that implement database-driven binding assignment and DKIM signing.
  • How to integrate Momentum bounce information with an external database.
  • How to integrate Message Scope with 3rd-party data feeds.
  • How to create custom remediation actions with Message Scope.

It’s going to be a great conference and I look forward to meeting everyone, to make it even more appealing, register now and use discount code VIP2S2 to save $250!

See you there!

Message Queuing & Segregation: Lessons from the Airline Industry

Note: This also appears at http://www.messagesystems.com/wordpress/?p=38.

Many email marketers are unaware of the importance of message queuing to the successful delivery of their email. As a component of their messaging infrastructure, queuing is something that marketers typically defer to their IT department to manage. Yet, the reality is that queuing and the segregation of message streams can make the critical difference between the success and failure of a company’s messaging programs, and therefore, should be of concern to both the IT and marketing departments.

Effective queuing really comes down to the choice of messaging infrastructure. When using a technologically advanced messaging platform, companies can efficiently manage parallel queues with messages assigned into multiple streams to ensure that each stream flows at an appropriate rate, providing efficient delivery of all classes of traffic. Unfortunately, those that rely legacy MTAs have no such options. They’re left trying to manage the bottlenecks and slowdowns that result from poor architecture with complicated priority schemes within a single queue.

Recently, one legacy MTA provider suggested that their routine for queue prioritization was the answer for reaching high-value customers first. While the business need is certainly legitimate, trying to prioritize messages within a single queue is both outmoded and a solution to a problem that should not exist in the first place. There are better ways to satisfy this need that are both simpler and more powerful at the same time. To illustrate my point, allow me to provide an analogy that should be familiar to my fellow business travelers.

One of the most common headaches for the air traveler is the security checkpoint; you get your ID checked, get in line, get your ID checked again, get in another line, empty your bags, take off your shoes and belt, get in yet another line and then get radiated in the name of public safety. During the highest traffic times these lines can become so long that people start missing their flights because there are a very limited number of security checkpoints and the airports were architected in a time that predated the need for such extensive security. This fundamental flaw in the architecture of the airports means that the current needs of travelers for additional parallel security screening checkpoints cannot be met, and everyone has to wait in a queue to get to their plane, sometimes with unacceptable results, leading to additional costs for all involved and potential lost business.

This is handled in a variety of ways, including performing the security check at every gate area (creating a very parallel security screening system) and by using priority security lines. Imagine for a moment that instead of this solution, the airport chose instead to assign priority on an individual basis to every single passenger and then tried to sort the individual travelers in the security line. As you can imagine such a solution would require additional work to make the individual assignments and then keep track of who ranks where in the line, with a risk that low-priority passengers would find themselves significantly delayed as they were repeatedly bumped. In all my travels I have never seen such an approach, primarily because the airports already have an approach that works.

Find a particularly efficient airport and what you’ll see is a fairly consistent set of practices:

  • Separation of passengers into queues based on their fitting into a certain profile.
  • A large number of parallel checkpoints.
  • Efficient handling of passengers.
  • Intelligent queue management that can modify queuing on the fly to meet circumstances.

Look at a particularly efficient airport and you will see multiple queues, including queues for:

  • Frequent / First Class travelers – The most frequent travelers and those who sit in First Class. These people bring a lot of value to the airlines and receive a lot of value in return.
  • Expert travelers – A new lane starting to appear in some airports, for those who are experienced in getting through security and unlikely to cause delays.
  • Family / Special Assistance – A slow lane, these groups will take longer to get through security.
  • Casual Travelers – A lane for those who move at an average speed through security.
  • Staff / Crew – While in some airports workers and air crew jump to the front of the line, the most efficient airports avoid this disruption by maintaining a separate checkpoint for those who work at the airport, minimizing disruption and ensuring that staff can get to work on time.
  • Specialty – From time to time I’ve seen the airport create a special temporary queue for unique groups such as chartered planes by opening a checkpoint and redirecting the group to the specialty group.

Not only has the separation of travelers into queues according to their profiles (including their priority as a group) proven sufficient to make prioritization by individual traveler unnecessary, it is much easier to manage.

While separating passengers into a number of queues can certainly benefit airports, the most efficient airports are also architected to operate a large number of parallel checkpoints, preventing a situation where every passenger in the airport needs to be funneled through the same metal detector. Imagine an airport trying to service millions of passengers a year on only one or two metal detectors and a single x-ray machine.

In addition to having many checkpoints, the best airports will also have efficient checkpoints, maximizing the flow of passengers through any given checkpoint through better design of the checkpoints and better training of the staff, all without compromising safety.

Perhaps most importantly to the smooth operation of an airport is intelligent management. I’ve seen airports where there were several queues open but empty because the people managing things weren’t flexible enough to reassign lines and adjust the queues to ensure well balanced passenger flow. The best airports will change the designation of queues, move staff around and even redirect passengers to alternate checkpoints that are less busy, all in the interests of moving the highest number of passengers per hour.

Senders can follow these same principles to get maximum throughput and deliverability in their own environment:

  • Segment mail by profile.
  • Choose a sending solution that supports highly parallel sending.
  • Choose a sending solution that provides sufficient throughput.
  • Choose a sending solution that is intelligent.

When sending, remember that segmentation is not just for who to send to or what to send them, but for deciding how to send a message and with what priority. You will want to create queues for high priority messages to satisfy your most valuable customers, queues for high-reputation traffic that delivers without issues as well as for traffic that you expect to deliver slowly (one example is traffic that results in human interactions, you may need to slow this to prevent overloading your call centers), test and administrative traffic that needs to go out as soon as possible and transactional traffic that should not queue up behind bulk sends. This is a common practice among ESPs, who often add specialized segmenting for scenarios such as new customers and customers with specific SLAs.

As with the airports, you need to architect your environment to be able to handle more traffic in parallel. This can be accomplished by adding more injectors and more messaging infrastructure, or by adding better infrastructure. Look at your existing solution: how many IPs can it send from? How many messages per hour can you send on a single machine and how many concurrent connections can it handle? Most Open Source solutions can handle one IP address, send 100,000 messages per hour at most, and can open less than one hundred connections. Low-end commercial solutions can often do over a hundred IPs, send close to a million messages per hour and can handle a few hundred to a couple of thousand connections. On the high end you have carrier-grade systems designed for the enterprise such as Momentum by Message Systems, which can utilize thousands of IPs to send millions of messages per hour across tens of thousands of concurrent connections (while you will never open more than a few connections to a given ISP on a given IP address, lesser solutions will fall short when sending across hundreds of IPs to thousands of ISPs, defaulting to ISP prioritization as a workaround).

Finally consider the intelligence of your infrastructure:

  • Can your infrastructure send across all servers simultaneously and fail-over in the event of an outage?
  • Can your infrastructure adjust throttles on the fly based on responses from the ISPs and bounce and feedback loop data?
  • Does your infrastructure handle queues so efficiently that performance is the same with thousands of messages in the queues as it is with millions of messages in the queues?
  • Can your infrastructure dynamically change from email to other protocols such as SMS and MMS based on subscriber preferences and ISP responses?

If not, we should talk.

Technical Considerations For IP Warmup

Introduction

In response to the recent news regarding Goodmail closing its doors, Tom Sather at Return Path published a blog entry regarding IP warmup and the difference it can make for inbox placement.

Tom sums up the need for IP warmup well:

If you had talked to any email marketer 10 years ago and asked them how they dealt with blocks on their IP addresses, the answer would probably be the same: “We just switched IPs.” Not only was this an unfortunate, albeit effective, way to deal with blocks, it also became a common method used by spammers. They would simply send from one IP address for a very short time and then move on to another, either with IPs they owned or through hijacked computers controlled by botnets. Because of spammers’ behaviors, ISPs and email providers respond by temporarily blocking and limiting the amount of email a new IP address could send. ISPs now treat any new sending IP address like a dog on a short leash, and only extend the leash when the senders’ reputation is proven.

I personally have seen what can happen when senders try to send too much, too soon, with senders trying to send millions of messages on their first day using new IP addresses and finding themselves blacklisted in short order. For a reputable sender the key is to start sending slowly and gradually increase volume on new IP addresses until a proper sending reputation has been established.

A Clarification

Before I get into some technical advice I’d like to clarify one thing from the Return Path article. Regarding the shutdown of Goodmail Tom has this to say:

There are a couple of reasons you still might have to send from a new IP address, such as moving to a new ESP, moving to a new data provider, or moving off of Goodmail. Goodmail had a unique way of tokenizing their customers’ mail by relaying mail through their own IP addresses, and consequently their reputation. Therefore, once you stopped using Goodmail, your traffic now goes through your IPs, which hasn’t had any traffic in awhile, which means you’ll need to work on building up your sending reputation again.

This statement applies to any customer’s of Goodmail’s hosted imprinting service but does not apply to in-house senders using products such as Momentum by Message Systems that had a built-in Goodmail Imprinter. For such users the shutting down of Goodmail involved shutting off the Goodmail Imprinter component of their infrastructure but IP warmup will not be required since those users were already sending using their own IP addresses.

Tom’s Advice

Assuming you didn’t get a chance to read the link, here’s the five points of advice provided by the article:

  1. Sign up for all feedback loops. Suppress from future mailings.
  2. Authenticate. Use SPF, SenderID and DKIM.
  3. Segment and mail your active subscribers. Put your best foot forward.
  4. Monitor. Use seedlists such as Mailbox Monitor and watch your IP’s Sender Score.
  5. Get Certified. Get your new IP Sender Score Certified.

Some of these warrant additional discussion from a technical point of view.

Feedback Loops

In order to be effective, Feedback Loop message handling needs to be automatic. Message Systems customers should already be aware that we have provided built-in Feedback Loop processing as of our 3.0 release in 2008. In addition to automatically unsubscribing recipients that trigger a Feedback Loop message, you should also take the volume of feedback loop hits as a metric to show the effectiveness of your mailings. Feedback Loop hits should also be used as a factor when determining traffic shaping rules, especially when an IP address is new. If you see a lot of FBL hits, you should throttling back on the sending IP address.

Monitoring

With regards to monitoring, seedlist monitoring is a good indicator of how ISPs are treating your mail, but they provide only part of the overall picture. To get a complete view of your deliverability you need to also monitor what happens before the ISP accepts a given message, taking into account what temporary (aka transient or 4xx) failures and permanent (aka 5xx) failures that are occurring as you try to send. When monitoring permanent failures, keep in mind that permanent failures can occur both during delivery (synchronous or in-band) and post-delivery (asynchronous or out-of-band) through the ISP sending back a DSN (delivery status notification) message to the return path (aka envelope sender or envelope from) address of the original message. You should be tracking and trending all failures, especially when sending on new IP addresses.

Additional Technical Considerations

Keep in mind that just like in life, you are judged not just by what you say, but by how you say it (and whom you say it to). With regards to deliverability this comes down to content and sending practices. From a technical standpoint we focus less on content (what you say), but it does have a significant impact on IP warmup. You should avoid sending riskier content on new IP addresses both on overall content and wording (I’ve seen deliverability dip just for using the word “sexy” in a mailing, even when the overall message was not sexual in nature).

Sending Throttles

Before you ever start sending from a new IP it is vital that you pre-configure your sending software to comply with as many published ISP recommendations as possible. A convenient resource is this page provided by Word to the Wise: ISP Summary Information. Pay particular attention to the Connection Limits and Sending Limits columns.

Sending Volume

When warming up IP addresses it is important to start slowly; ISPs do not trust new IPs and will not respond well to new IPs coming online and immediately bursting out large amounts of traffic. While there are no published limits online one of the recommendations I have heard is to avoid sending more than 10,000 messages per day to the major ISPs (Yahoo!, Gmail, Hotmail, AOL, etc.) when first sending, and I’d say it would be best to send less than 1,000 messages per day to any smaller ISPs. By reviewing your temporary and permanent failure messages you will be able to get a feel on whether your reputation is sufficient to increase volume, and after increasing volume you should pay particularly close attention to your failure metrics to make sure that the change has not had an adverse effect on deliverability. I generally recommend not increasing volume by more than 2x at a time and not more than once every day or so. Don’t hesitate to revert to a lower volume if you start seeing an increase in temporary and permanent failures.

Suppress Bounces

When sending, make sure to quickly and automatically suppress any recipients that the ISPs identify as being invalid. When first sending you should assume that you are being watched closely, and one aspect of that is your practices regarding bounce processing. If you repeatedly send to someone that an ISP identifies as invalid through a bounce message you will be penalized for it by the ISP, and that punishment can potentially come faster when an IP address is new due to the lower starting reputation.

Watch Out For Deferrals

There is a specific class of temporary and permanent failure responses that you need to keep a particular eye out for, the deferral messages. A deferral message from an ISP indicates that you need to quickly and decisively change your sending practices as they are the warning messages you receive from the ISPs prior to being blacklisted. You can get examples of deferral messages from Yahoo here and many ISPs will list examples of their deferral messages on their postmaster page. As an example, here is a hotmail deferral message:

421 4.16.55 [TS01] Messages from x.x.x.x temporarily deferred due to excessive user complaints

When you see such messages, you need to review your content and throttles and pause sending for a couple of hours to allow things to cool off while you determine what changes you need to make.

Automating IP Warmup

In his article Tom Sather advised:

If this looks like a lot of work, then you’re right. To be successful, you need to plan appropriately, be patient, send smarter, and constantly monitor.

Tom is absolutely correct, warming up new IP addresses requires research, preparation and diligence.

The good news for Message Systems customers is that we’ve taken care of this for you with our Adaptive Delivery module. Adaptive Delivery will automatically identify new IP addresses, set initial throttles and gradually increase volume as the IPs age, monitoring ISP responses to ensure that the ISPs are responding positively. If at any point the Adaptive Delivery module identifies a negative ISP response, it adjusts throttles in realtime and monitors for additional negative responses. If an ISP replies with a deferral response, Adaptive Delivery will suspend delivery, throttle back and send you an alert so that you can check the content being sent. All of this is built using intelligence that is constant reviewed and improved by a full-time, in-house deliverability specialist. In addition, our bounce processing system now supports live updates, allowing us to improve classifications thanks to automated feedback from customer systems. If you’re not taking advantage of these new capabilities contact Message Systems and we’ll help.

Congratulations, You Can Mail-Merge!

I had this exact thought the other day and need to share this great article by Morgan Stewart. Quick quote:

I’ve received over 500 emails so far this year in which the subject line included my first name. “Morgan, Book Now & Save on Top Travel Deals” “Morgan – Congratulations! Your Nomination to Cambridge Who’s Who!” “Morgan, Get Dad a 58″ Samsung Widescreen.”

When I see an email with my name in the subject line, my first thought is not “Phew! These guys know my name!” No, it’s become a red flag for spam.

He’s spot-on: these days seeing my name in the subject line is almost a sure sign of spam. Read the entire post here.

Good to know I share good company on the Cambridge Who’s Who list.

Mailchimp Makes Project Omnivore Public

Pretty impressive blog post at MailChimp today, in which they make public the details of their Project Omnivore: http://www.mailchimp.com/blog/project-omnivore-declassified.

In short:

Omnivore is a program that runs in the background and analyzes email campaign and user account data. Non-stop.

When it finds anything suspicious about a MailChimp user or his campaigns, it’ll do one of two things:

  1. Send the user a warning for something that looks problematic.
  2. Suspend a user’s account for something bad, send them a warning, and alert our abuse team to investigate the account.

The long version is at the link and the impressive part is how much data and prediction they are incorporating into the tool to help them avoid sending campaigns that will damage their sending reputation in the long run. It’s not just about filtering the mail stream to make sure it’s not going to trip filters, but watching the list management practices of their customers.

This is the kind of thing that all ESPs are going to have to start doing moving toward, using internal systems to ensure that the right message is being sent to the right people, then combined with tools like Adaptive Delivery and real-time bounce and feedback loop processing to make sure that the messages are being sent in the right way.

Some days I’d just like to see what a company like MailChimp would do with our toolset, give them a messaging server with internal scripting that can hook into their datasources and I’d wager some very cool things would come out.