SMTP servers are something we all rely on, but something that none of us really want to spend a lot of time thinking about. As a result, we end up not realizing that while there is a real need for test SMTP servers that are easy-to-use in a QA environment, there is a general lack of easily testable SMTP servers.
We all know the need for SMTP servers, but we don’t think about how we’re going to perform QA tests using these servers. We want to stress test our software and send thousands of test emails to users, but we also don’t want to actually send those emails to our users (since, after all, they’re test emails that are likely unfinished).
So, we’re forced to set up a test environment that isn’t pointed or even remotely connected to the rest of the world. That means a test SMTP server that for all intents and purposes behaves like a production one, but in reality is working in a staging mode.
This sounds simple, but unfortunately, it’s not.
Why Setting up Test SMTP Servers Is So Difficult
The sad truth is that test SMTP servers can be incredibly hard to set up, as evidenced by this Hacker News thread on how difficult it can be to create an SMTP staging environment, what the options are, and how annoying the entire practice of setting this function up is.
There are two underlying characteristics that drive the annoyance with the typical SMTP server testing setup. First, there is the need for an environment that is almost exactly the same as production environments. If you want to really find out the results of a QA test, you’re going to want to test the version of code that’s going to be live. Second, there is a need for keeping test emails from actually ending up in the inboxes of real users before the emails are supposed to be sent and while they are still in active development.
These two facts pull at each other. On the one hand, you want to live in a production environment, but on the other hand, you know that you should never be testing your services inside of a production environment. Hence the dilemma with easily testable SMTP servers.
Of course, this isn’t an unsolved problem. There are plenty of solutions. But, the hard truth is that there is no one-size-fits-all solution that works for every team, at every stage, working on every type of problem.
Your SMTP Options: Local, Self-Hosted, and Remote
Local SMTP Server: What You Need to Know
The go-to for engineers when they’re thinking about setting up a testable SMTP server is a locally-run service. After all, they have the technical know-how to get it setup and running effectively in no time at all.
Unfortunately, that’s not really true. While engineers do make the jump into trying to run it locally on their machine, they don’t realize the full complexity of the problem. They end up spending an order of magnitude more time setting it up than they anticipated, and often end up with a sub-par, unreliable system.
If that’s the case for very technically-savvy individuals, it’s going to be all the more true for QA testers and managers who don’t have that same degree of technical familiarity. That’s why it’s more often than not a bad idea to go for a locally-run SMTP server for testing needs and to start looking for a more professionally managed service.
Who it’s good for: Individual people working on early prototype stage testing of products.
Self-Hosted SMTP Server: What You Need to Know
Some teams realize that a locally-run machine isn’t the right idea, since there will be no shared access to the test emails that are sent. So, in order to resolve that issue, they decide to set up a server themselves that they’ll use for their SMTP server testing needs.
Then, two weeks later, when their real projects have slipped beyond the deadline and they spent hundreds of dollars purchasing equipment to run the email service on, they realize that they’ve made a mistake. Not only is this a common cost-sink, but it’s also a large time waster, since the SMTP server that they’re able to construct and hack together is nowhere near as high-quality as something they could find for a fraction of the cost online.
In addition, managing these machines is a complex problem, since they’re often configured to automatically enter Relay Mode, where all emails are actually sent. Suffice it to say that testing the server’s testability is all the testing that’s going to happen in the first few weeks.
Who it’s good for: Companies with very strict on-premise regulatory requirements that have grown beyond having a single person responsible for testing SMTP server reliability.
Remote SMTP Server What You Need to Know
The third option is ideal for the vast majority of teams, since it enables them to take advantage of a third-party with SMTP server maintenance experience. They end up with a professionally managed service, without the headache of figuring out how to professionally manage the server themselves.
Who it’s good for: Companies that have grown beyond the single-person, local scenario, but who have no other requirements than having a simple, testable SMTP server available at all hours of the day.
Things to Remember When Using Your Test SMTP Server
Avoid Using a Production IP Address for Your Test Emails
Don’t use your real, production IP address when you’re sending fake, test emails to users. Not only is it often against the Terms of Service for an IP address if you’re leasing it from a third-party email delivery service, but it’s also a good way to increase your spam and junk mail ratings.
Keep Your Environments Completely Separate
Oil and water don’t mix, and neither do staging and production environments. The risk of accidentally sending test emails with test data to real users is high, which is why you need to keep your environments completely separated.
Some Servers Aren’t Meant For Automated Testing
There are plenty of tools out there that are specifically designed to help automate QA testing. Similarly, there are plenty of tools that have been configured in a way to help automate QA tests — but, that doesn’t mean they were designed for QA testing. Some consumer email providers have their SMTP servers used to automate email testing, but this practice is often against the Terms of Service that users agree to.
Know Any Locally Applicable Laws (Especially in the EU)
Aside from the technical aspects of SMTP server testing, you also need to be aware of locally applicable laws. In the European Union, the General Data Protection Regulation is going to be implemented in May of 2018, and will have a significant impact on the ability of software engineers to test emails in the same fast and loose way that they’ve been accustomed to for more than a decade.
Final Thoughts
SMTP servers can be complicated, but they don’t need to be. If you spend your time planning how you’re going to automate the process of testing your emails in a staging SMTP server, you’re going to be well on your way to avoiding a world of pain. There will be hiccups along the way, but once you’re setup it’ll be smooth sailing. And, of course, if you run into issues you can always send us an email and we’ll help you out.