1. Overview

In a recent survey, 93% of respondents reported dissatisfaction with the large volume of unsolicited email (spam) they receive. [ref 1] The problem has grown to the point where nearly 50% of the world's email is spam [ref 2], yet only a few hundred groups are responsible. [ref 3] Many anti-spam solutions have been proposed and a few have been implemented. Unfortunately, these solutions do not prevent spam as much as they interfere with every-day email communications.

The problems posed by spam have grown from simple annoyances to significant security issues. The deluge of spam costs up to an estimated $20 billion each year in lost productivity -- according to the same document, spam within a company can cost between $600 and $1,000 per year for every user.[ref 4]

1.1 Security issues
In addition to the wasted time spent viewing and deleting spam, spam also poses security risks including:
Identity theft. Phishing and scams are distributed as spam, directly leading to identity theft and fraud. According to the Anti-Phishing Working Group, phishing spam increased 52% in January. [ref 5]
Viruses. New viruses, worms, and malware, such as Melissa, Love Bug, and MyDoom use spam techniques to propagate after being triggered by the user.
Combining exploits and spam. The distinction between malicious hackers and spammers has become less obvious. Many spammers have incorporated malicious code that targets browser, HTML, and Javascript vulnerabilities. For example, on 31-December-2002 a group of hackers in Brazil sent spam containing a hostile Javascript to millions of users. People that viewed this spam from Hotmail unknowingly compromised their accounts. As another example, the recent URL display problem with Internet Explorer, where a "%01" before the hostname can be used to hide the real hostname [ref 6], was incorporated into spam within a few weeks of the public announcement.
Combining viruses and spam. It is widely believed that some viruses are designed to assist spammers. For example, the SoBig worm installed open proxies that were used to relay spam. As spam becomes more prevalent, the use of malware and spyware to support spam is likely to increase.

The existing and proposed anti-spam solutions attempt to mitigate the spam problem and address security needs. By correctly identifying spam, the impact from email viruses, exploits, and identity theft can be reduced. These solutions implement various types of security in an effort to thwart spam.

Current anti-spam solutions fall into four primary categories: filters, reverse lookups, challenges, and cryptography. Each of these solutions offers some relief to the spam problem, but they also have significant limitations. The first part of this two-part paper looks at filters and reverse lookup solutions. The second part focuses on the various types of challenges, such as challenge-response and computational challenges as well as cryptographic solutions. While there are many different aspects to these solutions, this paper only discusses the most common and significant concerns -- this paper is not intended to be a complete listing of implementation options, solutions, and issues.
1.2 Common terminology
Sender. The person or process that is responsible for generating (initiating) the email.
Recipient. Any email account that receives the email. This may be specified in the email as a "To:", "CC:", or "BCC:".
1.3 Filters

Filters are used by a recipient system to identify and organize spam. There are many different types of filter systems including:
Word lists. Simple and complex lists of words that are known to be associated with spam. For example, "viagra".
Black lists and White lists. These lists contain known IP addresses of spam and non-spam senders, respectively.
Hash-tables. These systems summarize emails into pseudo-unique values. Repeated sightings of hash values are symptomatic of a bulk mailing.
Artificial Intelligence and Probabilistic systems. Systems such as Bayesian networks are used to learn word frequencies and patterns that usually are associated with both spam and non-spam messages.

Filters are ranked based on their false-negative and false-positive results. A false negative indicates an actual spam message that manages to pass the filter. In contrast, a false positive indicates a non-spam email that was incorrectly classified as spam. An ideal spam filter would generate no false-positives and very few false-negatives.

These filter-based anti-spam approaches have three significant limitations:
Bypassing filters. Spam senders and their bulk-mailing applications are not static -- they rapidly adapt around filters. For example, to counter word lists, spam senders randomize the spelling of words ("viagra", "V1agra", "\/iaagra"). Hash-busters (sequences of random characters that differ in each email) were created for bypassing hash filters. And the currently popular Bayesian filters are being bypassed by the inclusion of random words and sentences. Most spam filters are only effective for a few weeks at best. In order to maintain the viability of anti-spam systems, filter rule sets must be constantly updated -- usually on a daily or weekly basis.
False-positives. The more effective a spam filter, the higher the probability of misclassifying a desirable email as spam. For example, email containing the word "viagra" (e.g., the spam text "Free viagra" or a non-spam personal email "Hey, did you see that funny viagra commercial during the superbowl?") is almost certain to be marked as spam regardless of the content. Similarly, email from Comcast's 24.8.0.0/15 subnet is blindly blocked by the SORBS blacklist because it is associated with DHCP addresses and not because the sender is associated with spam. Conversely, spam filters that generate virtually no false-positives are likely to generate a large amount of false-negatives.
Filter reviewing . Due to the possibility of false-positives, messages marked as spam are usually not immediately deleted. Instead, these messages are placed in "spam mailboxes" for future review. Unfortunately, this means that users still must view the spam, even if only by the subject, as they search for misclassified email. In essence, filters only assist in sorting incoming email.

More important than the limitations of spam filters is the common myth around the success of filters -- there is a widely held belief that filters stop spam. Spam filters do not stop spam. In all cases, the spam is still generated, still traverses the network, and still gets delivered. And unless the user does not mind missing the occasional misclassified desirable email, the spam is still viewed. While filters do help organize and separate email into spam and non-spam groupings, filters do not prevent spam.
1.4 Reverse lookup

Nearly all spam uses forged sender ("From:") addresses; very few spam emails use the sender's true email address. Furthermore, most forged email addresses appear to come from trusted domains. For example, in 15 months our spam archive collected 9300 emails that claimed to come from 2400 unique domains. The "yahoo.com" domain accounted for nearly 20% of sender addresses in the archive, but spam that actually came from the "yahoo.com" domain accounted for less than 1%. Similarly, "aol.com" and "hotmail.com" accounted for 5% each, and "msn.com" accounted for 3% even though spam, originating from all of these domains (cumulative), accounted for less than 1% of all spam received.

Spam senders forge email for numerous reasons.
Illegal. Many spam messages are scams and illegal in most countries. By forging the sender address, the spam sender can remain anonymous and prevent prosecution.
Undesirable. Most spam senders are aware that their messages are undesirable. By forging the sender address, they can mitigate the repercussion from sending millions of messages to millions of angry recipients.
ISP limitations. Most Internet service providers have contract clauses that prevent spamming. By forging the sender address, they reduce the likelihood of having their ISP cancel their network access.

By addressing the forgery problem, spam senders will lose the ability to remain anonymous. Without being able to operate anonymously, laws such as the U.S.-based CAN-SPAM Act will become enforceable for spammers operating from and in the United States.

In an effort to limit the ability to forge sender addresses, a number of proposed systems have surfaced for validating a sender's email. These systems include:
Reverse Mail Exchanger (RMX). <http://www.ietf.org/internet-drafts/draft-danisch-dns-rr-smtp-03.txt>
Sender Permitted From (SPF). <http://spf.pobox.com/>
Designated Mailers Protocol (DMP). <http://www.pan-am.ca/dmp/>

These approaches are very similar to each other and in many ways they are identical. DNS is a global network service used to match IP addresses with hostnames and vice versa. In 1986 DNS was extended to associate mail exchanger ("MX") records. [ref 7] When delivering email, a mail server determines where to pass the message based on the MX record associated with the recipient's domain name.

Similar to MX records, the reverse lookup solutions define reverse-MX records ("RMX" for RMX, "SPF" for SPF, and "DMP" for DMP) for determining whether email from a particular domain is permitted to originate from any particular IP address. The basic idea is that forged email addresses do not originate from the correct RMX (or SPF or DMP) address range and therefore can be immediately identified as forged.

While these solutions are viable in certain situations, they share some significant limitations.
1.4.1 Host-less and vanity domains
The reverse lookup approach requires email to originate from a known and trusted mail server located at a well-known IP address (the reverse-MX record). Unfortunately, the majority of domain names are not associated with static IP addresses. Omitting cyber squatters, the general case includes individuals and small companies that want to use their own domain rather than their ISP's, but cannot afford their own static IP address and mail server. DNS registration hosts, such as GoDaddy, provide free mail forwarding services to people that register host-less or vanity domains. Although these mail forwarding services can manage incoming email, they do not offer free out-going email access.

Reverse-lookup solutions cause a few problems for these host-less and vanity domain users:
No reverse-MX record. People sending email from a host-less or vanity domain simply configure their mail application to send email from their registered domain name. Unfortunately, a lookup of the sender's IP address will not find the sender's domain, and a lookup of the sender's domain may not find the correct reverse-MX record. The former is particularly common for mobile, dialup, and other users that frequently change IP addresses.
No outgoing mail. One possible solution requires relaying all outgoing email through the ISP's SMTP server. This would provide a valid reverse-MX record for sending email. Unfortunately, many ISP's do not permit relaying when the sender's domain is not the same as the ISP's domain.

In both cases, someone that uses a vanity domain, or a domain that does not have its own mail server, will be blocked by reverse-lookup systems.
1.4.2 Mobile computing
Mobile computing is a very common practice. People take their laptops to conferences, off-site meetings, and home in order to work away from the office or in a location that is convenient. Hotels, airports, and even coffee shops cater to the mobile computing crowd. Unfortunately, the reverse-lookup solution will likely prevent many mobile users from sending email.
Sending directly. There are two ways to send email. A user can login to a mail system using an external POP/IMAP/SMTP account, web mail or similar service, or a user can send email directly. Most companies do not permit external access to their mail services; mobile users usually configure their laptops to send email directly. Unfortunately, the problems with sending email directly are the exact same as the problems with host-less domains -- a reverse lookup of the domain will not include the sender's IP address, and a reverse lookup of the senders IP address will not reveal the domain.
Mail relaying. The alternative to sending directly requires all companies and domain systems to provide external mail services for their off-site and mobile users. In many situations, this is both undesirable and impractical. As an example, from a strictly network-security viewpoint, POP3 transmits usernames and passwords in plain text. Thus, any attacker sniffing the network will see valid login credentials. IMAP can be used with SSL and supports secure authentication, but not all servers support this. SMTP also supports SSL or TLS but again, many organization's servers do not support this or use only server-side certificates. Web mail over HTTPS is only as secure as the client-side certificates. Since most sites only use server-side certificates, HTTPS offer very little protection from man-in-the-middle network attacks.

While reverse-lookup solutions are viable for internal networks, these are not globally practical for external practice. Companies that wish to support host-less domains, vanity domains, and mobile or off-site users may wish to reconsider implementing reverse-lookup anti-spam technologies.
2. Summary

Spam has reached epidemic proportions and people are looking for quick fixes of any kind. Spam filters are the most successful solution to date -- filters attempt to identify spam and limit a recipient's exposure. But filters do not prevent spam any more than recording a television show with a VCR prevents TV commercials. Reverse-lookup systems attempt to address the forgery problem. While reverse lookups are viable in closed environments, such as a corporate internal network, the solutions are not general enough for worldwide acceptance.

Part II of this investigation will focus on challenge-based systems and proposed cryptographic solutions.



About the author

Neal Krawetz has a Ph.D. in Computer Science and over 15 years of computer security experience. Dr. Krawetz is considered one of the leading experts in spam research and anti-spam technologies. In addition to studying the nature of spam, he leads the External Threat Assessment Team (ETAT) at Secure Science Corporation, a professional services and software company that develops advanced technology dedicated to protecting online assets.
References

[ref 1] "Majority in Favor of Making Mass-Spamming Illegal Rises to 79% of Those Online." The Harris Poll ® #38. July 16, 2003.

[ref 2] "Spam On Course to Be Over Half of All Email This Summer," Brightmail press release. July 1, 2003.

[ref 3] According to SpamHaus, a spam content tracking organization, less than 200 spam groups generate more than 90% of spam messages. SpamHaus ROKSO, September 22, 2003.

[ref 4] Source: "Spam Costs $20 Billion Each Year in Lost Productivity", by Jay Lyman. December 29, 2003.

[ref 5] Source: "Phishing e-mail fraud rises 52% in January, report says", February 18, 2004.

[ref 6] Reference: "Multiple Browser URI Display Obfuscation Weakness"

[ref 7] Source: "Domain System Changes and Observations", RFC973 by Paul Mockapetris. January 1986.

http://www.securityfocus.com/infocus/1763

And Part 2:
Anti-Spam Solutions and Security, Part 2
by Dr. Neal Krawetz
last updated March 9, 2004

Editor's note: part one of this article series is available here.
1. Overview
The Simple Mail Transfer Protocol was never designed for security. SMTP dates all the way back to a 1973 extension to the FTP protocol. [ref 1] In 1973, computer security was not a significant concern, and the Internet architects were not even certain about their implementation of the email protocol. For example, RFC 524 describes the bases for SMTP as a separate protocol. The author included this caveat:

"Although one can (I think) and might, implement software on the basis of this document, this REALLY IS a Request for Comments. Comments, questions, position papers are solicited. There are, I'm sure, bugs in the protocol specified here, and I hope that readers will point them out via RFC as they discover them."

Although the command-set has evolved over time, it appears that people implemented SMTP on the basis of RFC 524 and it was assumed that the bugs, such as security concerns, would be addressed later. Unfortunately, in 2004 the oversights from RFC 524 are still being addressed and SMTP is too popular to replace overnight. Spam is one example of an abuse of the SMTP protocol -- most spam tools are designed to forge email headers, disguise senders, and obscure the origination system.

As a brief review from part one of this article series, current anti-spam solutions fall into four primary categories: filters, reverse lookups, challenges, and cryptography. Each of these solutions offers some relief to the spam problem, but they also have significant limitations. The first article looked at filters and reverse lookup solutions. This second part now focuses on the various types of challenge-based systems and cryptographic solutions. While there are many different aspects to these solutions, this paper only discusses the most common and significant concerns -- this paper is not intended to be a complete listing of implementation options, solutions, and issues.
1.1 Terminology
Sender. The person or process that is responsible for generating (initiating) the email.
Recipient. Any email account that receives the email. This may be specified in the email as a "To:", "CC:", or "BCC:".
1.2 Challenges
Spam senders use automated bulk-mailing programs to generate millions of emails per day. Challenges attempt to impede bulk-senders by slowing the bulk-mailing process. People that send a few emails at a time should not be significantly impacted. Unfortunately, challenges are only successful when very few people use them. As their popularity increases, they are much more likely to interfere with desirable email than to deter unwanted spam.

There are two main types of challenges: challenge-response and proposed computational challenges.
1.2.1 Challenge-Response
Challenge-Response (CR) systems maintain a list of permitted senders. Email from a new sender is temporarily held without delivery. The new sender is sent an email that provides a challenge (usually a click on a URL or reply email). After completing the challenge, the new sender is added to the list of permitted senders and the original email is delivered. The belief is that spam senders using fake sender email addresses will never receive the challenge, and spam senders using real email addresses will not be able to reply to all of the challenges. Unfortunately, CR systems have a number of limitations including:

CR deadlock. Alice tells Bill to email her friend Charlie. Bill sends an email to Charlie. Charlie's CR system intercepts the email and sends a challenge to Bill. Unfortunately, Bill's CR system intercepts Charlie's challenge and issues its own challenge. Since neither user actually receives the challenge, neither user will receive the email. And since the emails are unsolicited and unexpected, neither user knows to look for the pending challenge. In essence, if two people both use CR systems, then they will not be able to communicate with each other.
Automated systems. Mailing lists and automated systems such as USA Today's "Send To A Friend" cannot respond to challenges. (The argument that "you can always manually add them" only works if you know that the email is coming. I frequently receive news articles that friends find interesting and forwarded to me from sites such as CNN and USA Today. These emails are unexpected and unsolicited, but not undesirable.)
Interpretation Challenges. Many CR systems implement interpretation challenges. These complex CR systems include character recognition or pattern matching that can be easily automated. For example, the CR system used by Yahoo for creating new email accounts is vulnerable to simple AI systems that perform character recognition. Hushmail's email CR system requires finding an image on a blue background (parse the background, find the image, and submit the coordinates -- no problem).


Figure 1. Yahoo's account creation challenge. This system is vulnerable to intelligent character recognition software.


Figure 2. Hushmail's graphical challenge. The user must click on the keyhole. This system is vulnerable to simple image processing.

The marketing myth emphasizes two misconceptions: (1) a human must perform the challenge, and (2) these problems are too complex for automated solutions. In truth, most spam senders ignore these CR systems because they do not account for a large recipient base, not because the challenge is difficult. Many spam senders use valid email addresses for their scams or for validating mailing lists. When CR systems begin to interfere with spam operations, spammers will automate the responses to these challenges.
1.2.2 Computational Challenge
There are many proposed Computational Challenge (CC) systems that attempt to add a "cost" to sending email. Most CC systems use complex algorithms that are intended to take time. For a single user, the time is unlikely to be noticed. But for a bulk mailer such as a spam sender, the small delays add up, making it take too long to send millions of emails. Some examples of proposed CC systems include Hash Cash [ref 2] and Microsoft's Black Penny. [ref 3] Unfortunately, CC systems have their own set of implementation issues that are likely to prevent rapid adoption and unlikely to prevent spam. Examples of these limitations include:

Unequal taxation. Computational challenges, whether based on CPU, memory, or network, will penalize people with slower systems more than people with faster systems. For example, a CPU challenge that takes 10 seconds on a 1Ghz computer will take over 20 seconds on a 500MHz computer. [ref 4]
Mailing lists. Many mailing lists have thousands or even millions of recipients. Popular mailing lists, such as BugTraq, will be penalized just as hard as spammers. Computational challenges make mailing list management impractical. And if there is a way for legitimate mailing lists to bypass the challenge, then spammers can equally bypass the challenge.
Robot armies. As we saw with Sobig and other spam-supporting viruses, many spam senders control hundreds of thousands of compromised systems. Spam senders can easily distribute any expensive costs across their "0wned" systems, negating the impact of any cost.
Legal robot armies. Spam senders generate spam because it brings in significant revenue. Large spam groups can afford purchasing hundreds of systems for distributing any computational cost. This can be done legally and without compromising anyone's system with a virus.

The currently proposed computational challenges are unlikely to be widely adopted -- they do not appear to mitigate the spam problem and do appear to inconvenience legitimate mailers.
1.3 Cryptography
A few solutions have been proposed that use cryptography to validate the spam sender. Essentially, these systems use certificates to perform the authentication. Without a proper certificate, a forged email can be readily identified. Some proposed cryptographic solutions include:

AMTP. http://www.ietf.org/internet-drafts/draft-...man-amtp-02.txt
MTP. http://www.ietf.org/internet-drafts/draft-...mail-mtp-00.txt
S/MIME and PGP/MIME. http://www.imc.org/smime-pgpmime.html

The existing mail protocol (SMTP) has no explicit support for cryptographic authentication. Some of these proposed solutions extend SMTP (e.g., S/MIME, PGP/MIME, and AMTP), while others aim to replace the existing mail infrastructure (e.g., MTP). Interestingly, the MTP author mentions "SMTP is more than 20 years old, whereas modern requirements developed within the last 5-10 years. The large number of existing extensions to the syntax and semantics of SMTP show, that pure SMTP doesn't fulfill these requirements and that is too inflexible to be extended without modification of its syntax. [sic]" [ref 5] It could easily be argued that the large number of existing extensions to SMTP demonstrates its flexibility, not inflexibility, and that a completely new mail transport protocol is unnecessary.

When using certificates, such as X.509 or TLS, some type of certificate authority must be available. Unfortunately, if the certificates are stored in DNS then the private keys must be available for validation. (And if a spammer has access to the private keys, then they can generate valid public keys.) Alternately, a central trusted certificate authority (CA) could be used. Unfortunately, email is a distributed system and nobody wants to see a single CA in control of all email. Many solutions even permit multiple CA systems where, for example, the X.509 certificate identifies the validating CA server. This extension is vulnerable to the situation where a spammer runs a private CA server.

When there is no certificate authority, there needs to be some method for distributing keys between the sender and recipient. PGP, for example, requires pre-shared public keys. While this approach is viable for closed networks or close groups of friends, this does not extend well across large groups of individuals, particularly when new contacts may be established between any sender and any recipient. Essentially, pre-shared keys face similar problems to white-list filters: only known and established senders may contact the recipient.

Unfortunately, these cryptographic solutions are unlikely to stop spam. For example, let us assume that one of these solutions (any one) is globally accepted. These approaches do not validate that the email address is real -- they only validate that the sender had the correct keys for the email. This creates a few issues:

Automated abuse. When implemented on a global scale, there will need to be a way to generate certificates or keys for all users (or mail servers, or mail clients, depending on the chosen solution). The system is likely to provide keys via an automated solution since there is a scalability issue for a manual solution: there is an estimated 605.60 million people online. [ref 6] Unfortunately, it is realistic to believe that spammers will abuse any automated system within a week of deployment and use it to continue to send "authenticated" spam.
Usability issues. There are also concerns about usability. For example, what happens if the CA server is unavailable? Would email hang, bounce, or assumed to be valid? Spammers recently conducted a month-long denial-of-service attack against nearly a half-dozen sites that provide blacklists. [ref 7] One widely held belief is that spammers were attacking blacklist services in order to prevent customers from receiving updates. It is not unrealistic to believe that a single CA server (or even distributed CA network) will be susceptible to a similar attack.
Anti-spam solutions summary
Spam has reached epidemic proportions and people are looking for quick fixes of any kind. There are many existing and proposed anti-spam solutions. While these options are viable in limited circumstances, they all appear to have significant limitations with regards to global acceptance and an ability to prevent spam.

In Part I we saw that spam filters, while being viable options for identifying spam, do not prevent spam and require constant maintenance. Reverse-lookup systems attempt to identify forged senders but restrict email's usability by preventing host-less and vanity domains, and restricting mobile users' abilities to send email from anywhere at anytime. In Part II we observed that challenge-response systems are only viable as long as they maintain a low profile, and computational challenges are unlikely to deter spammers. Cryptographic solutions, while accurately identifying forged email, do not easily expand to a global scale.

While many people believe that any anti-spam solution is better than nothing, most of these solutions impede regular users more than they prevent spammers. While some of these proposed options report to have effectively stopped spam in limited tests, they do not take into account that spammers adapt their code rapidly, on the order of days or weeks -- a good solution today is unlikely to be a good solution tomorrow.


About the author


Neal Krawetz has a Ph.D. in Computer Science and over 15 years of computer security experience. Dr. Krawetz is considered one of the leading experts in spam research and anti-spam technologies. In addition to studying the nature of spam, he leads the External Threat Assessment Team (ETAT) at Secure Science Corporation, a professional services and software company that develops advanced technology dedicated to protecting online assets.
References


[ref 1] RFC 458 (Feb. 20, 1973): Mail retrieval via FTP. RFC 510 (May 30, 1973): Network mailbox addresses (user@system). RFC 524 (June 13, 1973): Branching from FTP to a standalone protocol. RFC 561 (Sept. 5, 1973): Standard mail headers.

[ref 2] Source: http://www.cypherspace.org/adam/hashcash/.

[ref 3] Source: http://groups.yahoo.com/local/service-spamguard.html.

[ref 4] The delay is actually more than double due to operating system overhead.

[ref 5] Source: http://www.ietf.org/internet-drafts/draft-...mail-mtp-00.txt.

[ref 6] Source: Nua Internet How Many Online http://www.nua.ie/surveys/how_many_online/index.html, 5-February-2004.

[ref 7] Source: "Virus and dDoS Attacks on Spamhaus", http://www.spamhaus.org/cyberattacks/index.html.

http://www.securityfocus.com/infocus/1766

no mention of disposable email addr's?

good post either way.