Anti-Spam Workshop with Mark (7/30/08)

Below is the complete transcript from the Yahoo! Mail Anti-Spam Workshop, held on 7/30/08. Be sure to check the Mail Blog for upcoming Workshop times and details.

 

Ryan: Hi everyone. This is Ryan from the Yahoo! Mail blog. Thanks for coming to our special workshop event. Today I’m here with Mark, from the Yahoo! Mail Anti-Abuse team (also referred to as the Anti-Spam Czar). He’s looking forward to hearing your thoughts and answering some of your questions about spam.

Ryan: Please feel free to submit your questions and we will do our best to get them answered.

Mark: Hi everybody. Thanks for stopping in. I’ll do my best to make sure this as useful for you as possible.

Ryan: Onto the questions. We’ll be reviewing your questions as they come in, and I’ll also be covering questions that I’ve received through the blog.

dlippman: Why don’t emails with the word “Lottery” and a few other Spam characteristics automatically go into my Spam folder?
Mark: I really wish we could! Catching a specific word is really hard. On the one hand, there are the risks that we’ll catch something legitimate — “Campus housing lottery this Friday” — which is what we call a “false positive.” (A “false positive” is any time our filters mistakenly mark something as spam when it isn’t)
Mark: On the other hand, if we build a filter for one specific word, there are often about a bazillion other ways the bad guys can spell it and still get their point across. Did you know there were 600,426,974,379,824,381,952 ways to spell \/!@g.r.a? (check out http://cockeyed.com/lessons/viagra/viagra.html)

galvatron035: I would like to know how I can reduce the amount of Spam I receive in my e-mail accounts.
Mark: Some general tips include:
Mark: 1. Protect your e-mail address as you would your phone number
Mark: 2. Use the “Spam” button to report junk mail
Mark: 3. If it sounds too good to be true, it probably is
Mark: 4. Create a Sign-in Seal to avoid password theft
Mark: We also have a resource where you can learn more about protecting yourself from online scams here: http://antispam.Yahoo.com.

stilwebm: What does the Yahoo! team do with email marked as Spam?
Mark: We have some incredibly sophisticated computer systems analyzing mail to block the spam — we’re using advanced techniques like artificial intelligence, machine learning, and heuristics to identify the bad messages.
Mark: When you click the “spam” button, it sends a signal to those systems that “this message, and others like it, are bad.” We’ll then do two things: 1) we’ll stop messages from that sender from getting to you in the future, and 2) we’ll try our best to stop similar messages for all Yahoo! users. So it’s really the best thing you can do, and the best way to help us improve.

merlin: Why is it so hard to stop these “My name is ___ and I need your help” emails?
Mark: One of the biggest differences between humans and computers is the ability to see patterns. Our brains are wired to find the similarities between things – that’s why we can look at a sky full of clouds and see a dragon, a polar bear, and our first-grade teacher, Mrs. Delorme. Computers can’t do that. They stare at the sky all day and just see an alternating series of blue and white pixels.
Mark: What this means is that the bad guys can write “cl1k ths Lnk 4 cheeeap pilz” and most of us get what they’re trying to say; meanwhile the poor computer is staring at it looking for a subject and a verb.

Ryan: This is one of my favorites from the pre-submitted questions.

User: What’s with the hat in those pics?
Mark: It can get cold in the SpamGuard Fortress of Solitude. ;)
Mark: It’s called an ushanka — a Russian hat that’s kind of stereotypical — and it felt sort of czar-ish. It’s not, though. My little brother supplied this Wikipedia picture of Tsar Nicholas II of Russia, who’s clearly wearing something entirely different: http://upload.wikimedia.org/wikipedia/commons/thumb/f/fa/Tsar_nikolai.jpg/250px-Tsar_nikolai.jpg

bartonas: What is the effect, if any, other than putting it back in my in-box, of me selecting “not spam” for an email in the spam folder?
Mark: We’ve got some incredibly sophisticated systems trying to analyze the messages our users mark as “spam” and “not spam.” We’re constantly analyzing the feedback from users like yourself to figure out how we can improve.
Mark: The effect of clicking “not spam” on a message is that it sends a powerful signal to our systems that we’ve made a mistake. That’s one of the best ways we can learn, both to ensure that we don’t block messages from that sender in the future, and that our systems shouldn’t block similar messages next time.

dlgun: Is there anything I can do to ensure that messages from a given sender will always go to my Inbox? Mark: If you add a sender to your address book we’ll try to ensure that those messages always go to your inbox. There is a slight complication in that bad guys can false — or “spoof” — the “from” address of their messages as a way to try to sneak in, so we do sometimes need to rescan messages even if the sender is in your address book, but in the vast majority of times you should be safe.

Ryan: Great questions guys. Keep them coming.

bjames: Thanks for doing this Mark we appreciate it for sure.
Mark: You’re welcome, bjames. It’s a pleasure.

brip: Customer Care tells me that I need to forward with full headers if I’m reporting spam, but when I try to do that the headers are never there. What should I do?
Ryan: I’ll go ahead and take this one so Mark can catch his breath.
Ryan: To forward with headers users have to take two steps. First you will need to reveal the headers for the message. In Classic you can look for a “Full Headers” link just below the bottom right corner of the message. In All-New Mail there is a Header dropdown just above the top right corner of the message.
Ryan: Once you have exposed the full headers you can copy and paste them into the message as you are forwarding it.
Ryan: We’re also working with Customer Care, and the folks running the Help pages, to communicate this better.

G-192209172: How does Yahoo’s anti-spam currently work? Does it utilize any 3rd-party technologies? Mark: Yahoo! has a great team of incredibly smart computer scientists devoted to fighting spam. Most of the solutions we use were invented by these Yahoos right here in California, but we are always on the market for third-party products as well.

Ryan: If you are joining us late, you can click the transcript button just above the window to see prior questions.

wa9c: I wish we had a filter by country. “Allow only USA”, Trash “Nigeria, Great Brittain, Somalia, etc. etc. etc.”
Mark: It’s a good idea, and one that we continue to explore from time to time. One of the challenges with this one is how to present it in a way that users can understand exactly what they’re doing – it’s all well and good to say “I don’t know anybody in Africa” until your college roommate goes on a safari vacation and his message to you becomes a false positive.

G-195141752: I am mostly concerned about whitelisting. It has been an impossible task for me and my company is suffering dearly. Any tips/feedback you can provide about whitelisting would be greatly appreciated!
Mark: For the commercial e-mail sending companies out there, we have a bunch of resources at http://postmaster.yahoo.com that should help you out. We’ve been pushing more self-service content, and also we have a dedicated team that answers anything on the Contact Us forms within a pretty quick timeframe.

scottweber: Why am I getting more Spam today than every before. Things used to be fine but now I’m seeing dozens of SPAM messages from @yahoo addresses. What went wrong?
Mark: We’re working really hard on this one. Yahoo! has some of the smartest computer scientists in the field working on keeping the spam out of your inbox, and that involves using some truly state-of-the-art approaches that combine to block more than 1 billion spam messages every single day.
Mark: Unfortunately, there are these malicious people out there who really want to be heard, and they keep on adjusting their message until it gets through.
Mark: We have some really exciting filters coming online in the next few months to help with just these problems. One exciting area we’re exploring is using some of the same high-tech computer clusters that biologists use for looking at diseases and applying that technology to treating computer diseases like spam.

opher: SMTP requires a confirmed IP address between the sending and receiving servers. That means spammers can spoof the NAME of the sending server, but not the IP address. Since Yahoo knows the IP address of all of their mail servers, why not validate the IP address and when it does not match, drop the spoofed email?
Mark: Yahoo! has been a pioneer in advancing e-mail authentication – the ability to conclusively identify that a message that says it comes from somebody really comes from that somebody – and was the inventor of the open source DomainKeys and DKIM technologies.
Mark: As we see the adoption of these technologies continue to take off, we’re exploring ways to take action against messages that “spoof” a Yahoo! origin.
Mark: You’re right that IP address is one of the few, truly trustworthy parts of an inbound spam message, and it’s a major factor in our determination of whether a message is spam.

Mindy: What are you recommendations for handling blocks due to complaint volume, since FBL requests are not accepted at the moment?
Mark: The FBL, or feedback loop for those of you who aren’t familiar with the lingo – is a way that Yahoo! communicates back with commercial e-mail senders to let them know their messages are being marked as spam by Yahoo! Mail users.
Mark: One of the most important ways that Yahoo! Mail is able to block spam is by listening to its users. Yahoo! is the largest webmail system on the planet, and if someone is sending mail our users don’t want to receive, those users let us know.
Mark: We recommend commercial e-mail senders ensure they’re sending mail that Yahoo! Mail users want to receive. This means following recommended practices like confirming – and even periodically re-confirming – that users want to be on their mailing lists and proactively removing anyone who doesn’t read their mail.

pnnygrms: How can I stop receiving so many e-mails about money I have won from all over the world
Mark: We are hard at work on that very problem, and in fact Yahoo! has recently brought a lawsuit against some of the scam artists perpetrating the bogus “Yahoo! Lottery”
Mark: One thing my grandfather always used to say was, “If it sounds too good to be true, it probably is.” If you receive a message from somebody you’ve never met before offering you millions of United States Euro Dollars… it falls right into that “too good to be true” bucket ;)
Mark: Never ever reply to these scam artists, even in jest. As soon as you write back and they smell money — and these guys have no sense of humor — they’re likely to start sending 3x, 5x, even 10x as many messages to you.

G-184910729: throw us a bone, will ya!
Mark: I’m sorry, is that a question or a feature request ;)
Ryan: Yes, please make sure that any bone requests are accompanied by an actual question about spam.

Ace: To followup on dlippman’s question – is there a certain % of weight given to the “SPAM words” compared to how many folks are hitting the “SPAM button”
Mark: Our filters examine 100s of features on every message: what words it contains, what’s in the subject line, how many people it’s addressed to, what IP address or computer sent it, etc. Mark: We don’t have a fixed % for a particular bad word, but rather it’s a floating, multivariate equation that’s constantly adjusted as users click the “spam” button on bad messages.

Tron: what changes are you making to the current system? should we expect improvements?
Mark: We’re constantly working on the SpamGuard system, and have teams spread out around the globe making constant tweaks.
Mark: Yahoo! Mail is the largest e-mail provider on the planet, and protecting our users from spam is a top priority of the company. In fact, we block more than a billion scam messages every single day (a lot more than a billion ;)

Sandy: Hi! Thanks for doing this! Please tell me what I can do when I receive spam of an illegal nature. For example, there are obvious scams that show up in my inbox – stating, for example, that they are a trustee and a lawyer and need my help to cash a check for their client. They think I’m stupid enough to fall for this. But, there might be elderly folks or confused people being taken advantage of by these schemes and I feel protective of them. Where can we forward emails we think are scams and potentially harmful to people? Is there a “cyber police force”? Thanks!
Ryan: Mark’s fingers are cramping…so I’ll take this one:) Forward any suspicious email to phishing@cc.yahoo-inc.com

paulbeattie87: Recently my Yahoo mail inbox keeps filling up with spam, spamguard doesn’t seem to be filtering out the spam very well. My address isn’t posted anywhere online. I always click the “spam” button but it keeps on coming, some is even from other Yahoo! Members. How can I stop this?
Mark: The single best thing you can do is to click the “spam” button every time. We’ve made some significant improvements recently to block recurring spam messages, and you should see immediate improvement from that.
Mark: Honestly — I don’t mean it to sound like a simplistic response — that single action is the most effective way you can let me know there’s a problem. Each and every message you mark as “spam” runs through an amazingly complex — and ever-growing — series of filters so we can learn about it and block it in the future.

Ryan: For the record, these emoticons are not approved by the Yahoo! Messenger team. ;)

essietr: If the 500 limit in blocked spam addresses is full. More spam is coming thru from some of the ones that are blocked. What can be done to remedy this? Thank you
Mark: There are a few questions/requests to increase that 500-message limit. The main reason we haven’t is because we find that 99% of spammers use disposable “From” addresses — they’ll send out one or two spam messages and then move on to the next account.
Mark: This is particularly true for @yahoo.com accounts. We’re very fast at shutting down any Yahoo accounts that are being used for abuse.
Mark: So if you really need to block a specific address, you’re safe removing some of the older ones. We also made a recent change so that any sender you mark as “spam” will get automatically blocked.
Mark: Hope that helps ya.

stefanpollard: When using the spam button in my email account, it doesn’t seem to do anything. I report emails that are spam and keep getting them, and I un-mark emails I signed up for that are in my bulk folder from obvious companies (e-reader, tommy bahama, Dell). Why can’t Yahoo either make the button work or recognized obvious real emails from spam?
Mark: We work really hard to make sure that, when you report a message as spam, we block similar ones in the future. Unfortunately, it can be hard to tell what aspect of a message someone is marking: Was this message bad because of the “from” address, or was it the contents of the message, or maybe even the computer it was sent from?
Mark: Because of this, it may take a few times clicking on the “this is spam” button before our systems can catch up. Mark: Oh, one note: Unsubscribing to a message from a flagrant spammer is a definite no-no. Really unscrupulous scam artists sometimes put a fake “unsubscribe” link in their messages, and if you click on it, they actually send you *more* messages. So if you’re really suspicious, it’s safest to just click the “Spam” button.

iorgous: I used one of those Nigerian Emails to make 10 million dollars, they’re not all spam.
Ryan: Really? I suppose next you are going to tell me that you would like to share in your great fortune. All I have to do is wire you some money :)

Ryan: Hi everyone. We’ve got Mark for about 15 more minutes so be sure to get those questions in now!

Mel: Not really a question but a comment with the hope that Yahoo! will figure out why a lot of postdated spam shows up. For instance, on the 28th I got several spam emails that were dated August 1. Shouldn’t these be easy to stop since they obviously aren’t legitimate emails? I used to get pre-dated emails, i.e. emails that were dated from the ’70′s, before the internet came to be. Yahoo! ultimately learned to stop them in their tracks because it’s been about a year since I’ve seen one. Maybe the same can be done for postdated emails. Thanks!
Mark: Hi Mel,
Mark: I’m glad you asked. With hundreds of different spam attempts every day, we have to prioritize the feature areas we work on.
Mark: For this particular spammer, we’ve been throwing 100% of his messages into the spam folder for a long time. Talk about an unabashed spammer; instead of cleaning up his act or giving up the fight, he decided he just wanted to be at the top of the list – the spammiest of the spammers. So he started setting the date way into the future so that people who sort their messages by date would see his garbage first. Mark: While this one is really irritating – and I completely share your frustration – because the messages are in the spam folder, we’ve been focusing our efforts lately on other areas.

stilwebm: Speaking of forwarding with full headers, the ability to forward as an RFC 822 formatted attachement preserves the original content of the email and its full headers. This functionality used to exisit in Yahoo! Mail Classic and should be easy to add to Yahoo! Mail 2.0. Any hope for returning that feature? Mark: Forward as attachment is definitely on our roadmap. Thanks for the vote for that feature.

dpm: my wife runs a small business here in Santa Clara (near Y! world HQ). when she sends out a mailing from her ATT/Yahoo account, it winds up in my bulk mail folder, even though she is in my address book. why is that and how can I teach Y! mail to stop treating my wife as spam?
Mark: That’s definitely something we’d want to work on. Do you mark those messages as “not spam”? We need that input so we realize our filters are doing something wrong.
Mark: If she’s in your address book, you’re right, we should be delivering that to your inbox.

MARY: is it better to block a domain and remove addresses from blocked list or should i leave the blocked addresses as they are and just add the shared domain name?
Ryan: Once you have blocked the domain you can remove them from the blocked email list.

iorgous: Mark can we have a little bit more info on that. FBL really help us create clean lists, and remove people that opted in but no longer want our newsletters. Are you bringing back the FBL?
Mark: Yes, we have a shiny new feedback loop sign-up system coming very soon. Stay tuned on http://postmaster.yahoo.com for more details in the upcoming weeks.

Marine: With all the money you guys are spending building clusters and what not .. how is this profitable to your company. Should more time be direct toward other things?
Mark: Fighting spam, and protecting Yahoo! Mail users from it, is one of the top priorities of the company. Spam is costly and annoying for our users, and frequently is used to perpetrate very serious crimes like identity theft.
Mark: That’s why we’re here today talking to you, trying to learn how well we’re doing and what we can do to help. I really appreciate all of your comments and suggestions.

G-192209172: I use Yahoo’s Disposable Address feature quite a lot (over a hundred) to determine where spam originates from. In Ymails Classic version they were color-coded, but not in the latest version. Any idea what the plans are for Disposable Addresses?

Mark: I didn’t realize that; I’ll have someone look into it.

Zombieblaster: I have a small e-commerce website and when I send email to my customers it sometimes end up in spam folders or sometimes don’t show up at all. How can I ensure my customers get them?
Mark: If your company mails are being mis-routed, you should definitely let us know. Please have someone from your IT department (or yourself) visit the site at http://postmaster.yahoo.com and click on “Contact Us.” There you’ll find a Postmaster Whitelist request form.
Mark: The Postmaster team should get back to you very shortly to confirm any details or to ensure those messages are delivered correctly in the future.

trevlore: when I send mail to yahoo users, why do I consistently get a “451Resources temporarily not available – Please try again later” message about half the time. Is this how you are greylisting?
Mark: (Another question from a bulk e-mail sender or ISP)
Mark: If we’re sending a 4xx series reply code to legitimate mail, then your mail server is likely mis-categorized on our side. The forms and team at http://postmaster.yahoo.com should help you out straightaway.

tommy: why cant you block ips and or domains of spammers?
Mark: That’s a good suggestion. We do block a humongous number of IPs and domains from spammers, and that is a powerful tool in the fight.

C_Morr: I am a heavy user of the Yahoo Mail Plus AddressGuard feature which lets me create disposable email addresses so that I can see which one was used to send spam and terminate it. When spam is sent to “undisclosed recipients” including my disposable address, I can’t see which disposable address was used. I know one was since I filter email to a number of these addresses to a “shopping” folder. It is not enough to be able to look at the sending IP address in the headers because the spammer who stole my address sent mail from an AOL account. I reported this to support in March so hopefully it’s still on the books.
Mark: That’s an interesting point, C_Morr, and I’ll definitely look into that. We may have a header in those messages that tells which of your AG accounts it was to: check for X-Apparently-To in the full headers…?

Marine: I posted a question the other day which asked if the philosophy of yahoo giving the user the tools to detect and delete unwanted messages will stay the same. Will you enhance the tools provided to the users?
Mark: Hi, Marine:
Mark: We certainly value the opinions of our users, and we try to keep enough flexibility for you to customize your Yahoo! Mail system without being confusing for other users.
Mark: Our philosophy remains the same, and we will continue to add features for advanced users such as yourself to tailor the filtering experience. iorgous: Thanks for posting my comment, it was funny, sorry if It wasted resources.

Ryan: Don’t worry, it gave us a chance to catch our collective breath :)

Ryan: Thanks everyone for joining us. Mark is heading off to go ice down his fingers for a few minutes. We hope you all found it useful. I’ll be looking through the questions we couldn’t get to and try to find some themes for future blog posts.