Transcript from the Anti-Spam Workshop with Mark
- Posted August 6th, 2008 at 10:28 am by Ryan
- Categories: All-New Mail, Anti-Spam, Classic Mail, General, Security, Tips & Tutorials
Once again we’d like to thank everyone who submitted questions, and who stopped by to participate, during last week’s Anti-Spam Workshop with Mark (transcript here). We received a lot of great questions, and Mark enjoyed the opportunity to answer questions, as well as hear directly from the users. One of the more interesting questions lead to a stat that Mark had never shared with me … and totally blew me away.
dlippman: Why don’t emails with the word “Lottery” and a few other Spam characteristics automatically go into my Spam folder?
Mark: I really wish we could! Catching a specific word is really hard. On the one hand, there are the risks that we’ll catch something legitimate — “Campus housing lottery this Friday” — which is what we call a “false positive.” (A “false positive” is any time our filters mistakenly mark something as spam when it isn’t)
On the other hand, if we build a filter for one specific word, there are often about a bazillion other ways the bad guys can spell it and still get their point across. Did you know there were 600,426,974,379,824,381,952 ways to spell \/!@g.r.a? (check out http://cockeyed.com/lessons/viagra/viagra.html)
I actually had to look up what that number translates to (turns out it’s a quintillion) and who would have thought that the “little blue miracle” could be re-written that many ways.
Check out the complete transcript from the Workshop here. We will be doing more of these Workshops over the next few months, so if you have any themes you would like to recommend please feel free to suggest them (comments or email).
Ryan Knight
Community Manager
Yahoo! Mail
- Buzz up
- Subscribe via RSS
- 35 Comments
August 6th, 2008 at 10:51 am
WTF? Post a word doc to the web? No HTML or PDF version… It’s trivial to save-as PDF these days.
August 6th, 2008 at 11:57 am
KevinH,
You are right…I’m an idiot :). Not sure why my brain cramped so badly, but I switched it to an HTML page. Sorry about that.
-Ryan
August 6th, 2008 at 11:25 am
When do you thing you’ll have another Q&A? Thanks
August 6th, 2008 at 2:25 pm
Ryan, question. I subscribe to a daily newsletter from “Cute Overload,” a very popular animal lover website. It always ends up in spam. I say, “no, not spam.” I add the sending address to address book. I set up a filter to route it to inbox using the right criteria.
It still ends up once every couple days..in spam.
So now what? MOST annoying.
August 6th, 2008 at 2:33 pm
=bg=,
Have you checked to make sure that you don’t have any other filters that may conflict with the one you set up?
-Ryan
August 6th, 2008 at 8:49 pm
No other filters. I can’t seem to keep it out of spam.
August 6th, 2008 at 4:05 pm
About this spam stuff, I found that even if I list them as spam, they come back under another name and email address most are lottery’s from the UK, and You have money from someone that I don’t even know. Even when I tell them they have made a mistake I never entered or don’t know the person, it just keeps coming. I had to go to hotmail to get away from them.
August 21st, 2008 at 3:38 am
You had to migrate to Hotmail, to escape spam in Yahoo!??? -Precisely the opposite of my experience. When I first got started w/ e-mail, in the bad ol’ days of Win3.1, Hotmail was everybody’s (including my-) favorite. (It touts cute stationery, for spec. occasions.) Then, a veritable deluge of spam gushed from (Hell?), clogging up the works with porn links and every manner of digital detritus. -Got so bad, that I simply gave up. -VERY happily migrated to Yahoo!, and I’ve never looked back. My experience with Yahoo! has been just about perfect, in every respect; effective spam filters, good A.V. (-although I do sub. to the full-blown suite of McAfee products…), and REAL, LIVE help. -Not just llinks, to links, to more links, but, real, human responses to queries. Each time I’ve posted a question, they’ve gotten back to me so quickly, I couldn’t even go make a cuppa. Considering that they service a worldwide network, with (millions?) of subscribers, the personal and intimate level of their service is simply phenomenal. They do it, politely, efficiently, and (still) for FREE, to boot! -What more could anyone possibly ask for? Oh, yeah, and adverts are squeezed onto border-areas, where we can easily view -or ignore them. On a scale of 1~100: 99.999. We’ve all been “invited” to migrate to the “new” Yahoomail. I’ve resisted, until now, simply because “Good ol’ Yahoo” is precisely that; familiar and comfortable. But, am weighing my options. And what’s nice is, that if I don’t like the new ver., I can revert to the old. Further proof of what a great organization they run, and how they REALLY DO want us to be happy with their products, even if they’re not the newest ensemble of bells ‘n’ whistles.
August 6th, 2008 at 4:43 pm
Hey Ryan, I checked the site you linked and below it includes the following update
There are some characters that were left out of the One Letter Substitution technique:
Greg Knauss wrote and showed me how the G can be portrayed with a 6, Ryan Hill alerted me to the I-ness of an exclamation mark. SR pointed out that a 9 occasionally stands in for a g.
These three additions change the first equation to (3*13*17*4*3*17) variations, and boost the second equation to ( 192 x 3 x 192 x 13 x 192 x 17 x 192 x 4 x 192 x 3 x 192 x 17 x 192) = 1,300,925,111,156,286,160,896.
Thanks Greg, Ryan and SR, you helped push the total into the SEXTILLIONS!
August 6th, 2008 at 4:54 pm
Thanks for the link to the Viagra article!
August 7th, 2008 at 3:58 am
=bg= you are right.
August 7th, 2008 at 7:01 am
I’m glad the issue of “Forward as Attachment” and forwarding full headers came up. I truly hope that bringing back this feature is on the front burner, as it is an invaluable tool in the fight against spam. As effective as Yahoo’s efforts have been, occasionally spam does get through, and users can do their part by reporting it to relevant or helpful parties such as
phishing@cc.yahoo-inc.com
spamcop.net
phishing-report@us-cert.gov
spam@uce.gov
spoof@paypal.com
etc.
Reporting spam to these parties usually requires that full headers be included. Simply cutting and pasting them is grossly inefficient, as they often have to be manually reformatted for the receiving party to parse them. If the offending e-mail could be forwarded as an attachment, the headers can be parsed, and all hyperlinks can be detected.
In short, the fight against spam is EVERYONE’S fight. Please help users do their part by swiftly returning the “Forward as Attachment feature.
August 11th, 2008 at 9:01 am
HUGE +1 for this!
Very well articulated skepticalthinker
August 7th, 2008 at 1:57 pm
PS I have noticed many features of all new mail have said ‘we’re still tweaking..’ for quite a long time now. when will those be done?
August 8th, 2008 at 10:37 am
Hi Ryan,
Since I don’t get any reply from yahoo customer service I”ll give it a try this way to see if anything is going to move at yahoo.
Since quite some time I’m not able to search trough my emails. I get only results from the last month or so.
Try a search at Google and you’ll see that there’s a lot more people having the same problem.
I’m a mail plus subscriber and use Firefox 3.0.1
I would appreciate that some attention be brought to this problem.
Regards,
Ernst
August 8th, 2008 at 11:48 am
Thanks for posting the transcript and thanks to the Spam Czar himself for taking on our questions. My question was answered and I can say I had no idea that one guy was responsible for all of that post-dated spam. It’s pretty wild to get an email dated 2038. It was even funnier when I used to get email dated from the 1970’s.
I like the furry hat, even if some people think it’s a little, well, goofy.
August 8th, 2008 at 5:41 pm
I was also kinda surprised one immature individual is responsible for the spam from the future. I wonder if Yahoo keeps track of those IPs being used to send and blocks unadulterated time mail from them too. I bet those are Storm worm infected PCs.
I really like Yahoo Mail, but the SpamGuard is becoming a joke. I follow all the recommendations about hitting Spam and Not Spam, but for years there’s been really no difference in where the stuff goes. However, today and today only, ONE email actually went into the inbox for the first time that I have been hitting Not Spam on for the past several months.
One more note to add to my long comment.
My inbox and Spam box usually have daily races on who gets the most Nigerian 419 mails. Yahoo can’t seem to figure out that an email with all CAPS in the subject and/or body is spam 99.8% of the time. I’m sure all those technical wizards who were being praised to the gods could figure out a formula for the .02% of a false positive chance.
August 8th, 2008 at 8:30 pm
What about spam that appears to be from a legitamate source? I’ve gotten 2 e-mails allegedly from yahoo groups asking to confirm request to membership for groups that have to do with cams and sex videos. I don’t have cam and I don’t have videos… It does say if I didn’t ask to join then to disregard. I didn’t request to join, so why am I recieving them unless it is spam trying to get people to join a bogus group?
August 9th, 2008 at 12:44 am
Ryan, Love what you are doing. Thanks for all the work. BTW, thank you for the contact list print button and layouts. Made my life much more reasonable. To all who want it fixed, already. *The Internet, and all its components, is an ongoing process which, ideally, will continue to evolve with humanity to meet our needs. It will never be entirely “fixed”.* Signed, Grrranny
August 9th, 2008 at 9:08 am
In the spirit of this post I decided to write my name with the viagra substitution letters. I always wondered why no matter how many times I marked that as spam it could always creep back in to my inbox. Wow, that’s amazing. How do you deal with someone continuing to send you spam even though you have marked them as spam AND asked that they take you off of their subscribers list? Any suggestions?
August 9th, 2008 at 11:51 am
Following on skepticalthinker’s comment: If for some reason you’re unwilling to restore the “Forward as Attachment” feature, then _at least_ you should offer an option to “Forward with full headers”.
August 9th, 2008 at 3:36 pm
AddressGuard, or “Disposable Addresses” is a valuable anti-spam feature, but its value is _seriously_ diminished by the effort & time needed to create a new address or to give someone one of our existing addresses.
We should be able to create a new address on-the-fly, directly from the new-message screen, using whatever we’ve set as our default options. We should also be able to copy one of our existing addresses easily and quickly, so that we can paste it into a screen form. You could achieve both of these objectives if you make the entire “From” line an input box akin to the “To” field: it could use auto-complete as we type a keyword, and create a new address automatically if we persist in typing a new keyword.
Currently, creating a new address takes 2 clicks with a complete screen change, 2 more clicks with a new window opening, and still one more click (with another complete screen change), just to get to the “Enter a keyword” input box. That’s 5 clicks and 3 complete screen or window changes for something that should be right there in the new-message screen. And even then, we’re not nearly done! We still have to click to get our cursor in the “Enter a keyword” input box, and then (after typing the keyword) click on the Continue button, and then click again if we want to “Allow messages to be sent from this address” (because you don’t remember our previous choice for that option or allow us to set a default for it), and then click on the “Set up AddressGuard” button to complete the address-creation process. But we’re STILL NOT DONE, because now we have to return to our mail screen. Now, finally, we’re done. This is such an extaordinarily tedious process that I may be losing count, but it seems to be at least TEN clicks with 4 or more complete screen or window changes, almost all of which should be eliminated. Considering that AddressGuard comes with a fee, one would think you’d make its improvement a higher priority! (One would apparently be sadly mistaken.)
As for using the Yahoo! Toolbar to speed up the address-creation process, well, that suggestion (from your Help pages) has two problems: (1) The toolbar is not available on all the PCs that I use in various locations. (2) The toolbar is great for accessing mail features when I’m not in my Yahoo! Mail window, but it’s no excuse for making the app itself hard to use. And remember, Yahoo! Mail is a web app; so relying on a client add-in to make the app functional is missing the point.
August 11th, 2008 at 9:51 pm
Thanks for the suggestion, I’ll pass it along to the Anti-Abuse team.
-Ryan
August 11th, 2008 at 3:48 pm
My earlier comment seems not to have made it- have you been reading about this CNN.com spam that is all over the place- check SANS Internet Storm Center. They have all ended up in bulk, btw..but some people clicking on them are getting lots of headaches.
August 11th, 2008 at 7:19 pm
Oh, yeah. I’m getting lots of CNN spam. Richard Branson crashed his experimental craft, feared dead. Click here (on an .exe!) to read story. I don think I signed up for these alerts, but sure! Whatever CNN says!
August 11th, 2008 at 4:53 pm
Okay… i asked a q. of tech support, and they said i should post my suggestion on the blog. I’m not quite sure why, but here goes.
I would really love to see a feature added to the web Y!Mail that would allow you to turn on the BCC field to always be visible by default.
I BCC fairly regularly, but fairly frequently forget to actually PUT the BCC in since it isn’t visible by default. I then have to go back and forward the email to the people who should’ve been BCCed.
August 11th, 2008 at 9:48 pm
Hi eidylon,
Sorry to be the bearer of bad news, but that preference was removed in the recent streamlining efforts. There were some options that our records indicated were seldom (and in most cases never used) by the vast majority of users. So we tried to make sure that there would still be minimally obtrusive ways to use it on a case by base basis. I’ll relay your feedback, however I don’t anticipate it being restored in the short term.
-Ryan
August 12th, 2008 at 11:59 am
I had very little spam get past the Yahoo spam filter for the last couple of years. All of a sudden about mid July 2008 I have been getting 10-30 per day for Viagra and Cialis. Viagra and Cialis are correctly spelled in the spam emails. What gives? No spam is being delivered to my Gmail account.
I checked and my spam filter is still enabled. Did your programmers accidentally disable the spam filters?
August 12th, 2008 at 12:09 pm
I personally would love to see the spam from John Commuta permanently eliminated from the yahoo mailings. I get many every day in my spam mailbox. Also, if we mark an email as spam, why oh why does it have to go into the spam mailbox so we have to delete it from there? When it’s in our inbox, and we mark it as spam, why can’t it just disappear?
August 12th, 2008 at 2:17 pm
Dooley,
I’ll pass along that feedback to the guys on the spam team.
-Ryan
August 12th, 2008 at 6:29 pm
the 421/451 comments are interesting. Senders are constantly sent to postmaster pages….given lengthy forms….receive lenghty questionaires as autoresponses…..sometimes are told their mail has problems it doesn’t (in the rare isntance they have semi-direct communication with a human being) and rarely solve the problem
the postmaster pages are exceedingly vague as well
a simple web search will find millions of postings from senders, ISPs, education institutions, e-commerce sites complaining about 421/451 and decrying that after weeks and months of trying to resolve they have not been able to.
As a proposed step towards solving this problem….I’d suggest having a page in the postmaster section where you can enter a sending IP and get a more detailed explanation of WHY it is being blocked, I can understand to a degree not wanting to be 100% transparent as it aids the americans, new zealanders, and aussies pretending to be russian who send the bulk of the spam running around. But if it indicated a problem with header configuration verses user complaint driven verses “message looks like spam” would be very helpful. Perhaps also an email address to send a test email and get a email response stating problems with email configuration (headers/DNS/etc)…there are free third party ones but they often say things are fine but senders go on to discover they are not actually fine. Something that matches what yahoo uses as a litmus test would be nice.
On the consumer side, I’ve noticed a decent amount of spam in the offline messages section since it’s inception.
August 15th, 2008 at 4:58 pm
I need to know how I can get all of my old email back? there is something that I need to see and get?
August 18th, 2008 at 7:01 pm
I opted to use Spam Arrest but I could not set up my emails to be forward into my Yahoo! inbox. After back and forth comments with SpamArrest folks, they told me Yahoo! does not allow their interface to work with Spam Arrest so all that setting up and refiguring POP, etc, was a waste of time. Yahoo! should do something about spam. I’m thinking seriously of just using another email server - and I’ve been with Yahoo! for years. Just tired of the spam and no support from Yahoo!
August 21st, 2008 at 7:08 am
Ryan- I just received thru Spam, 2 emails from MY yahoo email address! What is going on here and what kind of security concerns should I be looking to correct? I didn’t send them to myself, or to anyone else (as Spam) so how is it someone else can “adopt” my yahoo address for these purposes? what can I do to stop it?
August 23rd, 2008 at 5:14 am
What I’d like to know, is why I am receiving emails that aren’t sent to my email address, but rather have the first four letters of my email. Why am I getting these??
it’s annoying to say the least. I open my email in the morning to check the spam folder to see that a legitimate email isn’t in the spam folder, just to see that most of those emails aren’t sent to my address…. WHAT GIVES??
also,
I blocked addresses from sending me junk, only to see that those addresses which were blocked by me, continue to get through.
I believe Yahoo has many improvements to make. My gmail account never has this problem… it does what I tell it to do and I don’t receive emails which are sent to a different email address.