For many years I’ve been inviting a group of friends over to brunch and/or wine/cheese via a personal mailing list. This is just a text file of the form
bcc: foo@bar.combcc: yow@baz.orgbcc: student@veryrich.edu
I prepare an email message in Emacs, to: philg, and then insert all of these bcc: headers. This then gets sent to a mailer at MIT and off to the world. Worked great in the mid-1990s before spam made the Internet the time-waster that it has become. Now many of my friends don’t get the email at all. Hotmail, for example, when it sees something to: philg@mit.edu, bcc: happy_user@hotmail.com, sends it straight to the junk mail folder.
The challenge now is the best way to divide this up so that 100 individual emails are sent, each one from: philg, to: person_on_list. One answer would be to write a Perl script on the Unix machine. It would take two arguments, one the filename of a message and the other a filename with one email address on every line. The Perl script would look through the email addresses and send out an email on my behalf. (Anyone know where to get a script that does this already?; I checked cpan.)
Unix boxes typically have list managers such as Majordomo installed but as best as I can recall the email from these programs usually has a bulk look and feel, being sent to “fish-lovers-list” instead of the recipient. I’m thinking that these are likely to be trapped by spam filters as well.
Another answer would seem to be Microsoft Outlook. I switched to Outlook a couple of years ago when I got a Handspring Treo. So why not just add every brunch guest to my Outlook contacts folder and somehow spam them from my desktop machine? This has the advantage that I’m only keeping one database of contacts. This has the disadvantage that it doesn’t work when on the road. The regular Outlook distribution list mechanism produces an email with multiple To: recipients, which I don’t want to do because when people reply they often unintentionally reply to the entire list (in this case about 100 people). Are there VB scripts out there that will force Outlook to send one email at a time to each person on a distribution list? It seems as though there is a product, http://www.mapilab.com/outlook/send_personally/, that claims to do the job.
Finally there are Web services such as Evite. I don’t really need a count of who is going to come. These tend to more drop-in sorts of events so Evite is rather too heavy-handed. Also Evite, I think, subjects users to banner advertisements and I’m not sure that I want to surrender control of my database to them.
Thoughts? Scripts? Recommendations?
[Update: My favorite solution so far is Ryan Tate’s very simple Perl script, referenced in the comments at http://www.ocf.berkeley.edu/~ryantate/massmail.pl.txt; I got this working with the help of one of the Unix wizards at MIT who installed the Mail::Send library. I modified the script to add a Reply-To header and will eventually modify it so that it takes a database file with multiple fields per line, e.g., first name, last name, group membership (e.g., “kids”, “nerds”, “night_owls”) and can send selectively to members only of one or more groups.]
[Conclusion: The Weblog seems to be truly powerful. At 4 pm I asked a question. By 6 or 7 pm I had a raft of workable answers. By 9 pm I used one of those answers to invite more than 183 people to a going-away party on Wednesday evening (I head off to Greece on Thursday morning).]
[December 2004 update: A friend and I managed to enhance Ryan Tate’s Perl script and the version with more features is available at http://philip.greenspun.com/software/brunch-spam.pl.txt]
Philip,
I have .Net code that will do just this. But I assume as you are talking of Emacs that you are running Unix?
Alex
The Microsoft way is to use the Mail Merge feature in Word. It’s really quite easy and does exactly what you want.
The Unix way is with the fastmail(1) program.
You can try GNU Mailman, http://www.gnu.org/software/mailman/. Because it is a by-the-book list server, it is easy to train a mail client to recognize the traffic as legitimate. In test runs against hot mail it has never been classified as spam.
If you are bored, you could write a script. At work we have a site with 21,000 users, and every week we invite a fraction to take web surveys (we do market research for pharmaceutical and bio science companies, so our 21,000 users are all scientists that do research or clinical work in the life sciences field), which means we are sending a few thousand emails at a time. To avoid the BCC issue, I wrote something that basically uses a message template and dynamically generates each email. Each email is sent personalized, so it actually greets you by salutation, first and last name. The invitation URL has a unique ID and a one-use password, so they can be recognized by the survey. We also hide a header with the user id, the id of the survey and the date and time the email was sent, so we can track it back later of it bounces.
Because each email is sent individually we do not have trouble with getting snared by a spam filter by accident. The only problem is that we have to be careful when we write these emails because newer filters actually try to analyze the language of the message to try to see if a trend is detected, like for example a nigerian scam letter or some other kind of scam.
Oops, forgot a third option.
If you have a Mac available, try Mail Drop, http://freshsqueeze.com/products/maildrop/
Take you 5 minutes to write a perl script (or bash/ksh) that does such a thing.
$messagetplate = <
Content-type: text/html
To: {{RECIP}}
Reply-to: phxnews <breaking@phxnews.com>
Subject: {{SUBJECT}}
{{BODY}}
EOF
foreach $eaddr (@mailoutlist) {
open(SENDMAIL, “| /usr/sbin/sendmail -oi -t -odq”) or die(“Can’t fork for sendmail: $!n”);
$message = $messagetplate;
$message =~ s/{{RECIP}}/$email/;
$message =~ s/{{SUBJECT}}/$subject/;
$message =~ s/{{BODY}}/$body/;
print SENDMAIL $message;
close(SENDMAIL);
}
Or you can use package like MIME::Lite or Net::SMTP with even less lines…
If you have sendmail or another mailer installed you should be able to use Perl code like this:
open(MAIL, “|mail -s ‘subject’ someone@somewhere.com“)
|| die(“Unable to open mail: $!”);
print(MAIL $message);
close(MAIL);
Or do it in python:
import smtplib
smtpserver = ‘mail.ptd.net’
sender = ‘wabisabi@ptd.net’
recipients = open(‘address.txt’,’r’).readlines()
print recipients
mesg = open(‘message.txt’,’r’).read()
session = smtplib.SMTP(smtpserver)
for recipient in recipients:
session.sendmail(sender,recipient,mesg)
address.txt is your mailing list, one address per line.
message.txt is your message in this form:
Subject: <subject text>
<body text>
Simple and effective. If you want to make it slicker, you can capture the return from the session.sendmail() call into a variable and print that to a logfile or the screen. That way you catch any failed send attempts that might not otherwise generate a bounce message (ie, bad address format). I use a variation of this that reads the addresses from a membership database to send out newsletters.
–Glenn
Our school uses a phone tree. It’s much less efficient than writing a script, but arguably more fun.
“You have been invited to brunch at Phil’s house on Sunday. Please remove the first two people from the list below, and send the modified text of this message to each of them. Don’t break the chain! Phil broke the chain and lost his company. Paul Graham kept the chain going and cashed out.”
http://www.ocf.berkeley.edu/~ryantate/massmail.pl.txt
——————————————
#!/opt/local/bin/perl -w
### massmail
## Script for mass mailing your friends, one at a time.
## (Usage: massmail messagefile addressfile)
## (messagefile must include header section separated by blank line,
## for Subject etc.)
use strict;
use Mail::Send; #Part of standard Perl distro
my $message_file = shift or die “Usage: massmail messagefile addressfilen”;
my $address_file = shift or die “Usage: massmail messagefile addressfilen”;
my $sender = Mail::Send->new; #Our interface to sendmail et. al.
open MESSAGE, “<$message_file” or die “Could not open messagefile: $!”;
my @message = ; #Slurp in the whole message file
close MESSAGE or die “Could not close messagefile: $!”;
#Extract headers from message
while (my $line = shift @message) {
chomp $line;
last unless $line; #Header section separated by newline
$sender->subject($1) if $line =~ /Subject:(.+)/i;
$sender->cc($1) if $line =~ /Cc:(.+)/i;
$sender->bcc($1) if $line =~ /Bcc:(.+)/i;
}
die “Message file must include a header section seaparated by a blank line.n”
unless @message;
open ADDYS, “<$address_file” or die “Could not open addressfile: $!”;
#Send messages
my $address_count;
while () {
chomp;
next unless $_; #allow blank lines in addressfile
$sender->to($_);
my $file_handle = $sender->open or die “Could not launch mailer: $!”; #Launc
mailer
print $file_handle join(”, @message); #Put message text in email
$file_handle->close or die “Could not close mailer: $1”; #Send message
$address_count++;
}
close ADDYS or die “Could not close addressfile: $!”;
print “Sent $address_count messages.n”;
http://www.ocf.berkeley.edu/~ryantate/massmail.pl.txt
PS in my tests this one worked with my hotmail and yahoo accounts
Actually the python script looks shinier … R
yahoo groups works well for such lists. No, the members don’t have to have a yahoo account. Yes, it’s free. No, spam filters don’t get confused by it.
Thanks, everyone, for such helpful ideas, many of which I had not considered.
Joel: I don’t think that I want to use Mail Merge in Word because wouldn’t that result in people getting a Word document? I want them to get a three-sentence plain text email.
Alex: I have a Windows XP desktop and would be happy to try your .NET code, especially if it works with a MSFT Outlook contact database.
Pedro: no Mac available. Only GNU Linux (Debian, installed by experts at MIT) and Windows (XP Pro, maintained by an amateur at home).
You can use outlook to do what you want.
Put the desired reply address in From: and all the destination addresses in Bcc:
Each recipient will be able to see only their own address, not the others.
If you send 100 similar messages at the same time they may still be tracked and marked as spam. Many antispam software report a “fuzzy” hash of every message to an online db. If the same hash appears too often the other MTAs may mark the message as spam. Basically you just can’t e-mail a lot of people and be sure they will get a meessge (as you said thanks to spammers).
May be you can set up a private RSS feed and have your friends subscribe to it? At least this guarantees the delivery of the message, but it has some other issues.
Pipe your message, headers and all, into a script:
#!/bin/bash
cat >/tmp/msg.$$
for to in …list.your.recipients.here…
do
formail -I “To: $to” </tmp/msg.$$
| sendmail $to
done
rm /tmp/msg.$$
Bud: Thanks for the idea but the bcc: headers are what got my invites triggered as spam in the first place. Sending from a Windows machine instead of old-faithful Unix/Emacs won’t help.
presidentpicker: my friends don’t have RSS clients! If they are computer nerds all of their knowledge and systems predate RSS and the Weblog. But thankfully most of them aren’t nerds and email and Google are all that they know how to use. I think that you’re wrong about being marked as a spammer. The only computer that knows that I’ve sent a lot of similar messages is my own machine at MIT, which will be contacting foreign mailers one at a time. And the machine at MIT knows that the email came from a logged-in-at-the-shell user.
Chris: thank you for reminding us that real men only program the shell and that Perl is for pussies 🙂
No, Word MailMerge sends them a plain text email message, not a word document.
Really, just try it if you have Office installed; it works quite well and is much easier to use than you might think.
I mistakenly thought Mail::Send was part of the standard Perl distribution. The following script uses Net::SMPT, which is in the standard distro:
http://www.ocf.berkeley.edu/~ryantate/massmail3.pl.txt
but it’s kind of a moot point since Philip has Mail::Send installed.
R
check out http://www.dadamail.org/
I am not sure if Joel has tested Mail Merge with email recently; I think you end up having to click “OK” to email each person, due to security patches for Outlook to guard against email viruses automatically spamming thousands of people using your machine.
Okay, the original problem seems to have been solved already, but there’s still something I don’t understand: isn’t the BCC header supposed to be stripped by the mail server *before* sending out your messages, so that the recipient never even sees them? (That’s why it’s “*blind* carbon copy”, isn’t it?). Maybe the Unix machine you’re using to send out your mailings is misconfigured (then again, if I was in your position, I’d also have better things to do than trying to figure out sendmail.cf…)
Mailman when configured to use “Full personalisation” for a mailing list will do as you request.
Your blog is powerful because you have “fans.” If I were to post an important question on my blog, it might be months before someone clicked on the “read next blog” link, randomly came to my page and by chance could contribute an insightful answer.
Philip, you have overlooked the obvious: build an anti-corporate solution to Evite, hosted at photo.net or philip.greenspun.com on the model of the now defunct and soarly missed Birthday Reminder service, make it free for users, and contribute further to the online community.
Philip,
Why not share the database aspect of your solution to your readers?
Thanks!
Rick
I modified the script to add a Reply-To header and will eventually modify it so that it takes a database file with multiple fields per line, e.g., first name, last name, group membership (e.g., “kids”, “nerds”, “night_owls”) and can send selectively to members only of one or more groups.]
I do exactly this to keep my friends updated with events, but I don’t do it often enough to bother with a real mailing list. I just keep a text file with all of the target email addresses in it, one per line. Then I write the message as a text file. Then I just use sh (bash, whatever) to send them out – I wouldn’t even call it a script it’s so short.
$ for address in `cat addresses.txt`
do
mail -s “Hey everybody” $address < message.txt
done
It sends out each message individually, and it should work on pretty much any unix-ish system.
183 people, in a condo with no elevator? You’re kidding, right?
Given a text file of email addresses recips.txt and the body of the email message in message.txt, the Rebol code would be:
foreach mail read/lines %recips.txt [send to-email mail read %message.txt]
This script would run identically on 40+ platforms on Rebol’s small 300Kb interpreter. Rebol is built specifically for cross-platform network scripting & DSL building, http://www.rebol.com
Frank: I will, of course, make the extended script available. But “eventually” means “at the very least after I get back from Greece”. So I probably won’t touch the code for at least a few months.
Michael: I did invite 183 people. One is a friend who moved to Shanghai but still asks to be kept in the loop. I don’t expect him. I don’t think more than 50 have ever shown up to one event and that was spread over 4-5 hours so there were never more than 40 guests in the apartment at any moment. I have a 4.5-ton air conditioner…
I love Perl dearly, but this is a job for the Unix shell. First, we’ll clone your massmail script in one line. Next, we’ll add support for combining groups of friends. I have used these for my own parties.
Suppose you have your email addresses in a file called emails.txt, and the message is in message.txt. The command is:
for e in `cat emails.txt`; do mail -s”Party at Philip’s Tonight” $e < message.txt; sleep 5s;done
(The “sleep 5s;” introduces a delay between message sending. Having read that my ISP, Comcast, thinks bursts of email-sending indicate a virus-infected machine and will shut off outgoing mail server access, the delay may be prudent. But feel free to take it away.)
Now, let’s extend this. Suppose you have multiple lists: nerds.txt, students.txt, and singlebabes.txt. There is probably no overlap between nerds and singlebabes, but a member of either group could be in students. Challenging? Hardly:
for e in `cat nerds.txt students.txt singlebabes.txt | sort | uniq`; do mail -s”Party at Philip’s Tonight” $e < message.txt; sleep 5s;done
The programs used–cat, sort, uniq, mail, and sleep–should be available on any Unix machine. Hope this helps.
Shimon, my script takes the messagefile and the addressfile as arguments, as well as the subject line (as specified in Philip’s post). It’s no coding achievement, but you have hardly cloned it. Which is too bad because I’d like to see how this is done in shell.
Phillip, I think that the power of the weblog is reserved for those who are truely popular (the causes of popularity being many). I suspect that the majority of your readers (Mr. Spolski being the only exception I immediately recognize) would find that any questions they posted on their weblogs would go quite a while before they were even read, let alone intelligently answered.
;The puzzle solved in one line, in Rebol language (www.rebol.com). No need sendmail and others bad programming things:
send/only read/lines %mail-list.txt read %message.txt
; Rebol really rox! 😉
BTW, a really big mail-list can be mailserver sorted, due server performance reasons.
This SQL query may helps the task:
# SQL query by Ari Stopassola Jr.
# Order emails by server.
SELECT DISTINCT
email,
SUBSTR(email,(POSITION(‘@’ IN email)+1),(LENGTH(email))) AS ordered_email
FROM mail_table
ORDER BY ordered_email;
Oops! Error: the correct command line is this:
send/only to-block read %mail-list.txt read %message.txt
Or refined:
send/only/subject to-block read %mail-list.txt read %message.txt “My subject”
I know I am late with this, but I hadn’t checked in on Philip’s blog recently and just stumbled on this thread today.
I want to thank Joel for the reference to the mail merge/email tool in Word. I just gave it a try, and it works great for my needs!
Also, someone mentioned the security feature that forces the user to click “Yes” over and over again for each email — After a few minutes of googling, I found out that this only happens when you select “plain text” or “attachment.” If you select “HTML,” the emails are sent without having to verify permission each time.