Quite a few years ago, when I was much younger and still naively believed the BDSM, fetish, and leather subcultures deserved any respect, I saw the writing on the wall and wrote an essay titled “FetLife Considered Harmful,” in which I argued that this subculture’s digital poster boy, FetLife.com, was “the worst of both worlds” with respect to both online and face-to-face community. Working with a number of collaborators, I developed a number of software tools to help people use FetLife in a safer and more personally empowered way, perhaps the most famous of which is the Predator Alert Tool for FetLife. But another tool was called FetLife Export, and it makes a wholesale copy of a FetLife account’s activity.
Using FetLife Export, you can make a complete, searchable snapshot of your own or any other FetLife user’s entire account activity with just a few clicks. The tool is very similar to wget in that it makes an archive of web pages, but in this case the archive is specific to a given FetLife user’s postings, likes, comments in group discussions, wall-to-wall conversations, and other FetLife-specific features. This is handy because it means that when you see someone make a racist or sexist remark on a FetLife thread, you can download a copy of their entire FetLife history in one fell swoop—everything they’ve ever said in public view on FetLife—and then retroactively search their past postings for other, similar or related remarks.
What they might once claim is an “isolated incident” will seem very, very different in the light of a complete historical log of their behavior on FetLife.
I’ve been using this tool regularly since it was first released. If you wonder how I can be so “plugged in” to what’s going on in the BDSM Scene, how I can so quickly connect the dots between who-knows-who, it’s partially because I make and use these archives to find patterns that you can only see when you surveil the entire BDSM Scene in near-real time. It’s an invaluable tool for anyone doing systemic analysis, muckraking, or investigative reporting about the BDSM Scene.
In this brief guide, you’ll learn how to install and use the FetLife Export tool on your own computer, and you’ll walk through a sample invocation with the help of a simple, text-based wizard.
What is FetLife Export?
At its core, FetLife Export is a very simple perl script that asks for a FetLife username and password, then logs into FetLife using that account and downloads the activity of whatever account you tell it to get. It can download your own account’s activity, in which case it also archives private messages, and it can also download other people’s activity. To do this, it simply reads the target account’s activity feed and follows all the links. So if the target user has “liked” (in FetLife jargon, “loved”) a different user’s picture, that other user’s picture and all the comments on that picture get downloaded, too.
Installing FetLife Export
Since FetLife Export is a perl script, it can be used on any computer that has perl, including Mac OS X, Linux, and Microsoft Windows. The first two systems come with a suitable perl out of the box, but Windows users will probably need to install perl themselves if they haven’t already.
In addition to perl itself, the FetLife Export script uses a number of functions provided by other perl scripts, called perl modules, that your computer may or may not already have. The FetLife Export README file explains:
Most systems will have the modules you need. However, if you experience errors running
fetlife-export.pl, you may also want to install all the required components yourself. To do this, run the following commands after you’ve installed your Perl:cpan App::cpanminus cpanm WWW::Mechanize cpanm HTML::TreeBuilder cpanm String::Escape cpanm Unicode::Escape cpanm LWP::Protocol::socks # Optional. Only needed if you'll use a SOCKS proxy.
That’s a bit terse, so head on over to the “Installing Perl Modules” guide on the Comprehensive Perl Archive Network (CPAN) website for more detailed instructions. The general idea is that if you try to run the tool but get an error that starts with something like this…
Can't locate WWW/Mechanize.pm in @INCthen you probably need to install one or more of the listed modules. In this example, the module that’s missing is the WWW::Mechanize one, so you should run the command cpanm WWW::Mechanize at your command prompt. Similarly, if you get an error that looks like this:
Can't locate Term/ReadLine.pm in @INCThe solution would be to install the Term::ReadLine module with a command such as cpanm Term::ReadLine.
If you’re having more trouble getting it installed than you know how to handle, feel free to contact me (publicly, privately, however you prefer).1
Walkthrough: Use the FetLife Export Wizard on Mac OS X
If you have a Mac, you can use the FetLife Export tool after it’s installed simply by double-clicking on the fetlife-export.command file that comes with the software (download the whole kit ‘n’ kaboodle by clicking on the “.zip” link for the latest version). Opening the .command file will launch a Terminal window that looks like the following screenshot, and begin walking you through a series of question-and-answer steps to help you make an archive of a user’s FetLife history:
FETLIFE EXPORT WIZARD
This software is released to the public domain. Fuck copyright.
Make a copy of your own or any other user's FetLife account.
You will need a FetLife account to use this tool. If you don't
have a FetLife account, you can easily create one at this page:
https://FetLife.com/signup
Creating a FetLife account is free and does not require a valid
email address or other personally identifying information. For
maximum security, create an account while using the Tor Browser.
https://torproject.org/
Type your FetLife username, then press return: SockPuppet
This first prompt asks you for the username of the FetLife user that the tool logs in as. This can be any account for which you can successfully log in to FetLife.com. If you don’t want to log in as your primary account, you can always create a secondary, secret account (a “sockpuppet”) that you use solely for the purpose of sleuthing around the site. If you want to make extra sure an account you create can’t be traced back to you, consider using the Tor Browser to register that account rather than your regular Web browser.
After you type your username, press return to continue to the next prompt:
Type the name of a folder to save to, or leave blank to use the default shown.
If this folder does not exist, it will be created.
Save to folder [/Users/maymay/Desktop/fetlife-export]: my-fetlife-archive
This next prompt is asking you for the name of a folder to save the archive in. It suggests the folder from which you run the tool, by default, but you can type whatever name you want here. In the example above, I’ve typed my-fetlife-archive, so the exporter will create a folder called “my-fetlife-archive” inside the “fetlife-export” folder on my Desktop, because that’s the same folder where the exporter program is on my hard drive.
Once you’ve chosen a name for the folder of your new archive, press return again to continue to the next prompt:
Type the ID number of the export target. For example, if you want to
create a copy of JohnBaku's entire FetLife history, type: 1
Leave blank to automatically detect and use SockPuppet's ID number.
Export target's user ID:
Now the exporter wizard is asking for the FetLife ID number of the user whose activity you want to download. The ID number is the last part of the URL in the profile of a FetLife user. So, for example, the profile URL for JohnBaku’s FetLife account is https://fetlife.com/users/1, so his user’s ID number is 1. If you want to download your own FetLife history (perhaps to make a backup of your account, or to keep a local copy so you can more easily search it, as I do) then you can leave this blank and just press return immediately, in which case the exporter will automatically detect and use the ID number of the account you’re logging in with. That’s what I did, in the example above.
The next prompt asks whether you want to use a proxy. We discussed proxies in detail when we learned how to use BitTorrent.
Use a proxy? (Leave blank to make a direct connection.)
If you want to use a proxy, enter the proxy's URL here. For example,
to make use of a default Tor Browser, type: socks://localhost:9150
Proxy URL: socks://localhost:9150
You can leave this blank to make a direct connection from your computer to FetLife.com, same as you do when you log in to FetLife normally. But if you want to make sure that no one can figure out that you’re the one making a copy of whatever it is you’re copying, then I strongly suggest downloading and running the Tor Browser and then entering its proxy address here, which is socks://localhost:9150, as shown in the example. As before, press return to move on to the next prompt:
fetlife-export.pl will now run with these parameters:
fetlife-export.pl --proxy=socks://localhost:9150 SockPuppet /Users/maymay/Desktop/fetlife-export/my-fetlife-archive
When prompted next, enter the password for SockPuppet.
Password: The wizard shows you the full command line whose arguments it asked for, in case you want to run the command again without its help, and then prompts you to enter the password associated with your account. This is functionally identical to typing the password directly into FetLife’s log in form on its website, except that what you type won’t show up as bullets or stars, so be careful to enter it correctly. When you’ve entered your password, press return a final time to run the exporter itself.
You’ll see text appear in stages that, when finished, will look something like the following:
userID: 163357709
Loading profile: .
Loading conversations: . 8 conversations found.
Loading wall: . 5 wall-to-walls found.
Loading activity feed: ........................... 842 statuses found.
78 pictures found.
19 writings found.
249 group threads found.
Downloading 842 statuses...
Downloading 78 pictures...
Downloading 19 writings...
Downloading 249 group posts...
logout
[Process completed]
When the exporter is done, you’ll see a folder of the name you specified, inside of which is another folder called fetlife, and inside that one will be a complete exported archive of the user’s FetLife account:
The archive contains any public images the target user uploaded to FetLife, along with all public posts they’ve made, all saved as HTML.2 Since the downloaded archive is just a bunch of HTML files, you can use your computer’s built-in search tools (Mac OS X’s Spotlight or Windows Search, for example), to find posts and comments with any given phrase in a matter of seconds.
Additional, advanced FetLife data mining tools
While the above exporter software is useful for a lot of reasons—it helps FetLife users keep backups of their account data, saves their FetLife content in a portable, HTML format, and more—there’s another, additional tool that might interest you if you do any serious investigative work amongst the (obscenely corrupt) BDSM Scene, called FetLife Maltego.
This tool is a plug-in for Maltego, an application designed to automate the process of collecting information about people from public sources such as their social media profiles, domain name registrations, email addresses, phone numbers, mutual friend connections, public records databases, and more. Both Maltego itself and the FetLife Maltego plug-in (called a local transform) are beyond the scope of this guide, but there are some good Maltego beginner guides and Maltego Tutorial videos out there.
Happy muckraking! :)
- One of the things this project is still missing is a simple packaged executable, such as a
pppacked one, so if you want to help improve this process, the maintainers welcome patches via GitHub. [↩] - Comments in the HTML file also record the original URL that the page was created from, in case you want to give someone the live link on FetLife.com, if it still exists. [↩]
