The Verge has an incredible post up about “content moderation.”
I have always felt that the hardest part of running an Internet business was insuring the trust and safety of the users and I am thrilled to see some light being shone on this part of the business.
There is always so much talk about the product and engineering parts of the business and so little about the extremely difficult work that goes into policing the product. And yet when you look at churn, so much of it in a scale Internet business is a result of users running out of patience with spam, trolling, and worse. This comment by Dick Costolo from the piece is telling:
“We lose core user after core user by not addressing simple trolling issues that they face every day, We’re going to start kicking these people off right and left and making sure that when they issue their ridiculous attacks, nobody hears them.”
Well as the post points out, that is not so simple. And, of course, there are free speech issues too. I constantly hear people criticizing Twitter for blocking users.
But trolling, as bad as it is, is not the worst part of this work.
A trust and safety team has to deal with the most awful kinds of people and actions imaginable. I often suggest that everyone should sit in a trust and safety organization for a week. Then a lot of the conversations we have about free speech, privacy, and the like would get a lot more nuanced. There are bad people out there doing bad stuff.
Sadly, as I have seen again and again, startups don’t understand how challenging these problems are going to be until some sort of situation forces them to react. Then they throw people at the problem but never their precious “engineering resources.” When trust and safety, fraud, compliance, and moderation teams start getting their own engineering resources, something that often takes years to happen, then you know the company is finally acknowledging the importance and seriousness of the work.
The people profiled in this Verge story are heroes in my book. They do hard work, are not paid as much as they should be, and they are working in incredibly difficult and dangerous (for their mental health) situations. It is high time we start acknowledging them and their work and investing in it.
