“Unethical Scope Creep”

Kaitlyn Ouverson (a friend and former student of mine) invited me to do a talk with her on the subject of what she calls ‘unethical scope creep,’ which is that tendency some tech companies have to collect personal data far beyond what is actually used to create a personal experience with their product.

The idea of wishing for something (as opposed to going through an entire process of working towards achieving that thing) shows up in a lot of stories. The magic that’s often employed in stories about wishes seems like a fitting metaphor for our current state of technology, which grants achievements, such as arriving at a destination, without requiring much effort on behalf of the user.

Smart devices, which people tend to carry with them wherever they go, offer to make all sorts of tasks easier to accomplish, but there’s always a cost. By wishing for something rather than achieving it through non-magical means, one misses out on opportunities to learn things and develop skills.

Technologies could incur those same costs (such as forgetting how to navigate, fly a plane, and call people on phones where their numbers aren’t saved), but also have a price hidden in their terms of service (end user license agreements). The expectation isn’t that each user reads the terms and enters into a consensual agreement with the company to use their product. Essentially the company asks the user to trust them (and then gives ample examples of why nobody should ever trust them- see Google, Facebook, Equifax, etc.) but doesn’t really give them a choice not to (what, you want to be lost forever?).

Smart devices (phones, watches, toasters, etc.) provide all sorts of useful features that motivate users to carry their devices with them everywhere. This is great for the various companies that collect information about location (regardless of whether their services are location-based), but those companies don’t stop there. The opportunity cost of people interacting directly with the world around them rather than through smart device intermediaries motivates companies to interject their devices, apps, and services, into every interaction and transaction.

Once the device starts granting the user’s wishes (like to get to where they want to go) it can establish itself as the go-to device for achieving whatever needs arise (this is what Siri, Alexa, whatever voice thing Google is doing, and Cortana are doing). The cost to the user doesn’t change – it’s whatever knowledge/skills they miss out on by relying on a device to do something for them + all of the things they agreed to give up under the terms of service.

Any data that the device could gather is gathered and analyzed to look for patterns. Statistical analyses are used to determine what’s similar across users (signal) and what’s different (noise), which could then be used to make predictions about user behavior. Those predictions could happen in the form of “if this ad appears for this type of user, they are likely to click on it.”

This might seem innocent… just one smart device talking in secret to untold numbers of companies and other devices about its user, but the information being peddled here isn’t just biographical – what company would care about the interests and habits of a single individual? What’s being sold is the power to influence behavior on an individual and mass scale.

It’s the power to tell someone exactly what they need to hear in order to get them to do the thing that the company wants (regardless of whether it’s something the user actually wants or needs). The alternative, for those disinclined to grant the wishes of the tech company, is to expend some measure of mental energy combatting the behavioral influence every time they interact with a device.

Plenty of people would defend targeted marketing because it sounds a lot better than the slapdash kind (at least the ads are relevant to me!), but how many people end up buying things they don’t need (spending money they could otherwise save), creating waste, and entering into endless cycles of credit card debt?

Targeted marketing isn’t the only end for collecting user data. Devices don’t just sit idly waiting for people to engage with them because there might be other activities people want to engage in when they are bored.

They’re designed in a way that promotes engagement with them above all other considerations. Sure, you can go for a walk, talk to friends and family, pick up a new hobby, fulfill a lifelong dream… OR you can spend your time scrolling an infinite feed of forgettable information.

Any of the former activities are in the best interest of the person (‘user’/customer), but the latter activity is entirely in service of the tech company. Did you want to be sad? angry? frightened? annoyed? scrolling through an endless feed could easily trigger any of those emotions (those tend to be the articles and videos that get the most play).

Kaitlyn then talks about different actions that researchers and designers could take to keep their technology out of unethical scope creep and make sure it doesn’t just grant empty wishes. She’s got great ideas and is skilled at synthesizing them and I’m so proud of her.