What are the people doing?

Jeffrey P. Bigham

5/30/2023

Author’s note:  for the past few months, I’ve been writing about generative AI, but haven’t been publishing what I wrote – in part that was because my blog was broken, but also with things changing so fast I wasn’t confident in what to say. Now that some time has passed, I’ve started publishing them, especially those that still seem relevant, as reflections on a fast few months.

It’s an exciting time if you work at the intersection of HCI and ML. With the release of chatGPT, generative models passed a qualitative threshold of capability and access that feels different than before. Yet, nobody is quite sure where all of this will go. In the near term, everyone is racing to deploy the low-hanging fruit (another AI autumn harvest??) – numerous companies have announced GPT-4 integrations via 30-second videos showing autocomplete and automation for everything!

What futures could result from the evolution of this technology?

Some predict a not so distant world in which most content, from text and code to music and movies, is “automatically generated” and most computing tasks are automated away. Aside from the huge disruption this would cause, the notion that all content will be automatically generated and all tasks will be automated is unsatisfying because of what that description leaves out.

What are the people doing?

“What are the people doing?”

Do people still provide prompts or other guidance on what to generate? Do they edit or curate the content that is created? Do people follow along as the machine automates away what they once had to do themselves, or do they just assume it’ll be done right? They must at least provide feedback in some way, right? They must learn what the system can do well or how it’s likely to approach a task? They must convey their preferences?

What do these future people want to be doing? If we’re still around as those future people, what do we want to be doing?

This is the primary question we should engage with, not ceding agency to some mythical unsteerable technological march.

As we think about what people will be doing in the future, consider the following examples of people grappling what to do as we use text generation –

Retaining Agency in the Presence of Generated Text
In my research lab at CMU, Stephanie Valencia-Valencia has for several years been exploring “agency in AAC” systems, which are a class of technical systems that people who have difficulty vocalizing speech can use to speak. Because typing with an AAC takes some time, many projects over the years have explored how to use predictive text (and now generated text) to make typing faster. In a study Stephanie ran, close conversational partners (parents, caregivers, etc) often spoke on behalf of AAC users in an effort to improve efficiency, but often AAC users would have preferred to say something else; often, the conversation would move on and change topics while the AAC user was typing, making the AAC user’s eventual comment seem not relevant to the current conversation [Valencia-Valencia et al.]. Outside of the AAC domain, we know that people are influenced by the text that is generated, for instance being willing to accept suggestions that alter their sentiment in exchange for efficiency [Arnold et al.]. How will people engage with text generation going forward? Will we design tools that protect user agency or will we focus squarely on efficiency? How do we understand agency in the midst of augmented creativity?

Coding Support
Most programmers have by now had a chance to work with code generation like codex/copilot. Studies are coming out that claim that code generation systems really do make programmers faster. I’ve personally found it fascinating to work with copilot. When you first try it, as a skeptic like me, you are immediately surprised and delighted by how well it works. It autocompletes things so much better than my IDE did before. Some of that is LLM magic, but some of it appears to be related to being smarter about the context it considers, e.g., I’ll edit some code in one file and then move to the next, and variables from that previous file are included appropriately in the generation. I’ve also found it super helpful for prototyping in languages or domains where I’m not super familiar with the tools and frameworks that are available. I’m very clearly still programming, and I don’t see how the current technology would end that. I’m         deciding what the program should do, and I’m still structuring how it should do it. I’m iterating. I’m debugging. But, I can move faster, especially but not only in prototyping, because I don’t have to do as much of the stuff that is hard for me – I don’t have to remember exactly how everything works, what APIs exist in a SDK I haven’t used before, and have to type a lot less of what I do mostly remember. Maybe debugging is harder, but a lot of code that is written really isn’t that interesting or difficult. Much of it is predictable once it is contextualized. How do we make more tasks have straightforward benefits like code generation? Coding is weird. Coding languages are designed specifically to be unambiguous instructions to computers, so maybe these benefits aren’t as possible to bring to other domains. How much will programming change going forward? Right now, it seems largely the same but slightly faster.

Only the Important Parts of Writing (if we can figure out what those are)

There is a lot of concern that writing is going to go away, and, worse, we might never learn to write. Educators worry about their students writing essays using chatGPT, and I’ve heard many people say things like, “it’s going to be so great when I don’t have to write X anymore,” where X is something necessary they have to write but that they don’t like doing, e.g., grant proposals, business plans, project status updates, etc. Meanwhile professional writers often say that they will never use tools like chatGPT. I wonder if this difference in perspective may also indicate a likely difference in what we will do with these tools and maybe even what we should aim for – how do we get rid of the monotonous rote parts of writing, while retaining the useful parts.

It was highly useful for me to write this essay, not only because maybe someone will read it and find it valuable, but also because I had to wrestle with putting the ideas down and iterating and building on them. It wasn’t just the artifact but also the process. In other cases, we write formulaic text for other purposes, and it’s certainly “ok to use chatGPT for the tedious parts of your job”. Even as tools can generate increasingly good enough content, how do we protect and accelerate the human benefits of wrestling with ideas that accompany manual content creation?

Finally, I think the current moment may provide a useful time to reflect – why are we writing so much? As an academic, I write a bunch of letters – tenure letters, PhD program letters, project reports, etc. Most, but perhaps not absolutely all, of what I write in those is completely predictable based on text that already exists, e.g., past letters I’ve written, research statements from the candidate, papers we’ve written with the funding we received. Much of what I’m doing is lending my name, but I’m also demonstrating via a kind of social proof that I feel strongly about this candidate because (until now?) I had to actually spend my time to write the letter. How will we separate out the work of writing from the message, or will they be inseparable?

If people are involved (and, they will be, otherwise who is this content being generated for and who is the automation benefiting?), then it makes sense to forefront what they’re doing. Ultimately, what people are doing is all that matters, at least to me. It seems obvious that people won’t want to be passive consumers of generated content. People will be involved. There will still be creatives who figure out how to generate and curate content that speaks to specific people, the moment or the context. There will be advocates and activists actively working against the predetermined statistical likely outputs.

People will want to own content that they see as theirs and will find ways to profit from it. They will build brands. They will generate bad stuff that nobody likes. They will fail, and they will succeed. People will wrestle with content generation because it benefits them, and because they want to be like someone they saw on TikTok. People will perform. People will want agency over what happens, and despite their desires they will often generate content that looks feels sounds like the content everyone else is generating.

Agency. Humanity. Struggle.

What People Will Be Doing Isn’t Pre-Determined

The mindshift that happens somewhere along the way of asking what people will be doing is the realization that what people will be doing in the future isn’t pre-determined.

As we build the tools people might use, we are choosing the likely path for how these systems will work, what they will make easy and what they will make hard. If all we do is build systems that target generating entire essays or songs or movies from start to finish with little to no human intervention, then we should not be surprised if people aren’t as involved with content generated in that way. If we instead build systems oriented toward human agency and engagement, then people will engage.

For the past few months, I’ve been helping to introduce local entrepreneurs to various generative AI tools. From this experience, it’s pretty obvious (to me!) that a lot more expertise is currently required to benefit from these tools than is commonly acknowledged – the iteration and experimentation required to generate good enough prompts is surprisingly technical (I personally relate it to how surprisingly challenging it is to do good keyword search), and most people who are effectively using generative AI are integrating it with many other traditional tools (e.g., Photoshop). As we design and build the tools of the future, we are also making choices about who will use them based on what foundational skills they require.

Already our language doesn’t adequately cover what is really happening. It is frustratingly common to hear about AI-generated content, but so rare to hear about the rich interplay of AI and humans that lead to interesting content and experiences. If we have already ceded our agency to AI in our descriptions, can we hope to build futures that recover our agency and make it primary?

I don’t know exactly what people will be doing, but what people are doing should be at the forefront when discussing, designing, and innovating AI futures. This post has focused mostly on the interaction level of what people will be doing, because I feel like that discussion has been the least developed – we should also think about how these technologies will negatively impact people, e.g., by repackaging their work, reinforcing negative biases or expressing harmful or unsafe positions. Even then, “what will people be doing?” is a great question to ask – if I feel the original content I create is likely to be gobbled up by a model and consumed disassociated from me, I might be less likely to share that content in the future. Systems that act on behalf of people without their involvement are likely to be much less safe than those that work to support and build human agency. Already, we have seen contributions to stackoverflow have gone down. How and why are the people of the future creating original content? How are the people of the future improving themselves and strengthening their agency working with machine learning systems?

Focus on the Futures We Want

“What are the people doing?” can be a valuable starting place for thinking about the future we want. But, this is just a short way to capture the value that Human-Computer Interaction, with its many methods of understanding and innovation can bring. We need to design, build and understand generative AI, not only following from what the early versions of the technology can do today, but imagining a future where people are working and engaging with generative AI in the ways they want.

Let’s focus on people and try to build the futures we want to live in.

This page and contents are copyright Jeffrey P. Bigham except where noted.
Blog posts are not intended to be final products, but rather a reflection of current thinking and/or catalysts for discussion, like tweets but longer.