A 1985 interview with Professor Weizenbaum.
What the professor who invented one of the first chatbots had to say about technology and society --- especially education.
I recently came across an old interview with Professor Joseph Weizenbaum, in a 1985 edition of the MIT newspaper. The interview is fascinating, as what we hear is the thinking of a man who not only deeply understands, but also even created, the very technology he is critiquing.
You can find the full original interview here, but I’ve pasted excerpts in-line below along with my own commentary. I promise, it’s worth the read (less for my commentary, and more for his thoughts).
Also, a couple other things about Professor Weizenbaum:
He created a chatbot called ELIZA, and observed people using it. He was shocked to realize that many people poured their hearts out to the program, and even attributed human-like feelings to it. Many were convinced ELIZA was intelligent and understood them. Sound familiar?
He helped design the first computer banking system in the United States, for Bank of America.
In his book Computer Power and Human Reason, he makes the distinction between deciding and choosing, asserting that a decision is essentially a calculation (which can be programmed), but that choice is a judgement—not a calculation.
He argued that the computer preserves that status quo, and is an obstacle to progress and innovation.
Ok, excerpts from the Weizenbaum interview below:
Q: What, if anything, do you think should be the role of the computer in education?
A: Yours is an often-asked question. In a sense, it is upside-down. You start with the instrument; the question makes the assumption that of course the computer is good for something in education, that it is the solution to some educational problem. Specifically, [your] question is, what is it good for?
But where does the underlying assumption come from? Why are we talking about computers?I understand [you asked because] I’m a computer scientist, not a bicycle mechanic. But there is something about the computer -- the computer has almost since its beginning been basically a solution looking for a problem.
“… a solution looking for a problem.” There are certainly things it has been, and can be, helpful with… but it is ultimately a tool that we look at and think… ok, how can I / we use this?
He goes on to say:
People come to MIT and to other places, people from all sorts of establishments -- the medical establishment, the legal establishment, the education establishment, and in effect they say, “You have there a very wonderful instrument which solves a lot of problems. Surely there must be problems in my establishment -- in this case, the educational establishment, for which your wonderful instrument is a solution. Please tell me for what problems your wonderful instrument is a solution.
The questioning should start the other way -- it should perhaps start with the question of what education is supposed to accomplish in the first place. Then perhaps [one should] state some priorities -- it should accomplish this, it should do that, it should do the other thing. Then one might ask, in terms of what it’s supposed to do, what are the priorities? What are the most urgent problems? And once one has identified the urgent problems, then one can perhaps say, “Here is a problem for which the computer seems to be well-suited.” I think that’s the way it has to begin.
This is exactly right. When considering how/where to use a new product, the first question should always be about goals:
What is OUR goal, as an institution?
What problems do we have / what is obstructing our goal?
Which of these can this product potentially help with?
What would the goal of this product be here in this institution?
I dug into this a few weeks ago in this post on the difference between learning to use technology VS using technology to learn.
Q: What are the problems of the educational establishment?
A: The first priority has to be, it seems to me, to lend to those to be educated a mastery of their own language so that they can express themselves clearly and with precision, in speech and in writing. That’s the very first priority. The second priority is to give students an entree to and an identity within the culture of their society, which implies a study of history, literature, and all that.
And the third, very close to the second, is to prepare people for living in a society in which science is important, which means to teach them mathematics, or at least arithmetic, and the fundamental skills important to observing the world.
A school system which meets these main objectives might think about introducing something new. Meanwhile, researchers should certainly work on innovative education -- including computer-aided education. But we ought not to use entire generations of schoolchildren as experimental subjects.
In part, this response is based on my belief that what primary and secondary schools teach about computers now is either wrong or can be learned by a reasonably educated person in a few weeks.
Ok so much to unpack here. But 3 things really stand out to me:
He essentially says that before schools start considering innovative techniques, must first be delivering on three key priorities: (1) mastery of language, (2) connection to culture/society, and (3) fundamental skills for observing, understanding, and navigating the world.
He explicitly says “… we ought not to use entire generations of schoolchildren as experimental subjects.”
He asserts that what is being taught about computers is either wrong, or easy to learn on your own.
I agree with all of the above.
Q: Where do you think the study of ethics fits in[to] all that?
A: Without being able to express themselves clearly, without having a mastery of their own language, I think it would be very difficult, to the point of impossibility, for people to think through ethical considerations. I think that mastery of the languages has to be first even in that respect as well. In the study of history of the culture, the literature of the culture, the politics of the culture, and so on -- that’s where I think ethics are exemplified.
Again, he’s focused here on mastery of language as the foundation of everything else.
He goes on to incisively critique the school system:
A question that we should ask is: Now how well are the schools fulfilling the first priorities? Certainly the answer with respect to language is miserably, absolutely miserably.
MIT certainly gets the cream of the crop of the product of the American school establishment, yet there was a headline in your paper just a few months ago which said that out of a 1000-some freshmen who took the writing test, 800 flunked. How is it then for people who are going to junior colleges? How does it look for people who aren’t going to college at all? How does it look for people who dropped out of school when they were 14 or 15? Clearly the American school establishment is failing very seriously.
Remember: this interview was in 1985, and he thought it was clear that schools were failing to meet their basic goals, and that kids were far from mastering language. Also: It is WILD that 800/1000 MIT freshman failed whatever their writing test was at that time. IN 1985. (Do I even want to know how that would do today?)
It is terribly important to ask the reasons the schools are failing so miserably. I think that even if one could show that the introduction of the computer into schools actually effected an improvement, say for example in reading scores, even if one could show that, the question, “Why can’t Johnny read?” must still be asked.
There is a very good reason that questions of that kind are uncomfortable. When we ask this question, we may discover that Johnny is hungry when he comes to school, or that Johnny comes from a milieu in which reading is irrelevant to concrete problems or survival on the street -- that is, there is no chance to read, it is a violent milieu, and so on.
You might discover that, and then you might ask the next question: “Why is it that Johnny comes to school hungry? Don’t we have school breakfast programs and lunch programs?” The answer to that might be, yes, we used to, but we don’t any more.
Why is there so much poverty in our world, in the United States, especially in the large cities? Why is it that classes are so large? Why is it that fully half the science and math teachers in the United States are underqualified and are operating on emergency certificates?
Well, the problems in education have certainly been consistent… Also, I absolutely love that this computer scientist sounds like a sociologist.
When you ask questions like that, you come upon some very important and very tragic facts about America. One of the things you would discover is that education has a very much lower priority in the United States than do a great many other things, most particularly the military.
It is much nicer, it is much more comfortable, to have some device, say the computer, with which to flood the schools, and then to sit back and say, “You see, we are doing something about it, we are helping,” than to confront ugly social realities.
And here, he absolutely nails it. “It is much nicer, it is much more comfortable, to have some device, say the computer, with which to flood the schools, and then to sit back and say, “You see, we are doing something about it, we are helping,” than to confront ugly social realities.”
Technology allows us to create the illusion of action and problem-solving, while allowing us to essentially mask the problem.
And kudos to this interviewer (who I presume was a student at the time) for asking the right follow-up questions…
Q: What do you think should be done instead?
A: I think that further questions should be asked, always “why?” just in the way I’ve indicated. And then I think it becomes necessary to respond to what these questions uncover, to change the fundamental facts that account for the difficulties, as opposed to papering them over by introducing some technological fix.
AGAIN. We can only address problems when we bring them to the surface. If we avoid or cover-up reality with short-term superficial fixes, we make it harder to tackle (or even see) the real problems. (Problems which, by the way are going nowhere and will absolutely resurface, likely even more complex than before.)
Q: Do you think that the computer is creating a technical elite, reinforcing old power structures, or remaking American society?
A: I think the computer has from the beginning been a fundamentally conservative force. It has made possible the saving of institutions pretty much as they were, which otherwise might have had to be changed. For example, banking. Superficially, it looks as if banking has been revolutionized by the computer. But only very superficially. Consider that, say 20, 25 years ago, the banks were faced with the fact that the population was growing at a very rapid rate, many more checks would be written than before, and so on. Their response was to bring in the computer. By the way, I helped design the first computer banking system in the United States, for the Bank of America 25 years ago.
Now if it had not been for the computer, if the computer had not been invented, what would the banks have had to do? They might have had to decentralize, or they might have had to regionalize in some way. In other words, it might have been necessary to introduce a social invention, as opposed to the technical invention.
What the coming of the computer did, “just in time,” was to make it unnecessary to create social inventions, to change the system in any way. So in that sense, the computer has acted as fundamentally a conservative force, a force which kept power or even solidified power where is already existed.
This is worth re-reading. What he’s saying here is that digital technology created the illusion of innovation and progress, while in reality protecting and even strengthening the dysfunctional status-quo.
This is a huge point, and of course leads me to wonder… what does the protected status quo look like in education? Healthcare?
Q: Did you have these concerns when you were designing the banking system?
A: Not in the slightest. It was a very technical job, it was a very hard job, there were a number of very, very difficult problems., for example, to design a machine that would handle paper checks of various sizes, some of which might have been crumpled in a person’s pockets and so on, to handle those the way punch cards are handled in a punch card machine and so on. There were many very hard technical problems. It was a whale of a lot of fun attacking those hard problems, and it never occurred to me at the time that I was cooperating in a technological venture which had certain social side effects which I might come to regret. That never occurred to me; I was totally wrapped up in my identity as a professional, and besides, it was just too much fun.
Notice how much he uses the word “fun” to describe the task of figuring out complex problems. Many of my colleagues at the USDS have echoed this sentiment. This is how smart, curious, and well-meaning people who end up building something novel miss the bigger picture and risks. Weizenbaum spent the later part of his career raising flags and critiquing the very technologies he helped create.
Q: When did it occur to you?
A: I think after spending say 10 years at MIT -- I came here in 1963. Much of that time, much of [the next] 10 years were very turbulent years politically ... Soon after I got here, President Kennedy was assassinated. There was the dream of the Great Society that President Johnson announced, and the civil rights movement, it was very hard-fought, and I of course participated, and the Vietnam War.
The knowledge of behavior of German academics during the Hitler time weighed on me very heavily. I was born in Germany, I couldn’t relax and sit by and watch the university in which I now participated behaving in the same way. I had to become engaged in social and political questions. Once that happened I started to think and write about issues of this kind, some realities became increasingly clear to me.
Writing is very much like computer programming; when you sit down to write a program chances are you have a very good idea of what it is you want to do, you have a very good idea of what algorithm you’re going to use. In a certain sense, you believe, or you act as if, you’ve already solved the problem and it’s only a question of writing down the solution. So it is when you start to write in ordinary language. It’s perfectly clear to many people, at MIT certainly, that in the act of programming you discover new ideas, and most particularly you discover that there are deep holes in your knowledge that you have to fill before you go on. That happens with writing too. So when I started to write about these things, sometimes just more or less for myself, or in letters to others, the realities I am talking about became clear to me.
He credits his understanding and beliefs on this topic to WRITING. “when I started to write about these things, sometimes just more or less for myself, or in letters to others, the realities I am talking about became clear to me.”
He describes the critical role that the PROCESS of programming and writing play in thinking and discovery: “in the act of programming you discover new ideas, and most particularly you discover that there are deep holes in your knowledge that you have to fill before you go on. That happens with writing too.”
Writing IS thinking. Creating is thinking. This is why generative AI and LLMs are so problematic for the human mind, especially in an educational setting — and even more so among young people who are developing the ability to think.
They go on to talk about how computers are essentially “a military instrument,” and Weizenbaum then lays out why technology is not neutral, and computers are not “mere tools.”
“… Other people say, and I think this is a widely used rationalization, that fundamentally the tools we work on are “mere” tools; This means that whether they get use for good or evil depends on the person who ultimately buys them and so on.
There’s nothing bad about working in computer vision, for example. Computer vision may very well some day be used to heal people who would otherwise die. Of course, it could also be used to guide missiles, cruise missiles for example, to their destination, and all that. You see, the technology itself is neutral and value-free and it just depends how one uses it. And besides -- consistent with that -- we can’t know, we scientists cannot know how it is going to be used. So therefore we have no responsibility.
Well, that is false. It is true that a computer, for example, can be used for good or evil. It is true that a helicopter can be used as a gunship and it can also be used to rescue people from a mountain pass. And if the question arises of how a specific device is going to be used, in what I call an abstract ideal society, then one might very well say one cannot know.
But we live in a concrete society, [and] with concrete social and historical circumstances and political realities in this society, it is perfectly obvious that when something like a computer is invented, then it is going to be adopted will be for military purposes. It follows from the concrete realities in which we live, it does not follow from pure logic. But we’re not living in an abstract society, we’re living in the society in which we in fact live.”
Re-read that last line: “… we’re not living in an abstract society, we’re living in the society in which we in fact live.”
This is exactly why I strongly believe that the potential of technology breaks down in its implementation. It may be true that a new technology has the potential to do all sorts of things that are good for humanity. And I actually do believe in that potential. That said, we also know that inevitably potential is undercut by business interests.
I’m now officially diving into a rabbit hole of this man’s publications and talks… but wanted to make sure you all saw this. Because none of what we are wrestling with right now is new, and all of it was foreseeable.
The question is: how can we enable ourselves and our communities to play an active role in our — and our children’s — future?
“The computer is a particularly seductive instrument because it offers immediate feedback,”
- Weizenbaum to a reporter in 1982.” Vancouver Sun via Newspapers.com