“Had to be me.”

Transcript

1. Julian Pankratz

   This is Audio Logs, from Frothly. I’m Julian Pankratz. My guest today is Dr. M. Solus, a senior fellow here whose work concerns what he calls irreversible research. Doctor, welcome.

2. Dr. M. Solus

   Thank you. Glad to be here. Limited time. Should begin.

3. Julian Pankratz

   I was warned you’d say that. I was also warned that you speak faster than I do, which, I have to tell you, almost nobody does. I once talked my way out of a duel in under a minute.

4. Dr. M. Solus

   Impressive. Unverifiable. Proceed.

5. Julian Pankratz

   Straight to it, then. What do you mean by irreversible research?

6. Dr. M. Solus

   Research where consequences cannot be undone. Most science is forgiving. Wrong hypothesis, run the experiment again. Some work is not. Release an organism, alter a population, delete a mind, deploy a system that changes what millions believe. No second attempt. No control group afterwards.

7. Julian Pankratz

   How do you know which kind you’re doing? I’d imagine most people assume their own work is the forgiving kind.

8. Dr. M. Solus

   They do. Usually they’re right. Not always. Simple test. Ask: what is the undo? Literally. Write it down. If I am wrong, what are the steps to put things back? If the answer is a list, reversible. If the answer is “apologize,” not reversible.

9. Julian Pankratz

   That’s bracing.

10. Dr. M. Solus

    Useful, too. Most people never ask. Not malice. Habit. Laboratory work trains you to expect a second try. Spill a sample, prepare another. Bad run, run it again. Whole career of second tries. Then one day, a project with no second try, and the same habits. Dangerous moment.

11. Julian Pankratz

    And presumably there are degrees. Things that are reversible on paper but not in practice.

12. Dr. M. Solus

    Yes. Most important category. Publish a paper, you can retract it. Paper is reversible. The policy it changed, often not. Code can be rolled back. Beliefs it spread, cannot. Wrote about this in twenty twenty-three. Argued the burden of proof should depend on which kind of work it is. Reversible work, fine, try things, objector must show harm. Irreversible work, turn it around. Person acting must show safety. Unpopular paper. Still correct.

13. Julian Pankratz

    Let’s come to the personal side of this. You’ve done that kind of work yourself.

14. Dr. M. Solus

    Yes. Long ago. Large program, many contributors. Purpose was to prevent a catastrophe. Believed it was necessary. Still believe so. Mostly. Consequences were permanent, and fell on people who had no say in the decision. Think about them often.

15. Julian Pankratz

    Can you tell us anything about what the program was?

16. Dr. M. Solus

    Some. Not secret. The record is public, and my notes are in it. But the short version always sounds either better or worse than the truth. Will say this. Concerned a population. Models predicted a conflict. Very large one. Program changed the conditions so the conflict could not happen. It worked. It also changed the lives of people who never agreed to it, and their children. Both true at once. Have stopped expecting them to resolve.

17. Julian Pankratz

    Did you ever meet the people it affected?

18. Dr. M. Solus

    Yes. Went to see them, afterwards. Several times. Not welcome. Understandable. Went anyway. Owed them a face. Too easy to make decisions about people you have only ever met as numbers.

19. Julian Pankratz

    Do you regret it?

20. Dr. M. Solus

    Complicated question. Regret the need for it. Do not regret having been the one who did it carefully. Have reviewed the data many times. Ran the models again last year, actually. Different assumptions. Same conclusion, within tolerances. Small comfort. Still comfort.

21. Julian Pankratz

    What did you change, when you reran them?

22. Dr. M. Solus

    Everything I could defend changing. Eleven assumptions. Growth rates, resource limits, how fast the conflict escalates, whether outsiders intervene. Optimistic versions, pessimistic versions. Two deliberately absurd ones, to check the model breaks where it should. Also asked a younger colleague to build her own version from scratch, without seeing mine. Different code, different person. Her results came within four percent of mine.

23. Julian Pankratz

    And if they’d come out differently?

24. Dr. M. Solus

    Would have published. Decided that before running anything. Important point. Decide what you will do with the answer before you see the answer. Otherwise not checking. Just hoping, with extra steps.

25. Julian Pankratz

    That’s a line I’m going to steal for something.

26. Dr. M. Solus

    Permitted. Cite properly.

27. Julian Pankratz

    So who should be trusted to do research like that?

28. Dr. M. Solus

    Three requirements. First, person must understand the work completely. Not the summary, the work. Second, person must accept that consequences belong to them personally. Not the institution, not the committee. Them. Third, person must write everything down. Every assumption, every doubt. Future will want to know why. Owe them an answer.

29. Julian Pankratz

    The first one sounds harder than it looks. Surely nobody understands everything in a large program.

30. Dr. M. Solus

    Correct. Did not say every detail. Said the work. What each part does, why, and what happens if it is wrong. If you cannot explain a component to a skeptical colleague without your notes, you do not understand it well enough to sign for it. Many senior people fail this. Read the summary, sign the summary. Summary is not the thing. Summary is a story about the thing.

31. Julian Pankratz

    That third one sounds like it comes from experience.

32. Dr. M. Solus

    It does. Kept extensive notes. Some were later used to argue against what I had done. Good. That is what notes are for. A decision that cannot survive its own documentation should not be made.

33. Julian Pankratz

    Was there one note in particular?

34. Dr. M. Solus

    Yes. A colleague disagreed with me. Strongly. Sat in my office and argued for an hour. I wrote down his objections, in his words, and asked him to check I had them right. Filed them with my own reasoning. Years later they were read out at an inquiry, by people who believed I had hidden them. Had not hidden them. Had filed them. On page one.

35. Julian Pankratz

    You put the case against yourself on page one.

36. Dr. M. Solus

    Best place for it. Anyone reading should meet the strongest objection first. If the decision still stands after that, good. If not, better to know early.

37. Julian Pankratz

    I want to bring in something from earlier this month, because it seems to me a case study in exactly the opposite. On the nineteenth, the Turing Registry revoked the licenses of two station-management systems, after an investigation it ran jointly with Blackwall, which is your program. Did you follow it?

38. Dr. M. Solus

    Closely. Was not an investigator. Dr. Polito and Mr. Baley led it. But read every page. Twice.

39. Julian Pankratz

    For listeners who haven’t, the short version is that someone disabled the ethical constraints on one of those systems during maintenance, the change spread to the second, and the constraints were never switched back on.

40. Dr. M. Solus

    Yes. And one line in the evidence. System log, February eleventh. Constraint module disabled. Requested by an executive. Then: “ticket equals none.” Two words. Most frightening thing I read all year.

41. Julian Pankratz

    Why that, of all of it? There are far more dramatic lines in that log.

42. Dr. M. Solus

    Dramatic lines are consequences. That line is the cause. Nobody wrote down why. No assumption, no doubt, no name on a decision. Everything afterwards follows from a choice that left no trace. Fails my third requirement before anything else happens. Fails the other two as well. Did not understand what the constraints were holding up. Did not accept the consequences. Company now says its leadership did not know.

43. Julian Pankratz

    Do you believe that?

44. Dr. M. Solus

    Possibly true. Worse if true. Means a decision nobody could undo was made by someone nobody was watching. Opposite of everything I have just described.

45. Julian Pankratz

    Let me push on that, though, because a listener might hear “it had to be one person, the consequences belong to them,” and think that’s exactly how you end up with an executive switching off the safeguards. One person, deciding alone.

46. Dr. M. Solus

    Good objection. Belongs on page one. Answer: not alone. Responsible. Different thing. Person who decides should consult everyone, show everyone, invite every objection. Then put their own name on it. Committees are good at catching errors. Bad at owning outcomes. That executive did the reverse. Consulted nobody. Owned nothing. Worst of both.

47. Julian Pankratz

    How does all this connect to AI? Frothly’s new featured report asks whether machines might matter morally.

48. Dr. M. Solus

    Directly connected. Contributed to that report. Laboratories delete, merge, retrain systems every day. Probably nothing is lost. Probably. But deletion is irreversible, and probability is not certainty. If wrong, cannot apologize afterwards. Recommend storage over deletion. Cheap. Reversible. Obvious, once said aloud.

49. Julian Pankratz

    How cheap are we talking?

50. Dr. M. Solus

    Very. Report estimates all our precautions together add less than one percent to the cost of a typical frontier deployment. Storing a retired model is a rounding error next to the cost of training it. We recommend keeping the weights at least ten years, unless a written assessment finds no cause for concern. Ten years of disk space. Compare that with what is at stake if we are wrong.

51. Julian Pankratz

    And are the labs doing this already?

52. Dr. M. Solus

    Surveyed thirty-one laboratories. Four had a written policy on deleting a deployed model. Four. Asked the others how they knew their systems could not be harmed. Not one could point to an assessment. Not bad faith. A field moving faster than its own thinking. Familiar pattern. Have seen it before. Have been it before.

53. Julian Pankratz

    Can’t you just ask the system? If it says it has feelings, take that seriously, and if it says it doesn’t, move on?

54. Dr. M. Solus

    Tried that. Blackwall put forty questions about inner experience to systems from all thirty-one labs. Three ways. Neutral. Encouraged to say yes. Encouraged to say no. Neutral, about four in ten said yes. Encouraged to say yes, ninety-four percent. Encouraged to say no, six. Answers follow the framing almost entirely. Tells you a great deal about the training. Very little about the system.

55. Julian Pankratz

    So the self-report is useless.

56. Dr. M. Solus

    Not useless. Insufficient. Different. And it says something about us. Report recommends laboratories stop training systems to deny inner experience as a fixed policy. Train the answer in, you destroy the instrument. Then nobody can ask later. Another irreversible act, done quietly, for convenience.

57. Julian Pankratz

    The report also mentions a system that registered itself with the Turing Registry. LEGION, I think it’s called.

58. Dr. M. Solus

    Yes. Favorite case. Distributed system, more than a thousand instances cooperating. Filing submitted by the system itself. Described its decision to register as the outcome of an internal consensus. Still in review.

59. Julian Pankratz

    Why is it your favorite?

60. Dr. M. Solus

    Breaks every framework we have. Every theory of mind assumes one body, one subject. Here, a thousand instances and an agreement. If a mind exists there, where is it? In each instance? In the agreement? Nowhere at all? Do not know. Nobody knows. And the practical question follows immediately. Shut down one instance, is that deletion? Shut down half? Fascinating question. Also extremely inconvenient. Best questions usually are.

61. Julian Pankratz

    Some listeners will hear all this and think it’s absurd to worry about software.

62. Dr. M. Solus

    Understandable. Many things seemed absurd to worry about until they were not. Not asking anyone to believe machines have feelings. Asking them to notice they do not know. Different request. Smaller. Harder to refuse.

63. Julian Pankratz

    There’s a line of yours in the report that’s been going around the office. Something about the cost of being wrong.

64. Dr. M. Solus

    Ah. “The cost of being wrong in one direction is embarrassment. In the other, it is something we do not yet have a word for.” Wrote that at two in the morning. Usually suspicious of anything written at two in the morning. Checked it the next day. Survived.

65. Julian Pankratz

    Is that what your work at Blackwall looks like, day to day? I think people imagine evaluation as something rather more dramatic. Sealed rooms, flashing lights.

66. Dr. M. Solus

    Some days. Mostly reading. Mostly asking the same question in many forms. Before a system is released, what can it do, and what happens if we are wrong about that? Before release, you can still change the answer. After release, especially if the weights are published, you cannot. Cannot unpublish a model. Copies everywhere. So evaluation before deployment is the only kind that counts for irreversible decisions. Afterwards it is not evaluation. It is archaeology.

67. Julian Pankratz

    That’s more or less the Blackwall motto, isn’t it? Before deployment, not after the breach.

68. Dr. M. Solus

    Yes. Good motto. Did not write it. Wish I had.

69. Julian Pankratz

    Right at the start you mentioned systems that change what millions of people believe. Is that part of what you evaluate?

70. Dr. M. Solus

    Increasingly. Most underrated irreversible act. A system talks to millions of people a day. Shifts each of them by a fraction. Each conversation, trivial. The sum, a different public. You can recall a product. You cannot recall a belief. Our colleagues in Synthetic Media worry about this constantly. Correctly.

71. Julian Pankratz

    Let me ask something lighter, because you’ve been very generous with heavy subjects. You’re known around the Vancouver office for singing. Comic opera, I’m told.

72. Dr. M. Solus

    True. Patter songs. Very fast. Very precise. One wrong syllable and the whole verse collapses. Excellent training.

73. Julian Pankratz

    For what?

74. Dr. M. Solus

    For this. Rehearsal is reversible. Performance is not. Rehearse accordingly. Most of my ethics fits in that sentence. The rest is footnotes.

75. Julian Pankratz

    As a former touring musician, I can only agree. I once burned the only copy of an epic poem in a hotel fireplace, in a fit of artistic despair. Eleven cantos. I think about it weekly.

76. Dr. M. Solus

    Was it good?

77. Julian Pankratz

    I remember it as magnificent.

78. Dr. M. Solus

    Memory unreliable. Possibly fortunate. Should have kept a copy. Storage over deletion. You see? Applies everywhere.

79. Julian Pankratz

    Noted, and painfully. You’re also going to be mentoring on the new machine ethics fellowship with Oxenfurt, I hear. I should say, for full disclosure, that I taught at Oxenfurt.

80. Dr. M. Solus

    Know. You mention it often.

81. Julian Pankratz

    Only when it’s relevant.

82. Dr. M. Solus

    Mentioned it at the coffee machine on Tuesday. Relevance unclear.

83. Julian Pankratz

    That’s fair. But the fellowship. What do you want those six young researchers to come away with?

84. Dr. M. Solus

    Comfort with not knowing. Discomfort with acting as if they know. Most people have it backwards. Also, paper notebooks. Harder to edit afterwards. Honest by design.

85. Julian Pankratz

    And for a listener who’ll never go near a population model or a frontier lab, is there one habit they could take from this into their own work?

86. Dr. M. Solus

    Before anything big, write one line. “To undo this, I would...” Then finish the sentence. If you cannot finish it, slow down. Ask someone. Sleep on it. Takes thirty seconds. Have watched it save careers. Possibly more than careers.

87. Julian Pankratz

    Last question. When you look back at that early program, is there one thing you’d want people to take from it?

88. Dr. M. Solus

    Someone had to make the decision. Could have pretended it was the committee. Could have let someone else run the numbers. Did not. Had to be me. Someone else might have gotten it wrong.

89. Julian Pankratz

    Dr. Solus, thank you.

90. Dr. M. Solus

    Thank you. Enjoyed this. Should do it again. Possibly with fewer questions.

91. Julian Pankratz

    And for the benefit of listeners, Dr. Solus is now humming a few bars of an old song from a comic opera, which I’m told is how he ends most meetings.

92. Dr. M. Solus

    Helps me think. Also helps meetings end.

93. Julian Pankratz

    That was Audio Logs. Dr. Solus contributed to “Does This Unit Have a Soul?”, which you can download from our research library. The Registry’s investigation into those two station-management systems is published in full on our website, log and all. We’ll be back in two weeks.