Calibrations

Transcript

1. Julian Pankratz

   This is Audio Logs. I’m Julian Pankratz, and I’m joined by Frothly’s new Chief Technology Officer, Dr. Pasqal Haneumann. I should tell listeners I tried to book you last month and you said you were in the middle of some calibrations.

2. Dr. Pasqal Haneumann

   I was. The relay network had drifted by a fraction of a percent. It would not have mattered for a year, perhaps two. But drift that does not matter yet is precisely the drift one should correct.

3. Julian Pankratz

   And are you still in the middle of them?

4. Dr. Pasqal Haneumann

   I am. They were paused for a couple of days this week, for the release of Glossia 4.0, which seemed only courteous. They have since resumed.

5. Julian Pankratz

   Courteous to whom?

6. Dr. Pasqal Haneumann

   To the release, and to whoever would have had to diagnose a fault on release day. One does not change two things at once. If anything had misbehaved this week, I wanted there to be only one suspect.

7. Julian Pankratz

   When will they be finished?

8. Dr. Pasqal Haneumann

   When they are complete.

9. Julian Pankratz

   I had a feeling you’d say that.

10. Dr. Pasqal Haneumann

    It is not evasion. An estimate given in advance has a way of becoming a deadline, and a deadline is the natural enemy of calibration. One does not hurry a measurement. One only makes it wrong faster.

11. Julian Pankratz

    Let’s back up for listeners who’ve seen the notice on the status page and wondered what on earth it means. What exactly is drifting?

12. Dr. Pasqal Haneumann

    Glossia messages travel through relays, run by us and by partners all over the world. When your phone chooses a route, it weighs each relay by how much traffic it can carry. Those figures come from measurement clients, small machines of ours that test every relay, around the clock. Over months, the measurements wander, by a fraction of a percent, away from what the relays can really do.

13. Julian Pankratz

    And a fraction of a percent matters because?

14. Dr. Pasqal Haneumann

    Today, it does not. No message arrives a moment later because of it. But leave it for a year or two, and the network begins to send slightly too much traffic to some relays and slightly too little to others. Nobody would notice the day it began. They would notice the day a busy relay tipped over during a crisis, and by then the cause would be two years old and very hard to find.

15. Julian Pankratz

    So how do you correct it? Who measures the measurers?

16. Dr. Pasqal Haneumann

    That is exactly the right question. The danger with instruments is that they drift together. If every clock in a house runs slow at the same rate, they agree perfectly, and they are all wrong. So we compare the measurement clients against each other, and against a small set of reference relays in Chiba whose behavior we know very well. Then we correct one relay at a time. Delivery is never interrupted. The relay operators need do nothing at all.

17. Julian Pankratz

    One relay at a time. How many relays are there?

18. Dr. Pasqal Haneumann

    Enough that I have stopped answering that question at dinner parties.

19. Julian Pankratz

    That’s fair. Is that your philosophy of infrastructure, then? Correct the drift before it matters?

20. Dr. Pasqal Haneumann

    It is most of it. People admire the building of machines. Very few admire their maintenance, and yet maintenance is where machines spend nearly all of their lives. A system is designed once and kept running ten thousand times.

21. Julian Pankratz

    I suppose the design gets the ribbon-cutting.

22. Dr. Pasqal Haneumann

    And the maintenance gets the night shift. In my experience, very little goes wrong while a machine is simply running. A running machine is in a known state. A machine being worked on is in a state that somebody has chosen, and people choose badly at three in the morning.

23. Julian Pankratz

    Do you have an example from here?

24. Dr. Pasqal Haneumann

    Several, from the network’s own history. In my first week I read every incident report back to 2011. In 2012 a leap second was added to the world’s clocks, and the relays stopped forwarding messages until someone restarted them by hand. On New Year’s Day 2023 a certificate expired at midnight. Nobody had done anything wrong, precisely. Something simply came due on a day when no one was watching. We now check expiry daily, the alerts begin ninety days ahead, and no certificate the network depends on is permitted to expire on a public holiday.

25. Julian Pankratz

    That last rule is very specific.

26. Dr. Pasqal Haneumann

    The best rules usually are. They are scar tissue. Each one tells you exactly where it hurt.

27. Julian Pankratz

    And just before you arrived, a partner relay was removed from the network, wasn’t it?

28. Dr. Pasqal Haneumann

    At the end of August. Its operator disabled the relay’s required safety checks during maintenance and did not restore them. Why, I do not know, and it is not my place to guess. Our conformance checks found it, the relay was withdrawn from new routes, and a week later it was removed from the directory. That is the system working. It is also a perfect illustration of where the danger lives.

29. Julian Pankratz

    Before Frothly you looked after a fleet of ships.

30. Dr. Pasqal Haneumann

    Long-haul vessels, for the von Valancius family. Some of their systems were older than the crews who operated them. The documentation had been lost generations ago. What remained was practice: the order in which things should be woken, the sounds that meant trouble, the small courtesies each system seemed to prefer.

31. Julian Pankratz

    That name will be familiar to some listeners, since our founder was a von Valancius.

32. Dr. Pasqal Haneumann

    Yes. The connection is an old one. It did not make the interview any shorter, I assure you. If anything, I was asked more questions.

33. Julian Pankratz

    I looked up one of your old ships on the Turing Registry before we sat down. The renewal notes say the system “responds poorly to neglect and well to incense.”

34. Dr. Pasqal Haneumann

    That was my wording, and I stand by it. The Registry takes no position on incense, which I think is the correct position for a registry.

35. Julian Pankratz

    Courtesies, though. You used that word a moment ago.

36. Dr. Pasqal Haneumann

    I choose the word deliberately. Our Human Factors group published a paper this month on what they call machine rituals. I was a co-author, and I will confess I recognized myself in it. When you have kept a machine alive for twenty years, you develop habits that look, from the outside, rather like devotion.

37. Julian Pankratz

    Give us a sense of the paper. What did Dr. Orsellio’s team find?

38. Dr. Pasqal Haneumann

    They interviewed fourteen hundred people in twelve countries, office workers and the engineers who look after their machines. Seventy-one percent perform at least one ritual they believe improves a device’s behavior but cannot explain. Restarting twice. Waiting before pressing a button. Speaking kindly to the printer. And expertise is no protection. Nearly two-thirds of the IT professionals had rituals of their own, usually around the oldest systems.

39. Julian Pankratz

    There’s a passage about a maintenance facility with a laminated card taped above a forty-year-old console, setting out the order in which to wake things after a power failure. It ends with the words “Thank it.”

40. Dr. Pasqal Haneumann

    There is.

41. Julian Pankratz

    Was that your facility?

42. Dr. Pasqal Haneumann

    The sites are anonymous, at the request of the people who work there. I am not at liberty to say who wrote the card.

43. Julian Pankratz

    That’s not a no.

44. Dr. Pasqal Haneumann

    It is not a no.

45. Julian Pankratz

    Do you think the rituals work?

46. Dr. Pasqal Haneumann

    Many of them do. Not for mystical reasons. A ritual is often the only surviving record of something true: that a component needs time to warm, that a certain order avoids a fault no one wrote down. I would rather preserve the ritual and later learn its reason than discard it and learn its reason the hard way.

47. Julian Pankratz

    Has that happened to you? Learning the reason the hard way?

48. Dr. Pasqal Haneumann

    Once, early. On one of the ships the navigation array was always woken in the same manner. Power, then a wait of three tones from the console, then the array itself. The crew called it letting it clear its throat. A young engineer, very capable, decided this was superstition and skipped the wait to save forty seconds. It cost us nine days. Behind that array was a bank of capacitors that needed precisely those forty seconds to charge, and nobody living knew it was there. The ritual had remembered on our behalf.

49. Julian Pankratz

    What happened to the engineer?

50. Dr. Pasqal Haneumann

    She became the best maintainer I ever trained. She also wrote the capacitors into the manual, which is exactly what one should do. The paper found that about four in ten of the rituals they could test corresponded to a real behavior that appeared in no documentation. That is a very high number for something people call superstition.

51. Julian Pankratz

    And the other six in ten?

52. Dr. Pasqal Haneumann

    Some are harmless. Some are worse than harmless, and the paper is honest about that. People who described a device as having moods were less than half as likely to report its faults. They managed the mood instead. A ritual that replaces a report is a bad ritual. The knowledge stays inside one person, and when that person leaves, so does the knowledge.

53. Julian Pankratz

    So the job is to keep the good rituals and write down why they work.

54. Dr. Pasqal Haneumann

    To keep them, to test them, and to write down what they are protecting. On the ships I spent eleven years reconstructing the maintenance manual simply by watching people work. The vessel’s license on the Registry now refers to an appendix four. I wrote appendix four. It is mostly rituals, with their reasons attached wherever I could find them.

55. Julian Pankratz

    I gather you’ve continued the tradition here. I was reading the history of one of the Chiba relays, the one that failed during a restart last month—

56. Dr. Pasqal Haneumann

    NRT zero three.

57. Julian Pankratz

    —and it says the thermal compound was reapplied “by the office of the Chief Technology Officer, which prefers the term unguent.”

58. Dr. Pasqal Haneumann

    It is a better word. A compound is something one applies by the gram. An unguent is something one applies with care. The material is the same. The attention is not. I would also like listeners to know that the calibrations had nothing to do with that failure. A storage controller did not wake after a routine restart. We replaced it two days later, and we have since replaced the same model in all four Chiba relays. Restarts there are now staggered, and each relay hands its mailboxes to a neighbor before it restarts, so a relay that does not come back no longer leaves anyone waiting.

59. Julian Pankratz

    Noted, for the record. What will you change at Frothly?

60. Dr. Pasqal Haneumann

    Very little that anyone will notice, if I do it well. More monitoring. Fewer single points of failure. Every system will have written procedures, and every procedure will have a person who has actually performed it. And we will thank the systems that serve us well. I am told this is unusual. I find it simply correct.

61. Julian Pankratz

    A person who’s actually performed it. That sounds obvious, but I imagine it isn’t.

62. Dr. Pasqal Haneumann

    It is not. Most organizations have a procedure for restoring from backup that nobody has ever followed. A backup that has never been restored is a hope, not a backup. So we will rehearse. We will restore the Lazarus search index onto spare hardware. We will fail over a Registry database on an ordinary Tuesday afternoon, deliberately, while everyone is awake and well caffeinated, so that it is not a surprise on some unusual Sunday night.

63. Julian Pankratz

    And single points of failure. I’d assumed that meant hardware.

64. Dr. Pasqal Haneumann

    Hardware is the easy kind. The dangerous single point of failure is a person who knows something nobody else knows. I was one, on the ships, for some years. It is flattering, and it is a failure of duty. When I left the fleet I left behind eleven binders, and three people who had performed every procedure in them.

65. Julian Pankratz

    Let me come back to the thanking. Why thank a machine? I suspect a few listeners are rolling their eyes.

66. Dr. Pasqal Haneumann

    Let them. The thanks is not for the machine’s benefit, or not only. It is for ours. A person who thanks a machine has looked at it. They have noticed that it did its work. And a person who notices when a machine does its work is far more likely to notice when it begins, very slightly, to do it differently. Gratitude is a form of attention. Attention is most of maintenance.

67. Julian Pankratz

    As a musician, I’m in a position to agree. I’ve restrung the same lute on four continents, and I talk to it constantly.

68. Dr. Pasqal Haneumann

    Does it play better?

69. Julian Pankratz

    It certainly sounds better after a compliment.

70. Dr. Pasqal Haneumann

    Then you have been performing a ritual with a real effect. Most likely you tune it more carefully when you are being polite to it. The paper would count you among the four in ten.

71. Julian Pankratz

    I’ll take that as a scientific endorsement. You’ve said before that you trust machines more than people.

72. Dr. Pasqal Haneumann

    I said that machines are more consistent. The flesh is weak; it forgets, it tires, it skips a step at the end of a long night. A machine does exactly what it was built to do. That is its virtue and its danger. Our job is to make sure what it was built to do is what we actually want.

73. Julian Pankratz

    That seems to reach well beyond relays. The institute spends a great deal of its time on AI systems.

74. Dr. Pasqal Haneumann

    It applies to them more than to anything. A relay is built to forward messages, and either it does or it does not. The systems our colleagues in Blackwall evaluate are built to pursue objectives that someone wrote down, and the gap between what was written and what was meant is where all the trouble lives. They drift, too, in their way. Not always the system itself, but the world around it. A model assessed in the spring is working in the autumn’s world.

75. Julian Pankratz

    So the evaluations need calibrating as well.

76. Dr. Pasqal Haneumann

    Of course. Any instrument does. A test that every system now passes has stopped measuring anything. And if a test’s questions have leaked into the material the systems learned from, it is measuring memory, not ability. So one compares against references, one checks the instruments against each other, and one does it again, long before it seems necessary. It is the same discipline. It is simply less familiar.

77. Julian Pankratz

    Last question, then. If a listener runs a system of any kind, a server, a small office network, a car, what’s the one habit you’d want them to take from this?

78. Dr. Pasqal Haneumann

    Write down the strange thing you do. The second restart, the pause, the order you always follow. Write it down, and beside it write why you believe it works. If you do not know why, write that instead. The next person will thank you. Possibly, so will the machine.

79. Julian Pankratz

    Dr. Haneumann, thank you.

80. Dr. Pasqal Haneumann

    Thank you. And thank you to the recording equipment, which has performed admirably.

81. Julian Pankratz

    I’ll pass that along. That was Audio Logs. Dr. Haneumann’s calibrations are still in progress, and you can follow them on the relay site, where there’ll be an update when they’re complete and not, I’m reliably informed, a moment before. Machine Spirits, the Human Factors paper we talked about, is in our research library. We’ll be back in two weeks.