A Responsibility Not Upheld
Opening Statement
June 17, 2021
Victoria Caldwell is in the middle of her opening statement for the Foreign Intelligence Surveillance Court of Review: “General Grant Bridgewater is on trial today for gross negligence. He had strategic command of the deployment of our mind control technology. With that kind of control of peoples’ lives comes a responsibility. A responsibility that the general has not upheld.”
Victoria Caldwell was talking about a brain-computer interface device that could edit and record any function of the human brain. It could make people do anything the operator wanted.
The Real Leader of This Gang
In Mexico one evening, the gang El Corredor de Los Perdidos were celebrating their latest score. They had ransomed yet another American tourist and this one for a record sum.
In an air-conditioned office somewhere in the United States, Sam was in a festive mood, too. A stop would be put to the party and he would be the one to do it. Him and his dear friend Armando.
Ignacio, the gang’s leader, is laughing and drinking with Armando, his best buddy and lieutenant. Out of nowhere, after telling a particularly funny joke, Armando throws his beer in Ignacio’s face and yells “that will improve your foul stink!” Ignacio instinctively buries his fist in Armando’s face. Armando kicks Ignacio between the legs and bellows for all to hear, “You are a castrated donkey. I’m the real leader of this gang!”
All eyes are on the pair. Armando knows he’s crossed a line that should not be crossed. “Get him! Now!” croaks Ignacio. Armando signals his people. Guns are drawn. Everyone is confused. Ignacio and Armando did not get to where they are by being stupid. They could have defused the situation with a brawl that leaves Armando battered senseless. Sam doesn’t give them the chance. Armando fires the first shot. Bullets loose and El Corredor de Los Perdidos is no more.
Accountable to this Court
June 18, 2021
It is the day of Victoria Caldwell’s questioning of General Grant Bridgewater.
“Who was El Corredor de Los Perdidos?” Victoria asked.
“It was a Mexican gang that kidnapped American tourists,” Grant answered.
“What happened to them?”
“We made them shoot each other.”
“Who did?”
“Sam Thorne had command of that operation.”
“And how did you get them to shoot each other?”
“We had a client in the gang and Sam had him attack the leader.”
“What is a ‘client’?”
“A client is someone that has our mind control device in their brain.”
“So you convinced the client to attack the leader of the gang?”
“No. He didn’t know what was going on. We made him do it with the device.”
“Do your clients have free will, general?”
“Not while we run an operation against them, but otherwise, yes.”
“Your client’s name for this operation was Armando Castro, was it not?”
“Yes.”
“Did Armando Castro receive a trial before you killed him?”
“Not a trial like this one. I personally approved the operation. I am authorized to make life and death decisions in the service of this great nation.”
“Operational life and death decisions are documented and subject to review, are they not?”
“Yes, but not this one.”
“Why not?”
“I hold the authority to make and review these decisions and for the purposes of this operation I delegated that authority to Sam Thorne.”
“So you hold unaccountable power?”
“I am accountable to this court—here I am in this chair.”
“So you thought Armando Castro deserved to die and that’s why you could run an operation against him?”
“He did deserve to die, but that’s not why we killed him. The reason I approved the operation was to stop him from kidnapping American citizens. He had done it many times.”
“Could you describe for the court what a ‘boffin mission’ is, general?”
“That is Sam Thorne’s name for an operation involving an intelligent and meritorious client.”
“What does ‘meritorious’ mean?”
“It is a person that we believe is doing good in the world.”
“So it’s someone who did nothing wrong?”
“Yes, or the good they did outweighs the bad.”
“So it was acceptable to run an operation on Armando Castro because he was not meritorious?”
“Yes. If he had been meritorious, I would not have approved the operation.”
“What about Benjamin Collins?”
For the first time, General Grant Bridgewater hesitated.
Ba-kaw!
Sam is having a day. He’s going to have some fun with Ben, his favorite boffin. Sam logs into Ben’s mind. Ben is standing, waiting at a red light. Sam takes control and makes Ben cluck loudly like a chicken while walking around in a circle. The crowd is surprised and amused at Ben’s behavior. Sam tells Ben’s brain that this was Ben’s own idea. It’s a funny joke that Ben thought of himself. Ben believes it because there is nothing else he can do. Then the light turns green. Ben continues with his day across the street. Sam laughs. He is no longer having a day.
I Was Too Trusting
June 18, 2021
“Who is Benjamin Collins?” Victoria continued.
“He is a legal consultant from Cincinatti. A very good one, I hear.” Grant answered.
“Is Benjamin Collins meritorious, general?”
“Yes.”
“Was an operation run with Benjamin Collins as the target?”
“Yes, but not one that I approved.”
“Who approved it?”
“No one did. Sam Thorne took it upon himself.”
“What happened to Benjamin Collins, general?”
“We have all read his account. We know what happened.”
“Have you looked at the logs for the operation?”
“We don’t keep logs.”
“Why is that?”
“Everything we do is sensitive and the logs, if they fell into the wrong hands, could reveal state secrets.”
“So after the operation on Benjamin Collins began, someone reviewed the operation?”
“No. No one knew what Sam Thorne was doing.”
“So some operations are not reviewed?”
“Not if we don’t know about it, that is true.”
“How was Sam Thorne able to log into Benjamin Collins’ mind?”
“Because he had the access key for Benjamin Collins’ device.”
“So it’s possible to log into a client without anyone else being present to monitor it?”
“Yes, if you have the key.”
“How did Sam Thorne get the key for Benjamin Collins’ device?”
“He got it because I approved it.”
“Why did you do that?”
“He asked for someone who spoke Dutch to look into a theory about how language functions in the brain. It was something that was supposed to take half an hour to investigate.”
“So Sam Thorne was only able to access Benjamin Collins’ mind for half an hour?”
“No, he had the key, so he had access indefinitely.”
“When did Sam Thorne gain access to Benjamin Collins’ device, general?”
“That was 35 years ago.”
“When did Sam Thorne begin an operation with Benjamin Collins as the target?”
“As soon as he got access, so 35 years ago.”
“When did Sam Thorne stop that operation?”
“When we found out what he was doing last week.”
Victoria Caldwell waited a moment before continuing.
“General, one of the principles in IT security is that if someone is given access to something sensitive, that access should have a finite duration that is no longer than required. What is your opinion on this principle?”
“That is an excellent principle.”
“Was this principle followed under your command?”
“No.”
“Why not?”
“I was too trusting. I didn’t think clients needed security from my own people.”
Victoria Caldwell left another pause.
“How old was Benjamin Collins when he became a client?”
“When he was three years old.”
“Why did you target a three year old’s brain with your mind control device, general?”
“Benjamin Collins has an extremely high IQ. We have ways of identifying gifted children at a young age and we collect data from them using the device.”
“Why do you do that?”
“We are attempting to train an AI to be as clever as they are. We need access to their thoughts from a young age and as they grow up to train the AI.”
“So Benjamin Collins’ parents agreed to this?”
“No, they didn’t know about it. It’s a state secret, they don’t have a security clearance.”
“Do you think Benjamin Collins’ parents would have agreed to this if they knew?”
“I don’t know if they would.”
“Who approved this operation?”
“I did.”
“Why did you think that this was an acceptable thing to do, general?”
“The reasoning was that there is no harm to a client like Benjamin Collins from having the device, while there is a national security benefit from training the AI.”
“Do you think Benjamin Collins would agree that there is no harm to the client, general?”
“No, I do not think that.”
Or Should Have Known
June 18, 2021
“Do operations that your team conduct usually involve torturing your clients, general?” Victoria continues.
“No. I have never approved an operation that involved torture. That’s not who we are.” Grant answers.
“Sam Thorne was able to torture Benjamin Collins, was he not?”
“Yes.”
“Did he invent special software for the purpose of torture?”
“No, he used the software that we use for all operations.”
“Why does your software have the capability to torture clients, general?”
“Our software analyses the brain and finds all its functions by itself. The brain has many functions that are available but that we do not use, such as pain.”
“Why doesn’t your software block those functions that you do not use, general?”
“I trusted my people not to do things they shouldn’t do.”
“I see, but surely, if someone on your staff does something egregious, or maybe inflicts high levels of pain for an extended time, the software will alert you?”
“No. I trusted my people.”
“Has it always been this way?”
“Yes.”
“So let me see if I got this right. Once someone on your staff has the key for a client, which is not hard to get, your software allows them to do anything they want to do to them for however long they want to do it, your software allows torture, no person monitors your operations, no software monitors your operations, no logs are kept, an operation can be conducted without your knowledge and it has always been that way. Is that right, general?”
“Yes.”
“Benjamin Collins was left to the mercies of a sadist for 35 years, general.”
“What justification would you offer to the court for this state of affairs?”
“There is no justification. That’s why I belong in this chair.”
Victoria Caldwell paused for a moment before bringing up a new topic.
“General, is it not true that you have one of these devices in your own head?”
“Yes.”
“Can I have your key?”
“There is no key for my device. It would be a security risk.”
“Who has access to your device, general?”
“No one does except me.”
“Is it not true, general, that you have to personally approve it every time a single bit of information goes in or out of your own device?”
“Yes. For security.”
“For your security. Not for Benjamin Collins’ security. Isn’t that right?”
“Yes.”
Mistakes Like This Are Not Supposed to Happen
It is 8:50 a.m. on November 9, 1979, in the North American Aerospace Defense Command at Peterson Air Force Base. A klaxon sounds and all heads turn to the the display on the wall. The Soviet Union has fired 250 nuclear missiles on the United States. Everyone in the room knows the timeline. The decision to retaliate must be made within three to seven minutes. There is frantic activity. ICBM sites across the country prepare to launch and nuclear bombers are prepared for takeoff. The display now shows 2,200 Soviet missiles en route. William Odom calls national security advisor Zbigniew Brzezinski who contacts president Jimmy Carter. At five minutes, it is already late in the timeline. The order is to retaliate.
William Odom does not hear that order. What he hears is to confirm with PAVE PAWS early warning radar. At 6 minutes, PAVE PAWS confirms that there are no missiles in the sky. The ordeal is revealed to be a technical malfunction.
In a control room elsewhere in the United States, there is disbelief. A trust in the United States government has been eroded. Mistakes like this are not supposed to happen. A client like William Odom was never supposed to be activated.
Judgement
June 20, 2021
The court finds General Grant Bridgewater guilty of gross negligence. The policies personally approved by General Grant Bridgewater shock the conscience. New policies will be enacted under new leadership to be appointed at a later date. Until then, provisional leadership will oversee the introduction of basic IT security practices. The general’s exemplary service outside of this matter is duly noted. He will remain with the project in an advisory capacity only.
I Prosecute, You Judge
A week earlier, Grant walks into Victoria’s office.
“I have a case for you.”
“OK, who is it against?”
“Me.”
“I prosecute, you judge, Grant. I can’t prosecute you. What is this about?”
“Sam did wrong.”
“So I’ll prosecute him.”
“That’s not good enough. I approved the policies that allowed this to happen. Someone other than myself needs to judge that. I have to be accountable to someone in a situation like this. People need to see that.”
“So this is serious, then?”
“Yes.”
“If I take this up, I won’t hold back. I’ll prosecute you for real.”
“That’s why you’re the one I’m talking to.”
Author’s note The general is fictional, but you aren’t. You who are someone involved with BCIs. When your trial comes, if it comes, let us hope that you will not have to say that you couldn’t ensure IT security or that you trusted your people too much. More than that, let us hope that we will never need a trial like this.
The general is the kind of person that you can punish for getting something wrong, but his motivation before and after will be the same because his motivation was always to do what’s right. You could punish those people harshly, but it won’t do much good. The courts have always rewarded good motivations, good character, turning yourself in and taking responsibility for your actions with lighter sentences, sometimes dramatically so. So it is with the general. The general was not the right man for his project - he is a very good thing but taken too flamboyantly far. Projects involving nuclear weapons, medical trials or BCIs are for the effective grey - personally courageous, intelligent, competent and highly conscientious people who will do the right thing. The general is all of those things, but he isn’t grey enough. He undertook his own trial when he didn’t have to. That is no small thing. If your leader isn’t like that, he can’t be a leader of a BCI project. If you are a country, a company or anyone undertaking a BCI project, you’ll have to find people like that. If you can’t, don’t proceed, do something else or, perhaps, be forever damned for what you have set in motion. Advanced BCIs are a very dangerous technology and the people who will first have access to them, which will be the research team and their leadership, or perhaps their minders from the police, military and/or intelligence community, are people everyone everywhere will have to be able to trust.