The Blue Screen of Felony Convictions

by Curtis Poe

Recently on this blog, I wrote You Have the Right to Read Your Accuser. In this, I argued that any software with substantial risk to harm your life or liberty must be open source. I specifically mentioned some breathalyzer software that people were fighting to see the source code of. Though the Florida legislature and many regional prosecutors obviously feel that the breathalyzer company's rights were more important the individual rights, the New Jersey Supreme Court ordered that source code to a popular breathalyzer be revealed. The results, if you'll pardon the pun, are breathtaking.


29 Comments

TDDPirate
2007-09-06 14:19:34
Software-driven instruments used for medical diagnosis are subjected to stringent quality requirements and strict regulations (see, for example, U.S. Food and Drug Administration Premarket Notification 510(k)) before the manufacturer is allowed to sell the instrument.


What, if at all, legal framework exists for regulating instruments used for law enforcement?

Frank Booth
2007-09-06 14:34:16
>
> I’m sure most programmers have horror stories of their bosses
> telling them to “just shipped the damn thing, we can fix it
> later.”


I've never really run into a developer who wanted to ship code with bugs. But the reality is that if your product is not available, people will not give you their money for it. They will give their money to somebody else who is willing to ship them a disk with some bits and bytes on it.


After contracts are signed, and money changes hands, it takes a lawyer to get your money back, even if the SW won't run. Often, the path through litigation is so unpalatable as to make suffering along until the next upgrade seem like the only viable course of action.


Your sales, marketing, and finance departments knows this. That's why they have Product Managers, and that's why you "just ship it."


Business will always want it Soon, and will opt for Cheap over Good every time.

chromatic
2007-09-06 14:37:11
Business will always want it Soon, and will opt for Cheap over Good every time.


Businesses still face liability for their products, right?

Ovid
2007-09-06 18:35:22
@Frank Booth: I've been developing software for many years and my background is actually economics (didn't finish the degree, though). I'm quite aware of why software ships prematurely and yes, I have shipped software prematurely (hey, I have to pay my rent). However, the impact on the business and the impact on the consumer tend to be separate issues. I personally feel that the consumer is more important than the business. When working for a company, I'll ignore a lot of problems in software that I ship, but I won't knowingly do anything illegal or unethical (the latter area being a bit grey).


@TDDPirate: the voting machines banned in California are also subject to stringent quality requirements and strict regulations. And my parent post to this was includes links to horror stories where poor software in medical instruments has killed people. Requirements and regulations aren't enough.

Robert
2007-09-07 06:32:55
How are you defining "open source" here?
Ovid
2007-09-07 07:04:42
@Robert: by "open source" I mean "you're allowed to read the source code". I'm not confusing this with free software. I don't have a problem with companies selling their software or having a restrictive license, but if that software threatens my liberty or safety, I should at least have the right to make sure it's fair.
Robert
2007-09-07 07:12:43
@Myself


He defines it as "read only". I agree 100% with that.

Robert
2007-09-07 07:17:11
Sorry about that post to @Myself. I read your use.perl posting and this site is refreshing really weird. I think I need to clear my cache. : )
Jim Keenan
2007-09-07 07:44:05
An excellent article!
Toby
2007-09-07 10:57:37
You missed George Akerlof in the reference to The Market for Lemons paper. The phenomenon is actually named after him ("Akerlof market") so you might want to correct this.
x1101
2007-09-09 08:54:19
I have to agree. Our rights as individuals, the right to a fair trial, the right to face our accuser, and probably more that I can't think of right now are ALWAYS more important than the rights of a company to its profits. What if the next step is jury software, wouldn't you like to know how that works before going to court!!
Clifton Hyatt
2007-09-09 17:29:56
Amen!
Miguel
2007-09-09 17:57:43
"Read Only" open source has a small catch: how do you know that the source you are reading corresponds to the binary that is run? So - we need build instructions and compile rights too. And a toolchain.


But then, how can we trust the toolchain? What if they employ someone like Ken Thompson? Well then, require an open source toolchain too. OS (if there's one)? Libraries? Bios? Microcode? Hardware? At which point do we trust that the thing is kosher?


Please note that I am not disagreeing with the open source requirement. Just saying that it is not enough, and that I do not really know what would be enough.

Jack Carroll
2007-09-09 19:14:52
The FDA's design verification and validation rules aren't all that
tight, nowhere near what the FAA requires for safety-critical
hardware and software. Read up on DO-178 and DO-254.


Regarding suitable rules for ballot-counting gear and other really critical stuff, here's the proposal I submitted to the New Hampshire House e-voting subcommittee:


Suggested provisions for incorporation into a bill to prescribe objective
criteria for approval of ballot counting machines


John A. Carroll 8/27/2007


No device for counting ballots shall be approved for reporting the official
results of public elections in New Hampshire, unless:


It has been proven with absolute certainty that all logic embodied in the
device, whether in the form of election configuration data, executable
software, microcode, hardwired electronic logic, logic performed by internal
elements of complex electronic components, or in any other form, is free of
any fault that could cause an incorrect total to be reported;


It has been proven with absolute certainty that if any component fails or
malfunctions in a manner that is physically possible for that component, the
device will not report an incorrect total without also giving a clear and
unmistakeable indication of failure, and that if any component fails without
giving such an indication of failure, that failure in combination with any
other component failure will not cause the device to report an incorrect
total without also giving a clear and unmistakeable indication of failure,
failure mode effects analysis not being required to be carried beyond two
simultaneous component failures;


No logic is contained in any component which is physically capable of being
erased or rewritten, or otherwise altered without visible evidence of
tampering;


No information except dynamic data from reading and counting the ballots,
internal machine states, and other information which necessarily must change
during operation of the device is contained in any component capable of
being erased or rewritten;


No component which contains election configuration data contains any other
form of logic or information;


The device delivers its totals marked indelibly on paper that is
sufficiently durable to remain legible during the handling normally
encountered by polling place returns and during storage for the length of
time ballots are required to be retained;


The device is incapable of being influenced in its operation by any signals
or sources of information other than the ballots and the device's operator
controls, and cannot be modified to be susceptible to such signals or
sources of information without visible evidence of tampering;


The device is not susceptible to being influenced by external electrical or
physical interference or disturbance in a way which would cause an incorrect
total to be reported, without also giving a clear and unmistakeable
indication of failure, damage or destruction of components being an
acceptable effect of such external disturbance;


The device is not capable of emitting information that could identify any
ballot with the voter who cast it, and cannot be modified to emit such
information without visible evidence of tampering;


All the technical documentation necessary to manufacture, maintain,
configure, operate, understand, and prove the correctness and failure-safety
of design of the device and all its components and logic, including the
election configuration data, is in the public record, and continuously
accessible to the public;


The election configuration data within the device is expressed in a form
which can be readily understood and proven correct by persons of ordinary
education without special training;


The design and construction of the device allows practical and affordable
means to exist by which polling place officials, candidates, political
parties, and their assistants can determine at the time and place of use
that the device is in compliance with its approved technical documentation
and election configuration;


The process for approving the device and its technical documentation
provides, at a minimum, sufficient time for independent peer analysis,
comment, and testimony between the publication of the documentation and
public notice of proposed approval, and the granting of approval.

K M Ashraf
2007-09-09 19:32:55
"Liberty-threatening software must be open source" should actually read as Liberty-threatening software must be "free" software.



Jean Magnan de Bornier
2007-09-09 22:36:10
Not only "you missed George Akerlof in the reference to The Market for Lemons paper. The phenomenon is actually named after him ("Akerlof market") so you might want to correct this." But Akerlof is the only author of the referenced paper, co-winner of the 2001 Nobel Prize with Spence and Stiglitz, who actually were awarded for developing Akerlof's intuitions.
grouch
2007-09-10 00:43:38
Excellent article. It should not have been necessary to go through a court to determine that the right of persons to confront their accuser is superior to the right of the accuser to use secret data or processes to accuse.


Regarding electronic voting, there is no way to guarantee a free and fair election using electronic devices. Electronic voting machines conceal the vote from the voter and the voting process from the electorate. Paper voting conceals the individual vote from other voters but keeps the voting process public.


When you push a button, touch an icon, flip a lever or perform any other action upon a machine, you cannot view what takes place within that machine. Neither you, your friends, enemies, neighbors, chosen candidate nor opposition candidate can vouch for the action or its results. The only thing you can vouch for is that you interacted with a machine. The machine could have been functioning only enough to convince you and an election official that it is error free. Maybe it actually added 1 to some other number somewhere. Maybe it subtracted 10. Maybe it printed Aunt Tilly's vote cast 9 hours earlier. For all you know, you just opened a garage door somewhere. Your fellow "voters" can only swear they saw you close a curtain. The election officials can only swear they saw you close a curtain and something about the machine changed. Absolutely nobody can swear that what took place in secret within the machine has any bearing on the voting intended by the voters.


A paper ballot is a secret ballot. You know what vote is on your particular piece of paper. Your fellow voters can see you place that paper into a box which they can then guard to their satisfaction. Election officials can see that you place one and only one ballot within the box and can see if the box remains tamper free.


Elections should be handled by persons. Election by machine proxy is antithetical to democracy.

giafly
2007-09-10 02:49:49
>> Frank Booth
Re: I've never really run into a developer who wanted to ship code with bugs.


Nonsense. All developers want to ship code with bugs, just not "important" bugs. Suppose they spot a typo in an text message (for example a comma before the word "and"), should they rebuild-and-retest just to fix it? Any good developer would keep a record of trivial bugs to fix as a bunch in the next release.

Markus
2007-09-10 04:44:45
I'ts about time that DUI convictions based only on a breathalyzer test get thrown out. Only blood-alcohol testing should be the basis for a DUI, everything else is just too unreliable.


Markus

Martin
2007-09-10 06:17:51
You're nearly right. I think you misunderstand open source. I believe that the source code should be viewable by the public. Open source is very different to "viewable by the public". For a company to open source their code should be the choice of the company. They can still make their code viewable by the public with copyright notices that are relevant to the type of license they choose. There is a large difference, why should they be forced to let others use and copy/modify the code? No one should be forced to open source - under many conditions it is a very good business decision under some it is not. In fact I think you would be violating many rights of the company you refer to if you force them to open source. If however they can make their source code available for public viewing (say in PDF?) under stringent copyright notices (or as stringent as they wished it to be) then that would be very different. Then everyone wins, the copyrights of the business are respected and the freedoms and liberties of the individual are respected. Also if this law applies to all companies in the same market segment/situation it will also allay the biggest fear of the company in question: code theft. If all competing breathalyzer companies have to make their code available for viewing then code theft would be very easy to spot.
...
@Robert: by "open source" I mean "you're allowed to read the source code". I'm not confusing this with free software. I don't have a problem with companies selling their software or having a restrictive license, but if that software threatens my liberty or safety, I should at least have the right to make sure it's fair.
Ovid | September 7, 2007 07:04 AM
...
In my missive I gave you the benefit of the doubt, reckoning you didn't think it through or something. But having read your post above I realised you don't actually know what open source is.
You said:


by "open source" I mean "you're allowed to read the source code".


If thats your definition then you should invent your own name for it because open source is already taken and it means something very different to your definition. Either that or do some reading and find out what open source really means.


Apart from that I definitely agree in spirit with what you're trying to do: stop people's lives being effected by software where they have no recourse to show that the product is in error.

Miguel
2007-09-10 07:08:27
Martin: "viewable by the public" would be fine if there were a way to verify that what you are viewing is really what is running in the machine. Is this PDF the source of that binary?


Maybe not for breathalyzers, but in the context of voting machines there could be an incentive to have two separate versions of the code: white for the public, greyish-to-black in the machine.

Ovid
2007-09-10 08:56:11
@Martin: you are entirely correct that my use of the term "open source" differs from the norm. I certainly don't feel that companies should be forced to make their code freely modifiable by others if they do not wish to. I do feel that liberty-threatening software must have the source freely available, though. In order to get this idea across, it helps to make it easy to say (a slogan, if you will). In this case, saying "liberty-threatening software must be open source" loses too much meaning. I'll have to think about that.
Lewis Archer
2007-09-10 10:40:30
That is a very interesting case your proposing. I think it should not be allowed as evidence as the lie detector is in most states. Maybe if they fix the broken code and do genuine 'unblind' testing, ie: real sober and real drunk people with measurable ingestion of alcohol. If the thing fails the tests, send it back to the lab. Do not allow it on the streets till it works.
As for overturning the cases that have already gone through the courts, it should be evaluated on a case by case situation, ie: die someone die, or was someone mutilated horribly. If not, just a pulled over for suspicion thing, sure overturn it and let them learn a lesson. Refunding their money in the fines should also be allowed instead of suites involving the cities or states, UNLESS is caused the person to loose a job or home or family due to the possibly invalid reading, in that case, sue the hell out of the manufacturer the city and or state involved.
anonymous
2007-09-10 11:44:20
Access to details of any process or procedure used to generate evidence must be available to a defendant. This includes software but this article undermines this case by it's hysterical and unbalanced analysis.


None of the issues identified by the author are necessarily of themselves evidence of poor design or implementation. Some of them may be evidence of good design such as decisions on measurement error handling.


A question like 'do voltage spikes matter?' would normally be addressed in the the risk analysis for a product, at design reviews and in EMC testing. The fact that this question and all the other similar questions are not answered by the source code is not suprising.


Only one genuine software issue is raised that the illegal instruction excetion is not handled but this is not of itself a problem and arguably not even poor practice if other measures such as check pointing and a watchdog are implemented.


The rest of the 'software issues' are design decisions that are imposisble to evaluate except in the context of the entire system (not just the sw) and may well be sensible.


The author is either inexperienced in the development of similar products or willing to sacrifice integrity and precision of analysis for hyperbole and exageration.


I have no connection to this device but I do not want flaming so will remain anonymous.

raf
2007-09-10 13:40:51
Blah to the "open source" means something else. The open source definition may be the custody of OSI, but I doubt they'd be able to uphold a trademark in court.
chromatic
2007-09-10 14:02:23

@anonymous,


Access to details of any process or procedure used to generate evidence must be available to a defendant...


Habeas corpus is important, but disclosure of the text of and interpretation of the body of law before being charged with a crime seems also important for a free society.

Carl Witty
2007-09-10 14:35:47
How about calling it "disclosed source"? (I got that phrase from people discussing source-available voting machines.)
Ovid
2007-09-11 08:26:32
@Carl Witty: "disclosed source" sounds like an excellent term. Thank you.


@Anonymous: I don't trust people who aren't willing to stand behind their words. You insulted me by calling my argument hysterical and hinted that I lack integrity. Cowardly lobbing stones and running away does nothing to bolster your point. Maybe I'm wrong, but I at least have the decency to stand up and be counted. I will say, however, that some of your points are valid and I think people would do well to consider them.

Fred Trotter
2007-09-14 08:18:45
In your last article you stated:
This is why I firmly believe that any software with substantial risk to harm your life or liberty must be open source.


Now you are talking only about liberty. This is a mistake. Your concerns over open source vs. disclosed source become nonsensical when you begin to consider software that impacts your life. All of the issues about white-box vs. black-box that you discuss apply to medical software. However, unless the software meets the definition of "free as in freedom" software, it is not possible to run, long term, without vendor intervention. As a result the freedom to change and use the program is a critical for medical software which clearly has the "risk to harm life". Your debate about whether companies should be able to "make money" are working with your current example as the assumed case. But what if is not possible to stop using the faulty black-box software?


Please consider researching AcerMed.


-FT