I agree imposing any quota (80%, 100%, etc.) on open source adoption is bad. However it turns out the Brazilian government is not doing that. It is merely conducting a three-year pilot project of having one of its ministries use Linux instead of Windows. Linux Today was actually linking to an article on PCLinuxOnline [6/7/03]. Scroll down to the last comment and you will see the translator of the original article explaining that the LinuxToday article (and resultant slashdot discussion [6/14/03]) were wrong to be talking about an open source mandate. The PCLinuxOnline article also has a couple of other interesting comments posted by people who work in the Brazilian public service.
The reasons given for the Linux trial are cost, security, and a desire to "foster the production of local software and 'democratize access to knowledge'". The translator summarizes it as "Making a choice about what software makes it easier for Brazil's government to respect the constitutional rights of its citizens to privacy and transparency of data seems like a perfectly legitimate choice to me. " Indeed!
The goal is mainly to level the playing field between proprietary and open source software: "discrimination and prejudice will be avoided in software procurement procedures, making choices based on merit, thus giving OSS and proprietary software (PS) equal opportunities to be selected." And, "Government will implement OSS where analysis shows it to be the appropriate option. The primary criteria for selecting software solutions will remain the improvement of efficiency, effectiveness and economy of service delivery by Government to its citizens."
However it does go on to say that "as OSS offers significant indirect advantages, opting for OSS will be preferable where the direct advantages and disadvantages of OSS and PS are equally strong, and where circumstances in a particular situation do not render such preference inappropriate." So it's not an OSS quota bill, but an OSS affirmative action bill. It's all quite reasonable and if you substitute "open data formats" for OSS it is a good example of how to phrase such things to indicate your intent without forcing a policy that is too inflexible.
The government is also getting behind OSS for its own work: "Full implementation of the OSS model imples that we not only acquire and use the freely available software, but also contribute to development," so therefore the proposal states, "Where no inhibiting factors exist, the OSS model will be adopted for development of Government systems and such systems will be developed to run on OSS platforms."
The benefits listed for open source are the typical ones, most of which are debatable: cost, security, empowerment of local software companies. The list does, however, include "Access to Government data without barriers of PS and data formats."
The proposal does not talk much about open standards specifically. It does state in the Executive Overview on p.3, "Open standards will be a prerequisite for all software development, thus contributing to the ease with which OSS can be implemented and adapted." Since there is no alternative given, and the rest of the proposal is very realistic about the possibility of complete OSS adoption, I assume this was meant to apply only to software that the government develops itself. Although later, on p. 27, it says, "The guiding principle of using open standards will apply for all Government software procurement, development and maintenance." It's not defined how specifically the principle will be applied to software procurement.
The definition of open standards, on p. 30 of the proposal, is quite strict, including:
This is taken from (and credited to) Bruce Perens' definition, which also expands on each principle.
The glossary and historical appendix are well researched and are worth reading.
NewsForge ran an article by Tony Stanco [6/18/03] from the Center of Government & Open Source, supporting the South African government's proposed strategy. Stanco suggests charging companies a 5%-10% "non-compliance fee" if they don't use open standards and open protocols. It's an interesting idea and it's good to focus on open standards instead of open source, but it seems out of kilter with the cautious tone of the South African proposal (which makes no mention of a non-compliance fee).
The South African proposal, and Stanco's analysis, were discussed on slashdot [6/18/03].
Ogg Vorbis: This is a new format that was defined to replace MP3 and other audio formats that have licensing issues (technically Ogg is a container format and Vorbis is a specific compression scheme). If you read the Ogg Vorbis FAQ, under "Licensing", it is made quite clear that "the Ogg Vorbis specification is in the public domain" and "there are no licensing fees for any use of the Ogg Vorbis specification."
PDF: This is Adobe's format, used in Acrobat and various other tools. Although PDF is covered by a number of Adobe patents, according to Adobe's Legal Notices page, the patents "are licensed on a royalty-free, non-exclusive basis for the term of each patent and for the sole purpose of developing software that produces, consumes, and interprets PDF files that are compliant with the Specification." Furthermore, the license can only be terminated if the terms are violated. After some discussion on the ODFI mailing list, it was decided that PDF is "open enough."
SWF: SWF is the format used to encode Macromedia Flash animations. According to the SWF license page, "No fees are required for access to the Macromedia Flash file format (SWF) or for the creation of products based on the SWF format." However, if you read the actual license, nobody is allowed to redistribute the spec (although anyone can, right now, get his or her own copy), and the license also states "You understand and agree that Macromedia may amend, modify, change, and cease distribution or production of the Specification at any time". Since Macromedia is reserving the right to effectively close the spec, it should not be considered open (despite the name of the website OpenSWF that discusses the format).
MP3/MPEG: Patents on MPEG (MP3 is short for "MPEG layer 3") are owned by the Fraunhofer Institute. According to the Licensing page, Fraunhofer has farmed the licensing out to other companies: Via Licensing for MPEG 2/4-AAC (audio), MPEG LA for MPEG-4 (video), and Thomson for MP3 (audio). The royalties vary, but are along similar schemes: PC Software royalties for the MP3 patents are 75 cents per unit (or a $50,000 one-time fee) for decoders (playback) and $2.50 per unit for encoders; MPEG-4 is 25 cents per unit for decoders and/or encoders, MPEG-2 AAC varies from 10 cents to 45 cents for decoders and/or encoders (based on unit volume), etc. Since these formats require license fees, they are not open.
Windows Media: Microsoft has its own audio and video formats used by Windows Media encoders and players; the Licensing page discusses the various royalties (and compares them to royalties for some other formats). For example, the audio format is licensed for 10 cents per decoder, 20 cents per encoder, or 25 cents per encoder/decoder. The royalties are higher on embedded systems than on PC software, and furthermore the page states that "No royalties are due on distributions of 'PC Software' versions designed to operate on versions of Windows operating systems." The format, needless to say, is not considered open.
RealMedia: This is the format used by Real's encoders and players. While the licensing page is somewhat confusing, it appears that Real does not license the format per se; it licenses binaries and even the source code in some instances (for free), but not the format itself for a set royalty. The client source code license talks about "the RealMedia File Format reader, the Real Data Transport network stack implementation, the RealNetwork Client-Server Challenge implementation" etc. which are the technologies that Real is trying to maintain control of. These are the key technologies, so the format is not open.
As an added note, the streaming done by Windows Media and Real are also not open. According to this article in Network Computing [2/19/01], open standards for streaming do exist: "RTSP (Real-Time Streaming Protocol) is an application-layer IETF protocol that allows the interaction between player and server to enable starting, pausing and transfer of information such as stream title. RTP (Real-Time Transport Protocol) is an IETF packet format for transmitting real-time media over UDP. Its companion, RTCP (Real-Time Transport Control Protocol), synchronizes media at the client and reports packet loss to the server."
However, as the article goes on to say, ""Streaming-media vendors have developed alternative proprietary protocols to handle server-to-client media transmission and server-to-server communication. Thus, RealNetworks uses Real Data Transport (RDT) instead of RTCP, and Microsoft's Windows Media Technologies relies on the Microsoft Media Server (MMS) and Media Streaming Broadcast Distribution (MSBD) protocols."
You can also subscribe via email. Send a message to discussion-request@odfi.org with the message body containing
subscribe [password]
where password is the password you will use (one will be assigned if you don't specify it). If you send the text "help" instead you will get a reply with documentation on the email request syntax.
Right now the list archives are only available to list subscribers. This is to avoid having email addresses picked up for spamming (the spam protection that the mailing list software does I consider to be insufficient).
The list right now is pretty busy (150+ messages in the first week) but it will probably slow down. You can also elect to receive messages in a daily digest instead. Or, you can elect not to receive mail at all, and just have an account so you can view the archives.
The one big issue that the sample law did not address was formats that have some intellectual property strings attached to them. They are covered by a patent, they are licensed by a standards body, etc. This was brought up several times (6120347, 6120617, 6120387). One person suggested simply prohibiting formats that require royalty payments (6120467), another suggested three rules for granting an exception to releasing formats, using Oracle as an example (6120410).
There was also some discussion of the openness of Adobe's Portable Document Format (PDF), since the reader is free but the official writing tools are not. One writer pointed out that there are other tools that write PDF (6121600), and another mention that an International Standards Organization (ISO) subset of PDF is being defined (6120729).
This is an issue that needs to be discussed further. One comment stated "I have yet to see a patent that prohibits decoding of file formats, even in USA." (6120920), implying that you could read (but not write) a patented formula without licensing it, but I do not know if this is true.
Other suggestions for improvements to the bill included tightening the language to prevent companies from weaseling out of it by storing their data as an executable (6120634, also posted as a comment on this site, 6120915); defining "useful life" more precisely (6120367); specifying how long before a program is released its formats have to be documented publicly (6133353); and using a Wiki to draft the sample bill (6120419).
Some suggestions on expanding the law will likely not be considered: requiring network access to data (6122275), open network protocols (6120550, 6122467), and not allowing governments to charge for access to public data (6121492).
Several comments expressed misunderstanding about what ODFI was trying to do, despite the fact that I tried to be very clear about this. One person claimed ODFI would force companies to migrate to new software (6120459), and two complained about forcing companies to adopt standards (6124115, 6124612). This shows that we need to keep emphasizing that these are not part of the goals of ODFI. Then again, one person who thought we were requiring standards expressed approval for that (6124730).
Two others believed that ODFI was requiring that all data formats be human-readable (6120589, 6120890). This is probably a misunderstanding of the statement that the documentation itself must be human-readable.
There were some comments against the bill. Some people simply don't want more laws (6121801, 6123346), while someone else asked isn't the current situation OK because programs will always be able to read old formats (6120561)--an argument that was quickly rebutted. One person mentioned that companies would complain that open data formats would hurt "security through obscurity", admitting that it was a bogus argument but still one that would come up in the debate (6120439).
A few people took the pragmatic approach; common sense would be better, but sometimes legislation is needed (6120407, which started a long tangent about seatbelt laws), and an open data format bill may be dogmatically inferior to an open source bill, but it is more likely to pass (6120614).
Several writers said that a program which by default used a proprietary format should be allowed as long as it had an option to export/save its data in a standard format. (6120867, 6121837, 6120585, 6129107). All of these were shouted down by others explaining why people would still tend to use the default format, and many of the standard formats are not as rich as the default.
There were many expressions of support for ODFI. One person wrote "This idea seems too obvious, too clear, too intuitive, and far too easy to implement for any respectable lawmaker to consider it for even a single nanosecond." (6120329). Another pointed out that "the argument for open data formats is the unassailable bastion of open source bills" (6121680). People liked ODFI because it would force more competition (6124468), or remedy Microsoft's monopoly (6126110).
Some quotes from comments include "People acting toward their own individual self interest often create an undesirable result. A little enforced cooperation benefits all." (6121045), "Wouldn't it just be better to have an open market for politicians and officials with an auction process?" (6121740), "I pushed for a similar 'rule' at the local government offices I work at" (6120530), and "You lose freedom when you lose the information that the government (which belongs to everyone) produces/compiles." (6123517).
A few final comments: One person said that many open source programs don't have their formats written down anywhere, so allowing open source as a good alternative would not necessarily work (6120882); another mentioned that a standards-based format for office documents was being proposed (6121464); and finally someone said that the Gnu Public License allowed companies to restrict access to their source code to only those who had binary copies (6121339). This comment was preceded with "IIRC", and wading through the GPL, it's not clear that the poster did recall correctly. It seems the GPL may prevent State Government A from simply taking code that is used by State Government B, but it would allow State Government B to give a copy to State Government A, so I still think government software vendors would be leery of releasing their code under the GPL.
In particular, I think opening up data formats makes a lot of sense for software companies that are dominant in a particular area, and for software companies that are niche players in a particular area.
Consider the case where one company dominates a class of software, with the rest of the market filled out by niche players. The file format used by the dominant company's software has undoubtedly become a de facto standard; its small competitors have reverse-engineered its format in order to give their software any chance of competing, since so much data is stored in that format.
So the dominant company will already have lost whatever advantage it had from owning a proprietary format. It may be worried that if it releases its data format, third party software will be reading and writing it, and possibly corrupting it--and the resulting calls to tech support will be to the dominant company's help line, not those of the niche company that wrote the offending software. But if the format has been reverse-engineered, this is going to happen anyway. Wouldn't it be better if the other companies were at least working from an official, complete format specification, rather than trial-and-error reverse engineering?
Meanwhile, the niche players, if they have their own format, have no reason not to make it public. In fact, in such situations niche players often adopt an existing public standard, trumpeting it as an advantage over the dominant company and allowing several niche players to effectively pool their forces in the battle to unseat Goliath. But even if they decide to invent their own format, releasing it publicly provides the same benefits: it can be used as a selling point, and each time other software gains the ability to read the new format, it increases the usefulness of the format and the niche company's software.
Now, dominant companies may feel that their format is a competitive advantage and that if others are going to use it, they should be licensing it. They should realize, first of all, that they are not licensing it now, so they wouldn't be losing any revenue. And more importantly, if they want to get it established as a de facto standard, they should be encouraging as many companies as possible to use it, not putting financial roadblocks in their way.
Large companies may also worry that "open data formats" means they have to use a standardized data format; as I wrote above, small competitors often do this and then use it as a selling point against dominant companies. The goal of ODFI, however, is not to require standard formats, but only to require that data formats be documented. Dominant companies can change the format as they wish, as long as they document those changes.
Companies may also be worried that documenting their data formats will require them to support older data formats longer. In fact that is false, and may actually save them some work. If the data format is proprietary than the company will need to keep conversion code around, since it is the only official supplier of such code; on the other hand, if the data format is public then in some cases the company may be able to offload the responsibility for that code onto someone else.
The only case where opening a data format might be a competitive disadvantage would be where there were two or three evenly matched competitors in the same market. In such a case, if only one company opened its formats, it might make it easier for competitors to take market share, since it would ease the migration path. Again, however, in such a situation it is likely that the companies have reverse-engineered each others formats or have some other solution for migration, so opening up your data formats should not be a competitive disadvantage.
What if a company has patented a part of its data format? This is an issue that I am working to come up with an official ODFI position on. I think, at a minimum, companies have no reason to charge money to license their patents; for the reasons described above, it is almost always to their advantage not to make it difficult for others to use their formats. One option is that a company could grant a license to use the data format, but only for reading it, not writing it. Most of the benefits of open formats come from others being able to read your format; most of the problems come from others being able to write it. If every word processor, as an example, could read the format of any other, but only write its own (which is essentially the way the market is now), then you would prevent users from being locked into one program and help with data retrieval in the future, which are two of the main goals of ODFI.
The first was a post from Ken Barber, who was the author of the Oregon bill. He wrote, "You are seeing things exactly as I saw them when I wrote the original bill: I felt that its real power was not in the open source provisions -- those were there to get media attention -- but in the requirements for open standards."
The second was a post from someone who said that "reliable" sources have told him/her that the Oregon bill might be revived by swapping the text of it into another bill. The bill that is supposed to be the host for this zombie-like process, Senate Bill 589, is a completely unrelated bill aimed at establishing the "Buy Oregon" Policy Council.
The Oregon legislature, I discovered while looking into this, meets only every two years, for about six months (as opposed to, say, the Washington legislature, which meets every year for about three months). Thus, it appears that if no open source or open data format legislature is passed this year, it will be 2005 before anything else can be done. This is unfortunate since I was hoping to work on open data format legislation for consideration in Oregon next year. However, as I have written, and as Ken Barber indicated, the current "open source" bill being considered in Oregon is really more of an "open data format" law.
A BILL FOR AN ACT
Be It Enacted by the [people]:
SECTION 1:
SECTION 2. For all new software acquisitions, the person or governing body charged with administering each administrative division of [government], including every department, division, agency, board or commission, without regard to the designation given the entity, shall avoid purchasing software products that are not open data format software.
If the software in question is not publicly available, but was written for the [government]'s internal use, the data formats need not be made available to the public as provided in Section 1(A) above, but instead can be provided only to [government]. The data formats used must still satisfy Sections 1(B) through 1(F) above, and alternatives considered as described in Section 3 below.
SECTION 3. If no open data format software is available for a particular task, then [government] shall consider the following alternatives, listed in descending order of preference:
In such cases, the [government] must provide written justification to the central purchasing agency as to why it was unable to obtain open data format software.
SECTION 4. For existing software, the [government] shall give manufacturers one year from the day this bill becomes law to provide documentation on the file formats used. If after one year a manufacturer has not provided such documentation, then a replacement for that piece of software must be sought, following the guidelines outlined above.
(NOTE: I use the term "open source laws," although in fact some of the laws refer to "free software" instead.)
The reasons are as follows:
The three points mentioned most often in favor of open source laws are cost, security, and open data formats. In the lobbying against open source laws, I have never seen any negative comments about open data formats; the focus is on the cost and security arguments.
When discussing cost, opponents of open source laws can point out (correctly) that the actual cost of the product is only one part of the total cost; Microsoft quotes a Gartner Group survey putting the number at 8%. Presumably they found the study with the lowest number, but the general fact is correct. Plus, the cost issue likely favors open source more on the server, where administration costs may be lower with open source software; on the client, where Windows is bundled with almost any computer anyway and support involves helping end users with unfamiliar software, open source won't come out looking as good.
Now, you could argue that even the study that Microsoft is pushing shows that the total TCO of open source is only 92% of what it is for proprietary software. The problem is that this then leads to a long debate about how open source affects the other costs of software (installation, support, administration, etc) and no clear winner will emerge.
Meanwhile, the security issue can easily get embroiled in a FUD battle between the two sides, each claiming that the other has more crashes and remote exploits, each waving studies that support their claims. If you want to convince a legislature to pass a law causing significant, possibly risky changes in government procurement, you can't get stuck in a battle like that. Keep in mind that properly designed secure file formats are not dependent on keeping the file format itself secret, so nobody should be able to argue that open data formats compromise security.
When the debate can be framed in terms of cost and security, the issue of open data formats can be conveniently ignored by opponents. Requring only open data formats would remove the abililty of opponents to attack the cost and security arguments, leaving them to come up with arguments explaining why open data formats are bad, whch I have not seen so far. Finally, governments have presumably always considered cost as a factor when evaluating software purchases, and these days they no doubt consider security too; having a law that focussed only on open data formats would open their eyes to something new, that they have probably missed in the discussion of open source laws.
Many open source laws seem designed to force a government to replace Windows/IIS/Office with Linux/Apache/*Office, but of course they aren't written that way; they discuss open source and proprietary software in general. This can take one of two tacks; either requiring open source or free software with no alternative, or making it difficult to buy proprietary software (for example requiring each purchase be accompanied by written justification).
The first approach takes too simplistic a view of the type of software that governments use. Much of the it is customized for specific tasks such as processing drivers' licenses, and the market for providers of such software is presumably small. If software vendors release their software as open source, they may find that cash-strapped governments in other states gladly help themselves to it for free, so the vendor may get only one paying contract instead of fifty. Therefore, it's quite possible that governments won't be able to find companies willing to provide them with open source software, and then what alternative do they have?
The second solution, requiring case-by-case justification of proprietary software purchases, is putting pressure on the wrong people. In a market with few providers, it is not unreasonable to assume that all companies will refuse to provide their software as open source. The fact that government employees will have to do extra paperwork to justify the purchase is of no concern to software companies as long as the same would be true if the purchase was made from their competitors. Thus the pressue on government employees caused by the need for written justification will likely not be transferred to the actual software vendors who would make the decision to go open source.
In contrast, with open data formats, you are much more likely to find a software vendor who will open its data formats, since it does not impact their ability to sell to others and is an easy way for them to get a marketing advantage on their competitors. And if all companies bidding are willing to open their data formats, then so much the better. An open data format law could be written such that in the case where the software is not generally available, but is sold only to governments, the data formats themselves would only need to be made available to the governments that purchased the software.
Although proponents of these laws may dream that Microsoft is going to open up their source to avoid losing government business, in practice that won't happen. Thus, moving to open source will entail a risky and difficult migration, especially for desktop users. It is much more likely that companies will open up their data formats to hold on to government contracts. It's certainly a gamble, since companies may refuse to even open their data formats and thus force a migration also, but it's a gamble that is much more likely to succeed.
It would be nice for governments to save money on software, but unless you are someone who thinks that governments waste a large percentage of taxpayer money on long lunches and excessive highway landscaping, it won't make a huge difference in your life, and it's not clear open source software would be significantly cheaper in the long run. Similarly, it would be great if governments ran software that didn't allow my personal data to be hacked, but there's no guarantee that open source would be more secure than proprietary software.
The software that the government buys has little effect on the software that I use every day. Whether the government uses Linux or Windows isn't going to change the operating system that I run. In contrast, open data format laws would have a dramatic effect on my life because (hopefully) the data formats that I use will now be made public. If the government uses Office and the Office data format is made public then suddenly the copy of Office that I use will be using a public data format.
With open source software the data format can be determined by examining the source code, so if Microsoft opened the source to Office (which is highly unlikely) then I would, eventually, get some documentation on the Office data format. But this is because people would leap at the chance to document Office; as you consider programs of decreasing popularity, it is less likely that someone is going to bother examining its source code enough to produce complete documentation on its data formats. I would much rather have a authoritative, company-produced, human-readable specification of the data format than depend on the source code and the kindness of strangers.
5. Open source laws are too difficult politically
Open source laws are going to be disruptive to someone: Either Microsoft and other vendors are going to have to completely change their business model, or the government is going to have to do a whole lot of migration and/or justification. Thus an open source law is bound to get significant opposition from both camps, as we saw in the debate on Oregon's law.
Open data format laws, on the other hand, don't necessarily require much disruption. Microsoft has the Office data format fully documented; it just needs to make the documentation public. Since the format has been reverse-engineered anyway, is this such a disruption? The company can continue to charge whatever it wants for software, under any license it wishes, and can keep complete control of the format, including changing it as often as it wants--as long as it documents the changes.
Microsoft will undoubtedly still lobby against open data format laws, but its arguments will be weakened significantly. When deploying its usual counter-arguments about cost and security, it can argue at a higher level, claiming that the opponents' arguments are just wrong. It doesn't have to get into reasons why open source laws would be bad for Microsoft personally, which would come across as much whinier and self-serving. But I don't see how it can make a philosophical argument that open data formats are generally bad, so I would be interested to see what kind of excuses it can come up with as to why they shouldn't be required.
In an ideal world, perhaps an open source law would work. But in the real world, we should focus on open data format laws as a battle that can be won.