Things learned when searching Google

While looking for something else entirely, I tripped over some general computer knowledge pages at http://vlaurie.com/ which included the surprising bit of trivia that there is a kind of font editor shipping in XP, but not well known.

It is the Private Character Editor, and is intended to allow an end user to design characters in the Unicode range dedicated for private use. This is the recommended place to put a company logo character, for example. Apparently, there is a mechanism in place in Windows to allow characters built in this tool to be logically attached to individual fonts.

The private character editor is not found in the start menu. To launch it, click Start, Run, type eudcedit in the box and click Ok. If you want to create a shortcut to it, it seems to live in %SYSTEMROOT%\System32\eudcedit.exe

The private characters can be retrieved for use in a document with the aid of Character Map. Of course, once you use a private character in a document you are faced with a different dilemma, namely how to let anyone else use the character too... I'll leave that as an exercise to the reader.

Mooning Again

Since I last mentioned the Lua programming language in these pages, it has seen a surge in visibility. Yes, I know that is a coincidence, because I am pretty sure I don't have enough readers of my random musings to have had an effect.

But it is interesting to see that it has sustained its place in the TIOBE top 20 at the respectable position of #15 this month. Since TIOBE doesn't appear to keep an archive of each month's scoreboard, here is the list as of August, 2007:

Rank	Language	Ratings
1	Java	21.768%
2	C	15.699%
3	Visual Basic	10.646%
4	C++	10.111%
5	PHP	9.696%
6	Perl	5.320%
7	C#	3.987%
8	Python	2.749%
9	JavaScript	2.575%
10	Ruby	1.906%
11	PL/SQL	1.833%
12	SAS	1.389%
13	D	1.251%
14	Delphi	1.222%
15	Lua	0.645%
16	COBOL	0.600%
17	ABAP	0.587%
18	Lisp/Scheme	0.585%
19	Transact-SQL	0.549%
20	Ada	0.537%

A couple of things struck me about this list of languages. One is the absence of FORTRAN despite the continued presence of COBOL and SQL, both more venerable. This month, FORTRAN is found down at #21, or just pushed out of the top 20. The presence of D in the ranking at 13 (D is the 13th letter for the numerologists in the audience) is amusing at least partly because Walter Bright, D's creator, is a friend of a friend. I am sad to see that Java continues to dominate, but heartened to notice that the C/C++ family if counted as one language would both push Java down a notch, and allow FORTRAN to return to the top 20.

The real surprise is Lua. A year ago, TIOBE had hardly heard of the language. Last December, Lua just slipped on to the top 50 list, at a position quite near to Objective-C (near and dear to Mac OSX developers--interestingly, Objective-C hasn't budged much at all since December, since it is sitting at #50 exactly today). Lua crossed into the top 20 list by landing at 18 in July, and clearly it is on track to improve its current standing again for September.

The full story on what the TIOBE index is calculated from is found here. In short, they run hand tuned queries against all the major search engines for each language, count the significant hits, and do a lot of statistics to arrive at the score for each language. The final ranking is then simply the list of languages sorted by their scores.

In case my biases aren't clear, Lua is my new favorite language especially when combined with just enough C to get the job done. The majority of my productive work is done in C, Lua, and Perl, with the occasional diversion to C++ and on rare occasions, Java. Although I have not personally used any of the .NET family, I firmly believe that .NET is to be preferred to Java any day.

Raving about a moon

Several months ago I tripped over a little language called Lua. It seemed like a novel idea, but its absence of curly braces and lack of ++, += and similar operators felt uncomfortable after years of C, and I didn't look very close. Then I started to notice it hiding in the most unlikely places, culminating in Adobe Photoshop Lightroom where the development team claims that over 40% of LR is implemented in the Lua programming language. It seemed like it was worth a second look (along with SQLite, the database engine behind LR, but that is another subject altogether) so I started to play with it.

In a word, "wow".

Lua is a scripting and extension language designed to be easily embedded in a host application. It has a simple C language API that permits the application to easily call functions and manipulate lua values. It also makes it easy to call C functions from lua code, and to add those functions by dynamic linkage to a running program from shared libraries (DLLs on Windows).

Lua was invented by a resourceful group at a Brazilian college to serve needs they saw for configuring and extending applications. Since it was a successor to an earlier language called Sol (Portuguese for Sun), it got named Lua (Portuguese for Moon).

As I see it, several properties of the language have contributed to its success. First, the Lua team has chosen to use the MIT license which is an open source license that also permits effectively unrestricted use in commercial products. This has made it legally possible for commercial developers to consider using the language in their products; and probably as a result of this leniency a significant number of commercial product teams are regular contributors to the lua programming community.

Second, Lua is implemented in standard C, sticking to the subset of the C language that has identical semantics if compiled as C++. The Lua language itself is compiled for a virtual machine and can be saved as (non-portable) bytecode before execution in the VM. Since it is possible (and encouraged) to write lua programs that don't need to compile new code at runtime, it is easy to separate the compiler from the interpreter and distribute only pre-compiled byte code files. Because of this, it is amazingly portable. It is found inside a shocking number of game consoles, PC games, more general products like LR, and even a wide array of embedded systems. In an embedded system, it is most likely to have the compiler stripped off to reduce the amount of code and data space required in a limited environment.

Third, and certainly not last, lua is a very expressive language. The designers have emphasized mechanisms over policies, and as a result although lua is not an object oriented language, you can use its core metatable features to implement object oriented features such as single and multiple inheritance, private data, and object and class methods. Functions are a first-class value, and hence can be stored in variables, passed as arguments to functions, and even be generated (as closures) as the return values from functions.

The core language provides numeric and string data types, along with nil, tables, functions and threads, and two kinds of data wrappers for opaque data belonging to the C side of the fence. The language provides an incremental garbage collector that can be tuned to avoid long stalls for collection.

The C API makes it easy to provide wrappers for third party libraries and system calls that can be loaded on demand. By putting the most demanding inner loops and algorithms into C libraries and leaving the overall program structure to lua, powerful utilities can be built quickly.

My first project in Lua was a serial protocol debugger for a new feature for one of Cheshire's projects. In about 1500 lines of lua and another 1500 lines of C (and only a couple of days work) I was able to build an interactive serial port debugging tool that could log traffic, load and save the logs, send arbitrary data to the device, send random data to the device, and play saved files at the device at an adjustable rate. Building something like that from scratch with C++ and MFC would have taken a lot longer and may never have been as feature rich.

Since then I have built a protocol analyzer for an unrelated Cheshire project with a much more complex packet based protocol, and even built a live status monitor with meters and displays. I have created my own generalized C language host application so that I can deliver an installable windows application as an executable, a few DLLs, and a nest of lua scripts. When my host engine is ready for public view, I will be releasing a version of it back to the Lua programming community as my thanks.

Learn more about Lua at www.lua.org where you will find links to the language itself, its reference manual, books published by its authors, and of course links to the most prominent bits of the lua community: the lua wiki and luaforge.

The tools I've built for Windows would not have been possible without some of the other work contributed by people associated with Lua's creation: IUP for portable user interface design, CD for 2-D graphics, and IM for image processing.

Java, the Coffee Cult

<rant>
The Wictionary defines “cult” as “1. A group or doctrine with religious or philosophical identity viewed as a sect, often existent on the margins of society.”

I have had occasion in the past month to learn Java by the full-immersion method. While there are many ideas and practices embodied in the language and its tools that I admire, I have skeptical of the more extreme claims made for the language. After a month, I am convinced that the Java user community is, in effect, a cult.

Who am I to have an opinion?

I have been working professionally in embedded systems design and implementation for over twenty years. For much of that time, I have found that C remains my language of choice when faced with manipulating bits in the registers of actual hardware devices. Naturally, you can’t escape from the native assembly language of the system’s CPU for some aspects of device drivers, interrupt handlers, and thread scheduling; but for just about everything else, C serves well. When building tools for general use, I have generally reached for C as well, but C++ and Perl are gaining ground.

The customer, however, wants this project to include a substantial chunk of user interface built in Java, with a thin layer (device drivers and low-level routines at most) built in C.

Recent versions of Java are showing signs of becoming a usable language. Version 5 of the Java API introduces a limited kind of template and a first-class enumeration type. But since version 1.0 more than a decade ago we have been hearing claims about Java revolutionizing the world. What happened?

It’s a cult.

One hallmark of a cult is the sweeping claim of a new order and revolutionary changes. Another is isolation of its members from their pasts. The Java Way teaches that Java is the only way. “Write once, run everywhere” they loudly trumpet. But many core technologies sacrifice compatibility with any particular platform on the altar of broad portability. Many of the tutorials at java.sun.com are proud of the fact that Java is not particularly good at playing with others.

If they really wanted to reach a world-changing number of programmers they could open up the JVM to the point where compilers targeting Java bytecode could be written for other languages. Perhaps a few tweaks or minor new features would make it easier to write C++ code that runs in the same JVM and interacts with Java classes. Perhaps someone would even release compilers for FORTRAN, COBOL and BASIC, not to mention Ada, APL, Eiffle, Forth, Haskell, Lisp, Logo, Modula 2, Pascal, Prolog, Scheme, and Smalltalk…

Oh wait, Microsoft actually did just that already. Their oddly named .NET platform is based on a just-in-time compiled virtual machine that has open source implementations available in addition to the “official” release from MS. The open-source implementations run on a variety of platforms (at minimum on Linux and other Unix-like places). MS is responsible for a slew of languages built on it, and I counted a dozen or more research groups using the platform for niche research language projects, while the commercial language vendors are happily eking new life out of APL, COBOL, and FORTRAN on the platform.

But Sun didn’t want to play along, and packed their sandbox up and took it home.

Both platforms weigh in at a hefty 150MB or more for just the platform installation. Both can be installed free for the download time and bandwidth. The Java compilers and development tools are largely also free, and many are open source, while the .NET tools (at least the “official” ones) are all commercial products.

Up to now, I have avoided building utilities and tools in either platform. If I wanted to release a tool internally, it has seemed more productive to require the modest impact of a recent Perl installation which actually is quite portable even back to Windows 9x versions. For products delivered to end users (who in my companies experience are usually running on Windows) rather than among a development team I have generally preferred to build true Windows applications out of C++ and MFC.

To be fair (why, I’m ranting here… oh never mind) a lot of work in Java Swing was devoted to efforts to make its user interface components work well on lots of platforms while making it possible to mimic the look and feel of many of them. That was the result of a lot of hard work by some enormously clever people, and on the two platforms I have personal experience with, it works reasonably well.

Garbage collection has been a feature of Java since before version 1 was released, and it certainly does have benefits by allowing the programmer to stop worrying about memory allocation. But memory is not the only resource that needs an allocation strategy, and because I have yet to fully grok the Java way of handling non-memory resources I won’t try to use my lack of knowledge of a best practice as evidence that there is or isn’t one.

However, users I have talked to over the years rarely, if ever, change platforms. To an end user the idea that the same program (no, the very same binary file) can be executed on more than one platform simply has no value. If that user perceives that performance or usability has been sacrificed, then he may seek another product entirely.

Java is a Coffee Cult.
</rant>