Ross Randomly Rants: technology

Showing posts with label technology. Show all posts

2009-07-07

How not to measure temperature

Former TV Weatherman Anthony Watts asked an interesting question one day. He wondered if the kind of paint used to protect the shelters that hold the thermometers used for over a hundred years to measure local temperatures mattered. So he did a little research, found some recipes for classic whitewash and some samples of modern paints and set up an experiment.

Along the way, he got a look at a couple of official weather stations. They were not a pretty sight. He looked at a couple more. They weren't a lot better. Then he had a bright idea. Why not go and personally inspect all 1221 stations that make up the US Historical Climate Network (USHCN). This is a "high quality, moderate-sized data set of daily and monthly records of basic meteorological variables from over 1000 observing stations across the 48 contiguous United States" according to its maintainers.

This is a huge undertaking, but odder things have come to pass with the help of volunteers and the Internet to organize them. Thus came about the Surface Stations site and the army of volunteers that have surveyed, photographed and rated over 70% of the USHCN stations to date. Site ratings are based on NOAA's own guidelines and criteria.

Along the way, stations are found that have issues so clear that they have been singled out to become the subject of blog posts. Admittedly, these are generally the worst of the worst of the stations, so this list clearly shows a kind of selection bias...

To date, 89 examples of How Not to Measure Temperature have been posted at Watts Up With That. I've listed them all here, because there isn't an easy way to get this list directly from the site and they make interesting reading. Naturally, this list will be out of date shortly...

To balance the hugely unfair selection of stations shown above, here are some posts about stations that are exceptionally well sited and maintained:

I wouldn't want to sound too judgmental, but the survey results show that the USHCN is anything but a high quality network, because about 89% of stations surveyed fail to meet reasonable standards for siting. Standards published by the same people who created the USHCN in the first place.

A report was published at the 70% coverage point, and more analysis is in the works.

Worse, it is hard to imagine that the situation in much of the rest of the world is a whole lot better.

2009-02-14

I told you RFID didn't belong in passports!

<rant>

A recent RISKS issue contains this item contributed by the editor, Peter G. Neumann:

Using inexpensive off-the-shelf components (a Motorola RFID reader and antenna, and a PC) ... Chris Paget ... built a mobile platform in his spare time that can clone large numbers of the unique RFID tag electronic identifiers used in U.S. passport cards and next generation drivers licenses. While driving around San Francisco for 20 minutes, he was able to harvest two passport tags without knowledge of their owners from up to 30 feet away.

Read the rest of the RISKS item over at the archive, or a complete article over at SecurityFocus.

Given the huge security risks that ought to be self-evident here, what justifies the use of a technology like RFID that can be read at a distance without the awareness of the document holder?

The only sensible approach in the first place was to use an optical technology such as a 2-D barcode symbology if it really was necessary to put more information into the electronic record than just the passport number itself. Printed inside the cover (or on the back of a card) it would be immensely more difficult to read at a distance, and since the cover would normally not be left open, it would rarely be possible to get the content without the holder knowing. Even better, because the symbol is visible the owner would be more likely to know that the threat exists, and avoid leaving it laying out open to the symbol in plain view.

In contrast, the RFID tag is invisible to the owner and can be read without opening the document (or without even removing it from a pocket or wallet).

Until the world comes to its senses, wrapping your passport in copper foil may well be the only sensible action. Of course, you will then have to explain yourself to the nice TSA agent at the security checkpoint...

</rant>

2007-12-23

Things learned when searching Google

While looking for something else entirely, I tripped over some general computer knowledge pages at http://vlaurie.com/ which included the surprising bit of trivia that there is a kind of font editor shipping in XP, but not well known.

It is the Private Character Editor, and is intended to allow an end user to design characters in the Unicode range dedicated for private use. This is the recommended place to put a company logo character, for example. Apparently, there is a mechanism in place in Windows to allow characters built in this tool to be logically attached to individual fonts.

The private character editor is not found in the start menu. To launch it, click Start, Run, type eudcedit in the box and click Ok. If you want to create a shortcut to it, it seems to live in %SYSTEMROOT%\System32\eudcedit.exe

The private characters can be retrieved for use in a document with the aid of Character Map. Of course, once you use a private character in a document you are faced with a different dilemma, namely how to let anyone else use the character too... I'll leave that as an exercise to the reader.

2007-12-15

An impressive single-tasker

We've all had the experience of taking a string of Christmas lights out of the box, testing them to prove they work, stringing them up, and finding that now they don't work. Until now, I haven't known of a better solution than to inspect each bulb individually, which is a painful process.

This year, I found the right tool for the job: the Lightkeeper Pro.

Its advertising, packaging, and even its documentation reads like a Super Bass-o-Matic '76 commercial. But actually does work as described. Honest!

Within its vaguely gun-shaped package is a bulb remover, a 60 Hz hum tracer, a bulb tester, a fuse tester, storage for spare bulbs, and a flashlight.

But wait! There's more!

The real magical feature is the piezoelectric light string repair device. Just remove any bulb from a dark segment, plug the gun in where the bulb was, and pull the trigger. After a couple of pulls, the string is likely to just turn back on. All except the bad bulb, which you can then replace.

Now how much would you pay?

Remember, all this can is done with the string installed and power on. And with only one short trip up the ladder to the top of the eaves.

MSRP is $25, but it can be had for $20 or so in stores like Amazon, Target and Lowes. Even if you don't need one this year, grab one in the post-season sales and add it to your toolbox. You'll thank me next year when you pull the lights back out of storage.

2007-08-24

RFID Passports Revisited

I've written about RFID and passports before, but have let most of the recent news about its interaction with the new biometric passport standards slip. But I can't pass up the latest tidbit.

The problem with RFID in general is that you can't make reasoned arguments against it without sounding like a kook that is afraid of black helicopters and mind probes. The thing is, that there really are some serious issues raised by the widespread adoption of RFID that simply are not possible if simpler (and cheaper, and more suited to the application) technologies like 2D barcode are used instead.

In short, the advantage of RFID is that the tag's information can be retrieved without the operator needing to actually see the tag. The disadvantage of RFID is that the tag's information can be retrieved without the operator needing to actually see the tag. But that actually isn't the reason for today's bit of ranting.

The ICAO is the agency of the UN that sets standards for passports, which are generally adopted by the member nations, including the US. The ICAO established a standard requiring that passports carry biometric information about the holder in addition to the usual date of birth, full name, and issuing country which is about all the information that passports classically carry.

Apparently, a typical bit of biometric info to include is an image of a fingerprint. Rather than storing an encoded hash of a matching print that would require a border checkpoint to use an equivalent (usually patented if not trade secret) technology to match a bearer's print, someone chose to store a simple image of the print. Since it is important to use as little space as possible to store the image, JPEG2000 is used to compress a photograph of the fingerprint. If you are going to store a fingerprint in a document with a hoped-for lifetime of 10 years, that is all reasonable.

Unfortunately, the makers of the passport readers all appear to have used JPEG2000 implementations that have some (or all) source code in common. In particular, there is a known bug in the JPEG2000 implementation that makes it vulnerable to a buffer overrun attack.

A determined individual demonstrated recently that he was able to use off-the-shelf technology to clone a copy of a passport's RFID chip. That is bad enough. But his latest revelation is worse: he was able to modify his cloned copy of a passport's otherwise valid RFID tag to include a fingerprint picture that exercises a known buffer overrun attack on JPEG2000. That tag has been demonstrated to crash every passport reader it has touched.

Today's attack is just a denial of service. The passport station would be crashed, and under normal conditions it could be restarted relatively easily and wouldn't crash again until another bad passport was brought through. (Of course the fiasco at LAX the other day involving 10000 people stuck on the tarmac outside of the customs barrier for as much as seven hours could be the result of that denial, especially if the official reaction to such a crash was institutionally stupid.)

But one of the better ways to inject code of the attacker's own choosing into a device is to carefully craft a buffer overrun attack so that rather than crashing the targeted system, it actually takes it over. At that point, you have code provided by an attacker running inside a passport terminal. These terminals are networked, and it is not inconceivable that such an attack could have effects that are both subtle and far reaching.... I'll let you fill in your own risks from that point.

2007-08-22

Mooning Again

Since I last mentioned the Lua programming language in these pages, it has seen a surge in visibility. Yes, I know that is a coincidence, because I am pretty sure I don't have enough readers of my random musings to have had an effect.

But it is interesting to see that it has sustained its place in the TIOBE top 20 at the respectable position of #15 this month. Since TIOBE doesn't appear to keep an archive of each month's scoreboard, here is the list as of August, 2007:

Rank	Language	Ratings
1	Java	21.768%
2	C	15.699%
3	Visual Basic	10.646%
4	C++	10.111%
5	PHP	9.696%
6	Perl	5.320%
7	C#	3.987%
8	Python	2.749%
9	JavaScript	2.575%
10	Ruby	1.906%
11	PL/SQL	1.833%
12	SAS	1.389%
13	D	1.251%
14	Delphi	1.222%
15	Lua	0.645%
16	COBOL	0.600%
17	ABAP	0.587%
18	Lisp/Scheme	0.585%
19	Transact-SQL	0.549%
20	Ada	0.537%

A couple of things struck me about this list of languages. One is the absence of FORTRAN despite the continued presence of COBOL and SQL, both more venerable. This month, FORTRAN is found down at #21, or just pushed out of the top 20. The presence of D in the ranking at 13 (D is the 13th letter for the numerologists in the audience) is amusing at least partly because Walter Bright, D's creator, is a friend of a friend. I am sad to see that Java continues to dominate, but heartened to notice that the C/C++ family if counted as one language would both push Java down a notch, and allow FORTRAN to return to the top 20.

The real surprise is Lua. A year ago, TIOBE had hardly heard of the language. Last December, Lua just slipped on to the top 50 list, at a position quite near to Objective-C (near and dear to Mac OSX developers--interestingly, Objective-C hasn't budged much at all since December, since it is sitting at #50 exactly today). Lua crossed into the top 20 list by landing at 18 in July, and clearly it is on track to improve its current standing again for September.

The full story on what the TIOBE index is calculated from is found here. In short, they run hand tuned queries against all the major search engines for each language, count the significant hits, and do a lot of statistics to arrive at the score for each language. The final ranking is then simply the list of languages sorted by their scores.

In case my biases aren't clear, Lua is my new favorite language especially when combined with just enough C to get the job done. The majority of my productive work is done in C, Lua, and Perl, with the occasional diversion to C++ and on rare occasions, Java. Although I have not personally used any of the .NET family, I firmly believe that .NET is to be preferred to Java any day.

2007-04-18

Raving about a moon

Several months ago I tripped over a little language called Lua. It seemed like a novel idea, but its absence of curly braces and lack of ++, += and similar operators felt uncomfortable after years of C, and I didn't look very close. Then I started to notice it hiding in the most unlikely places, culminating in Adobe Photoshop Lightroom where the development team claims that over 40% of LR is implemented in the Lua programming language. It seemed like it was worth a second look (along with SQLite, the database engine behind LR, but that is another subject altogether) so I started to play with it.

In a word, "wow".

Lua is a scripting and extension language designed to be easily embedded in a host application. It has a simple C language API that permits the application to easily call functions and manipulate lua values. It also makes it easy to call C functions from lua code, and to add those functions by dynamic linkage to a running program from shared libraries (DLLs on Windows).

Lua was invented by a resourceful group at a Brazilian college to serve needs they saw for configuring and extending applications. Since it was a successor to an earlier language called Sol (Portuguese for Sun), it got named Lua (Portuguese for Moon).

As I see it, several properties of the language have contributed to its success. First, the Lua team has chosen to use the MIT license which is an open source license that also permits effectively unrestricted use in commercial products. This has made it legally possible for commercial developers to consider using the language in their products; and probably as a result of this leniency a significant number of commercial product teams are regular contributors to the lua programming community.

Second, Lua is implemented in standard C, sticking to the subset of the C language that has identical semantics if compiled as C++. The Lua language itself is compiled for a virtual machine and can be saved as (non-portable) bytecode before execution in the VM. Since it is possible (and encouraged) to write lua programs that don't need to compile new code at runtime, it is easy to separate the compiler from the interpreter and distribute only pre-compiled byte code files. Because of this, it is amazingly portable. It is found inside a shocking number of game consoles, PC games, more general products like LR, and even a wide array of embedded systems. In an embedded system, it is most likely to have the compiler stripped off to reduce the amount of code and data space required in a limited environment.

Third, and certainly not last, lua is a very expressive language. The designers have emphasized mechanisms over policies, and as a result although lua is not an object oriented language, you can use its core metatable features to implement object oriented features such as single and multiple inheritance, private data, and object and class methods. Functions are a first-class value, and hence can be stored in variables, passed as arguments to functions, and even be generated (as closures) as the return values from functions.

The core language provides numeric and string data types, along with nil, tables, functions and threads, and two kinds of data wrappers for opaque data belonging to the C side of the fence. The language provides an incremental garbage collector that can be tuned to avoid long stalls for collection.

The C API makes it easy to provide wrappers for third party libraries and system calls that can be loaded on demand. By putting the most demanding inner loops and algorithms into C libraries and leaving the overall program structure to lua, powerful utilities can be built quickly.

My first project in Lua was a serial protocol debugger for a new feature for one of Cheshire's projects. In about 1500 lines of lua and another 1500 lines of C (and only a couple of days work) I was able to build an interactive serial port debugging tool that could log traffic, load and save the logs, send arbitrary data to the device, send random data to the device, and play saved files at the device at an adjustable rate. Building something like that from scratch with C++ and MFC would have taken a lot longer and may never have been as feature rich.

Since then I have built a protocol analyzer for an unrelated Cheshire project with a much more complex packet based protocol, and even built a live status monitor with meters and displays. I have created my own generalized C language host application so that I can deliver an installable windows application as an executable, a few DLLs, and a nest of lua scripts. When my host engine is ready for public view, I will be releasing a version of it back to the Lua programming community as my thanks.

Learn more about Lua at www.lua.org where you will find links to the language itself, its reference manual, books published by its authors, and of course links to the most prominent bits of the lua community: the lua wiki and luaforge.

The tools I've built for Windows would not have been possible without some of the other work contributed by people associated with Lua's creation: IUP for portable user interface design, CD for 2-D graphics, and IM for image processing.

2006-10-03

More RFID Madness

I noticed this on the MAKE blog... Johnathon Westhues has designed an inexpensive device based on a PIC microcontroller that implements everything needed to clone a VeriChip. As built, he needs to get the antenna into close proximity, but that is just a matter of social engineering; the sort of social engineering that pickpockets have been good at for as longer than pockets have been common.

VeriChip Corporation is pushing for adoption of their implantable RFID technology in hospitals (if every victim had a chip, the ER could have their medical records as the gurney arrives), for child protection (some blue-sky idea about tracking lost or kidnapped kids), for elder protection (similar to the kid tracking, but for the elderly with a tendancy to wander away).

Their web site says: "And unlike conventional forms of identification, the VeriChip™ cannot be lost, stolen, misplaced, or counterfeited. It is safe, secure, reversible, and always with you."

Right. It has been successfully cloned (read "stolen") by under $30 of commonly available parts. There are companies (and an agency of the Mexican government!) out there that are using these things as a security token... but they contain nothing more secure than a 16-digit unencrypted identifier. Publish the identifier belonging to a specific individual and what security it had is now lost.

They even claim that their readers work up to 10 feet away. That means that chipped individuals can be logged at public locations by simply requiring that the general public pass through a choke point (say a door or hallway) no more than 10 feet across. And if a 10 foot range was commercially viable, then it is obvious that a hacker with a nose for antenna design can build a suitably sensitive device to read the chip at 100 feet or more.

2006-02-11

Zillow, Your Edge in Real Estate

I don't know how to react to this.

I have just discovered Zillow, the latest project from the people who created Expedia.

This is a blend of Google Maps with every tax assessor's database to produce an easy to browse view of home values. Zoom out and it shows aggregate prices by region. Zoom in and the regions get more precise. Zoom all the way in and get detailed estimates of the values of individual homes.

I'm not sure if any privacy lines have been crossed. Everything is based on public databases. The value they add is all in the accessibility and aggregation. And they appear to have resisted any temptation to include the (also public record) legal owner of the parcel in their reports.

They claim their estimates are within 10% of the actual selling price (by checking historical estimates against actual sales) 66% of the time for LA metro, and better than that nationally.

They also claim my home is worth 50% more than I thought it was....

Hmm.

2005-03-24

Java, the Coffee Cult

<rant>
The Wictionary defines “cult” as “1. A group or doctrine with religious or philosophical identity viewed as a sect, often existent on the margins of society.”

I have had occasion in the past month to learn Java by the full-immersion method. While there are many ideas and practices embodied in the language and its tools that I admire, I have skeptical of the more extreme claims made for the language. After a month, I am convinced that the Java user community is, in effect, a cult.

Who am I to have an opinion?

I have been working professionally in embedded systems design and implementation for over twenty years. For much of that time, I have found that C remains my language of choice when faced with manipulating bits in the registers of actual hardware devices. Naturally, you can’t escape from the native assembly language of the system’s CPU for some aspects of device drivers, interrupt handlers, and thread scheduling; but for just about everything else, C serves well. When building tools for general use, I have generally reached for C as well, but C++ and Perl are gaining ground.

The customer, however, wants this project to include a substantial chunk of user interface built in Java, with a thin layer (device drivers and low-level routines at most) built in C.

Recent versions of Java are showing signs of becoming a usable language. Version 5 of the Java API introduces a limited kind of template and a first-class enumeration type. But since version 1.0 more than a decade ago we have been hearing claims about Java revolutionizing the world. What happened?

It’s a cult.

One hallmark of a cult is the sweeping claim of a new order and revolutionary changes. Another is isolation of its members from their pasts. The Java Way teaches that Java is the only way. “Write once, run everywhere” they loudly trumpet. But many core technologies sacrifice compatibility with any particular platform on the altar of broad portability. Many of the tutorials at java.sun.com are proud of the fact that Java is not particularly good at playing with others.

If they really wanted to reach a world-changing number of programmers they could open up the JVM to the point where compilers targeting Java bytecode could be written for other languages. Perhaps a few tweaks or minor new features would make it easier to write C++ code that runs in the same JVM and interacts with Java classes. Perhaps someone would even release compilers for FORTRAN, COBOL and BASIC, not to mention Ada, APL, Eiffle, Forth, Haskell, Lisp, Logo, Modula 2, Pascal, Prolog, Scheme, and Smalltalk…

Oh wait, Microsoft actually did just that already. Their oddly named .NET platform is based on a just-in-time compiled virtual machine that has open source implementations available in addition to the “official” release from MS. The open-source implementations run on a variety of platforms (at minimum on Linux and other Unix-like places). MS is responsible for a slew of languages built on it, and I counted a dozen or more research groups using the platform for niche research language projects, while the commercial language vendors are happily eking new life out of APL, COBOL, and FORTRAN on the platform.

But Sun didn’t want to play along, and packed their sandbox up and took it home.

Both platforms weigh in at a hefty 150MB or more for just the platform installation. Both can be installed free for the download time and bandwidth. The Java compilers and development tools are largely also free, and many are open source, while the .NET tools (at least the “official” ones) are all commercial products.

Up to now, I have avoided building utilities and tools in either platform. If I wanted to release a tool internally, it has seemed more productive to require the modest impact of a recent Perl installation which actually is quite portable even back to Windows 9x versions. For products delivered to end users (who in my companies experience are usually running on Windows) rather than among a development team I have generally preferred to build true Windows applications out of C++ and MFC.

To be fair (why, I’m ranting here… oh never mind) a lot of work in Java Swing was devoted to efforts to make its user interface components work well on lots of platforms while making it possible to mimic the look and feel of many of them. That was the result of a lot of hard work by some enormously clever people, and on the two platforms I have personal experience with, it works reasonably well.

Garbage collection has been a feature of Java since before version 1 was released, and it certainly does have benefits by allowing the programmer to stop worrying about memory allocation. But memory is not the only resource that needs an allocation strategy, and because I have yet to fully grok the Java way of handling non-memory resources I won’t try to use my lack of knowledge of a best practice as evidence that there is or isn’t one.

However, users I have talked to over the years rarely, if ever, change platforms. To an end user the idea that the same program (no, the very same binary file) can be executed on more than one platform simply has no value. If that user perceives that performance or usability has been sacrificed, then he may seek another product entirely.

Java is a Coffee Cult.
</rant>

2005-03-03

A bad use for RFID

Ok, so it really begins...

<rant>
I am coming to a conclusion that makes me sound like a paranoid freak, or a Luddite. I don't like RFID. Perhaps it has narrow applications where its use is essentially harmless. However, it is being foisted upon all of us like a magic bullet solution to all problems. It ain't. It's really just a fancy barcode that can be read without the holder's knowledge, and at a distance.

Crypto helps, but is not enough. A recent RISKS has a link to a research group that has worked out a way to completely reverse engineer the entire secret key from a particular TI encrypted tag that is widely deployed (a big-three automaker's "smart" ignition key, and Mobil SpeedPass among other applications) from fewer than 10 queries requiring no more than 2 seconds to gather the data. From the collected data, they can duplicate the tag in only a few hours using less than $5000 worth of off-the-shelf hardware.

Worse is the idea that tags could become ubiquitously buried in consumer goods. Walmart is the major driving force here. They have this pipe dream of doing whole-cart checkout by just letting *known* customers just push the cart past a reader. The problem is all the extra inventory data that will just sit their sewn into seams in clothing, cast in soles of shoes, and so forth. Some of that could be used to track people. Some could be used simply to violate privacy. (Hey, like those red panties you're wearing, Mr....) And then there's the easy opportunity for "extra charges" the second time you go shopping carrying or wearing one... (There may be a reader out there that remembers the trouble that we caused a classmate once by sticking a Ralphs anti-theft tag inside his Caltech ID?)

But the worst is the fact that thanks to some damn fools in some part of the US government, all Passports are supposted to use RFID to enable them to be read and validated. Not just US passports, all countries are supposed to be joining in. The tags will hold the country or origin, identifiers like the passport number itself, as well as lots of identity details of the bearer. When it was suggested that this would enable identity theft at a distance, the concern was laughed down as paranoid as well as because the tag reader is designed to work close up. But it wasn't designed to not work at a distance. And close up is enough of a threat. It wouldn't be at all difficult to leave a package in a taxi that will act on the next American (or...) to get in the cab.

The US will start issuing Passports with RFID in late 2005, so renew now to avoid the pesky things for 10 more years in hopes that they come to their senses.

For most of these applications, optical codes of one sort or another are a much better solution. For passports, especially, a 2D code could be used on an inside page, allowing many KB of data to be carried, robustly read, and avoid any possibility of accidental information leakage.

I do carry a SpeedPass, but the mechanical design of a gas station makes it highly unlikely that I could possibly pay for someone else's gas. Besides, we do check the monthly statements, and the total risk exposure is negligible since it really is fundamentally a credit card and those risks are managed by the issuer. I don't happen to own a vehicle with RFID in the ignition key, but I have heard horror stories about the cost of replacing a lost key, while (at least in the LA basin) the thief is likely to make more by chopping the car up for parts and so really doesn't care if he has keys.

All in all, it often seems like there is just plain an urge to use the highest possible tech to solve problems, even when there really isn't a problem there that needs solving.
</rant>

Ross Randomly Rants