Plotting temperatures

I've rewritten my temperature plotting code in C++, so that there can't be any complaints from the Mono haters, and made it into a Google code project. So now anyone can easily plot historical temperature data in a variety of ways. You can look at temperatures for countries, geographical areas, latitudes, individual weather stations or globally. Also you can plot the number of active weather stations over time, either worldwide or within a particular area, and produce KML files so that their locations can be viewed using Google Earth.



I havn't tested this on Windows, but since both GCC and gnuplot can be run on Windows I expect that it could be recompiled for that OS with minimal or no modifications.

Singularity Research Challenge: funding the wrong stuff

It's kind of sad how SIAI seems to have become obsessed with "AI risks" and human extinction. Perhaps they always were from the beginning, but it's just my perception of them that was at fault. There's certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.

The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.

The sorts of thing which SIAI ought to be focusing its efforts upon are answering questions such as:

• Do we know how to build an AI? Why have we failed so frequently in the past?
  

• What sort of uses do we want AI put to, or not put to?

• Can we formulate a set of requirements and benchmark tests which are useful to researchers?

• Can we identify a set of key problems which need to be overcome so that we can make further progress?

• Can we advocate for research to be done on the problems which will yield the greatest benefit? If not, can we raise funds to do this research ourselves?
  

• Can we discourage research which is frivolous or wasteful of funds, and does not contribute anything significant to the sum of knowledge (eg. Minsky's "little robots" criticism/robot "bees").

As a more concrete example much has been said about the so-called demographics problem or "pensions bomb". What kinds of AI technology can be used to assist the elderly? Can we put together a specific list, then encourage researchers to work on those problems in a smart and coordinated way? Could open source/volunteer enthusiasm be rallied around the problem in a way that's constructive? Perhaps there could be prizes or medals given out as milestones towards the desired systems are reached, to add a little incentive.

There are ways in which the state of the art could be advanced in a positive direction, with perhaps fewer side tracks and dead ends, but alas SIAI doesn't seem to be interested in this sort of stuff. Instead they're spending time producing very questionable graphs for armchair singularitarians, or talking about fallacious AI risks. It all seems like such a wasted opportunity.

Operating system fanboy matrix

Are webcams racist?

Probably they are.



The built in face detection on webcams is based upon coloured blob detection. Typically they're looking for a blob within the image which has a relatively high red component indicative of the flesh tone of skin. The algorithm may be looking for pixels having a chrominance value within certain limits, or satisfying a simple test such as:

2R - G - B > Threshold

These pixels can then be grouped into blobs, and the largest blob may be selected as a possible face area.

For most skin tones this kind of strategy will work, but the above video demonstrates the limitations face detection based only upon colour. For people with very dark skin there is obviously insufficient red component for the blob finder to work. However, this isn't necessarily a fundamental obstacle, since a modified version of the algorithm could just look for relatively dark or light blobs (a light face against a dark background or vice versa). Computing a luminance histogram then binarising the image and grouping pixels as before would probably do the job, and be sufficiently simple to implement as part of the camera hardware.

Within the next five or ten years I think that the way faces are detected in this sort of situation is going to change, and we're going to be seeing many more of the infrared time of flight devices similar to the 3DV Zcam which will be part of future games consoles. Using those devices you don't need to rely upon colour to locate faces, which is unreliable and depends upon ambient lighting, but can detect people based upon their shape. With shape based detection it doesn't matter what race you are, and the face detection should be much more reliable.

Omnidirectional stereo vision

Here I'm doing an experiment with combining omnidirectional and stereo vision. A single camera with a wide angle lens is mounted underneath five hemispherical mirrors (christmas tree decorations sawn in half). The obvious disadvantage is the small resolution for each mirror, but the advantages are:

• You only need a single camera
• There are no camera synchronisation issues - all the mirror images are guaranteed to be synchronised
• The materials needed to make it are extremely cheap - you could make this for less than £50.
• Very wide field of view conducive to mobile robot navigation problems
• Multiple triangulations and multiple baselines permits better disambiguation

The geometry is more complicated than for traditional stereo vision, but it shouldn't be intractable and I'll be able to create a lookup table between 3D voxels and pixel coordinates.



A similar vision system in the natural world.

Kinetic architecture

There are many possibilities with this sort of thing. If this was an office building it would be nice to link the external appearance to things going on inside. So you could maybe data mine the emails of the workers and change the building appearance based upon how happy or sad they are, or how often people are moving around or what type of information workers are entering or searching for. The building becomes more like an organism and less of a static fixture.

SIFT demo

I found a nice SIFT demo on the web site of Rob Hess. Here is an example of features being matched between two images.


SIFT is a commonly used feature detecting algorithm within the computer vision research literature, and from this demo seems reasonably fast. It's main disadvantage is restrictive licensing and the dreaded software patents. If you read the license file which comes with the demo it says:

"Any commercial use or any redistribution of this software requires a license from the University of British Columbia."

Followed by the patent details. One thing which is amusing is the statement "Method and apparatus..." - as if this isn't a pure software patent and you need some kind of special apparatus other than a general purpose computer to implement it.

So the upshot is that even if I thought this was cool and wanted to use it I wouldn't be able to, other than for purely private research which was not redistributed in any way, without incurring the potential of being sued by someone, or thrown off of the internet if Mandleson gets his way. Also in this demo I'm not sure that the way in which both GPL code and code under a restrictive license are distributed together is strictly GPL compliant, but as they always say I'm not a lawyer.

If you wish to try the demo you may need to install the GNU scientific libraries and also doxygen before running make. On Ubuntu within the package manager search for gsl and doxygen.

Shopping with robots

This is ok as a media stunt, but probably not as a practical use of robots. It's an unimaginative future in which you simply replace things that people do today with automated versions, without realising that automation itself transforms the ways in which things can be done. In this future you would also have a robotic checkout assistant.

Assuming that for the foreseeable future people will still want to do grocery shopping in person (i.e. not online) what would be an intelligent way to more highly automate the process? Instead of a humanoid with a shopping basket trying to make conversation with you, why not just automate the shopping trolley itself. When you walk into the store a shopping trolley automatically navigates to where you are and follows you at some discreet distance. As you put items into the trolley they're automatically scanned and the total value displayed on a small screen. For familiarity with the way things worked in the past maybe it also makes a checkout-like beep sound as it adds a new item to the total. When you're done the screen asks if you want to charge the items to your account, and if so the transaction is done entirely electronically either using a card or a mobile phone with no traditional checkout process. You can then select either to take the goods to your car or have them delivered separately. If necessary the trolley follows you into the carpark, then when it's empty it navigates itself back into the store to wait for another customer.

Probably supermarket executives have already thought about this kind of scenario, and are just waiting for the automation technology to be cheap enough and reliable enough for general use. In terms of staffing you might need a security guard to discourage looters, but warehousing, shelf stacking and loading of trolley contents into dispatch vehicles could also be automated.

45 minutes

I think in the US it was a different story, but in the UK the entirety of the reason for going to war in Iraq in 2003 was the "current and serious" threat to national security posed by weapons of mass destruction developed by the Iraqi regime. In the forward to the september 2002 dossier, entitled Iraq's Weapons of Mass Destruction, Tony Blair made a statement which was to remain the subject of speculation and debate for years afterwards.

"Saddam has used chemical weapons, not only against an enemy state, but against his own people. Intelligence reports make clear that he sees the building up of his WMD capability, and the belief overseas that he would use these weapons, as vital to his strategic interests, and in particular his goal of regional domination. And the document discloses that his military planning allows for some of the WMD to be ready within 45 minutes of an order to use them."

The implication, laid out in greater detail in the rest of the document, was that Iraq could use the long range ballistic missiles which it was developing to deliver WMDs into the heart of Europe, and perhaps even to Britain.

Now it transpires that the 45 minute claim:

"had originated from an emigre taxi driver on the Iraqi-Jordanian border, who had remembered an overheard conversation in the back of his cab"

When I read that I laughed out loud, recalling the earnestness and apparent sincerity with which many politicians had cited the 45 minute claim as the most urgent reason for which military intervention was required. That Britain had gone to war on such a farcically flimsy basis just seems totally absurd, and shows just how easy it is to mislead people. But really it's no laughing matter, since many people have been killed, injured and tortured as a consequence.

Warming in the most industrialised nations

Plotting temperature history for the G8 nations using the Global Historical Climatology Network data we get the following.



There's a spread of warming and cooling trends, and if we then take the average of all these graphs the result looks like this.



At first this seems exciting, because it appears to show some increase in temperature in the late 19th century, which could indicate an anthropogenic origin. However, comparing with the first graph we can see that the low temperatures up to 1870 are the result of missing data for Japan. Missing data, if there's enough of it, can produce phantom increases or decreases in the global average trend.

To get more reliable averages more data is needed, so I expanded the scope to the G20 nations. I'm picking the G8 and G20 because these are highly industrialized nations which could be expected to be producing large quantities of atmospheric pollution.



Then calculating the average.



Again the initial low temperatures are not due to anything dramatic or sinister, but are simply an artifact of missing data. You can also see that the downward blip around 1993 is the result of missing data which makes the average value slightly smaller. But overall this gives a very convincing linear warming trend, indicating a rise of about two degrees per century.

So at this point I'm sitting smugly thinking to myself that I've identified the signal behind the noise. But wait - could this two degree slope be purely an artifact of missing data? We already know from the pre-1870 period, and the 1993 blip that missing data plays havoc with the averages. To investigate this possibility I removed three nations from the list which do not have a very complete temperature record over this period - Korea, Turkey and Saudi Arabia (sorry guys), then re-plotted the graphs. The resulting plots look cleaner than before.


When we again take an average of these, something unexpected occurs.



The nice slope, which fitted to a line so satisfyingly, almost completely disappears. Removing pre-1880 temperatures the graph looks like this.



Bear in mind that these are the majority of the world's most industrialized nations, including the Russia, China, USA and the manufacturing giants of Europe. They should be producing an increasing volume of atmospheric pollution over this time period.

Conjecture: could the global average temperature slope be the result of significant missing temperature data, with the relatively recent introduction of temperatures for countries closer to the equator (eg. within Africa) producing what appears to be a warming trend?

To find out I extracted temperature histories from the entire data set which were more than 98% complete. This means that for these countries there has been an almost unbroken series of temperature measurements, so apparent structure in the resulting overall average can't be ascribed to missing data.

30 countries fit this criteria. This is a larger number than the G20, so the overall average should be more reliable.



From this the two degree per century linear slope once again emerges triumphant.


Even if this hasn't convinced anyone else, I've managed to satisfy myself using the raw data that global warming is occurring, that it's following a linear trend which likely precedes the industrial revolution, and that the temperature graphs typically shown in the media and on the Met Office web site are significantly distorted by continuity gaps in the temperature data such that what appear to be sudden jumps or dips are actually just phantoms.

Another thing which has become apparent from these experiments is that selectively adding or removing temperature series can alter the slope of the overall average temperature graph, so by inclusion, omission or perhaps attempted interpolation a variety of climate theories could be supported. For example, if I included temperatures for Canada, Russia and China in the above graph, which are all following a cooling trend, this would tend to pull the slope down towards a flatter profile. This should be borne in mind when assessing the likelihood of climate change predictions.

Global warming not occuring in the UK

I've also done some independent analysis on historical temperature data from the UK. This data was gathered by the Met Office from the mid 19th century onwards. Again I've gone through a similar procedure, plotting both monthly and annual averages.



Since no monthly mean temperature is given in the data I have taken an average of the minimum and maximum values (Tmax+Tmin)/2, shown as the blue line below. The minimum and maximum values are however taken directly from the raw data as averages over all weather stations. Also the stations seem reasonably well dispersed over the whole of the UK.



The results here also seem clear. On average the UK has neither warmed nor cooled over the last 160 years, despite a substantial rise in industrial activity and the expansion of cities, rail networks, roads and air travel.

This is odd because it contradicts the previous global calculations, and I had expected to see a steady linear increase in average temperature. However, perhaps it could be argued that the UK is a special case, and it's certainly only a small percentage of the surface of the globe.

Despite the lack of temperature movement in either direction, this also supports the notion that global warming is either non-anthropogenic or only very weakly anthropogenic (such that the signal is buried beneath larger random fluctuations). If there were a significant human component I would expect to see the temperature track the 19th century rise and late 20th century fall of British industrial output (particularly due to the burning of coal).

If average temperature is indeed linked to carbon dioxide emissions it's possible that any successful future attempts to reduce pollution may actually result in a slight overall cooling of UK temperatures.

Bizarrely, the graphs shown within Climate change - the facts seem to contradict the Met Office's own data. They're showing the so-called "hockey stick" temperature change graph, which I havn't been able to confirm/reproduce using the GHCN data.

Cross checking the GHCN and Met Office data

Because the UK doesn't seem to fit with the global warming trend I checked the data from the Met Office against the data from the Global Historical Climatology Network (GHCN), just to make sure that the two aren't out of step. Fortunately the GHCN data is nicely labeled with country codes, so it's quite straightforward to extract data only for the UK.

The resulting graphs show that although the match isn't perfect it's sufficiently close that we can say that this is probably very similar data, and I expect that most or all of the temperature data for the UK within GHCN comes from the Met Office.



So, in conclusion it seems that the UK's temperature history is anomalous compared to the global averages. This is not so bad, because you might reasonably expect there to be variation when looking at small regions in isolation. Still it is conspicuous that signs of of industrial activity don't show up, especially since the UK must have been one of the first countries to industrialize. There were problems with smog in many of the large cities in England right up until the mid part of the 20th century, and use of domestic coal fires for heating has only decreased within the last few decades.

Comparing the UK to the rest of the world

Plotting the global average temperatures against only those for the UK gives the following graph. From the mid 19th century onwards the UK is increasingly diverging from the global average (or vice versa, depending upon which way you look at it).

Murdoch's paywalls are an opportunity for bloggers

Rupert Murdoch's decision to add paywalls to his online news sites will probably further accelerate the demise of traditional news reporting. As a typical news reader, if I see a paywall, or even just an irksome login process which requires me to generate an account and enter a lot of details, I usually just go elsewhere. I expect that I'm not dissimilar from most internet users in that regard. I'd only consider going through the paywall if there was some specific reason to do so, such as if I was researching a particular historical event, but most news reading is not this focused an activity.

The decision could be a golden opportunity for both news bloggers, and also for Google. If the conventional news sites wall themselves off then blog traffic is bound to rise. Since many blogs also use ads supplied by Google, both Google and news bloggers stand to profit. Google can facilitate this transition even more effectively by limiting access to conventional news sites.

If the walls go up then this might be a phase change in how modern journalism works, where a few bloggers can actually begin to make enough income to be able to support activities such as investigative journalism - going out into the field to make news reports, rather than merely acting as commentators.

A platform for personal robotics

A presentation about the Willow Garage ROS software and PR2 robot. This ticks many of the boxes in terms of moving things towards the emergence of a successful industry, such as:

• The basic system is open source
• Potential for commercial applications
• Attempting to build something which can become a standard
• Can be made to work on different hardware platforms
• Removing the need to reinvent hardware and software from scratch

Although the PR2 is impressive, it looks like only ever being a research robot. I'd estimate that the sensors alone are at least $5000 (probably more like double that), and the engineering looks fairly heavy duty, similar to an industrial robot. I have doubts about the use of scanning laser range finders for personal robotics, both in terms of cost and safety, but eventually those sensors might be replaced either by using additional cameras or time of flight infrared sensors.

The main purpose of a research robot like PR2 should be to demonstrate the feasibility of being able to carry out different tasks, together with robustness of the perception and planning software under varying conditions. If that can be done, then other companies can look into re-engineering the hardware to cut down the cost and make it suitable for target markets.

Also, as mentioned in the presentation, simply gathering logs of sensor data will be very useful to help improve the development of algorithms. Realistic data sets allowing the performance of different algorithms to be fairly compared have been historically lacking in robotics, and their existence will transform the field into more of a science than it is currently.

Overall, this is definitely going in the right direction. At least it's not robot bees, or yet another photogenic yet useless humanoid.