It's kind of sad how SIAI seems to have become obsessed with "AI risks" and human extinction. Perhaps they always were from the beginning, but it's just my perception of them that was at fault. There's certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.
The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.
The sorts of thing which SIAI ought to be focusing its efforts upon are answering questions such as:
- Do we know how to build an AI? Why have we failed so frequently in the past?
- What sort of uses do we want AI put to, or not put to?
- Can we formulate a set of requirements and benchmark tests which are useful to researchers?
- Can we identify a set of key problems which need to be overcome so that we can make further progress?
- Can we advocate for research to be done on the problems which will yield the greatest benefit? If not, can we raise funds to do this research ourselves?
- Can we discourage research which is frivolous or wasteful of funds, and does not contribute anything significant to the sum of knowledge (eg. Minsky's "little robots" criticism/robot "bees").
There are ways in which the state of the art could be advanced in a positive direction, with perhaps fewer side tracks and dead ends, but alas SIAI doesn't seem to be interested in this sort of stuff. Instead they're spending time producing very questionable graphs for armchair singularitarians, or talking about fallacious AI risks. It all seems like such a wasted opportunity.
29 comments:
FWIW, a while ago, I looked at Stefan Pernar's material. My conclusion was that he was probably more confused about the issues he was discussing than those at the SIAI are.
They probably won't like me saying it, but the SIAI seem to be primarily a marketing organisation. Before being too hard on them, consider that they *have* been quite successful at raising cash.
This seems like an impressive vapourware marketing feat to me.
If someone told me ten years ago that you could run a machine intelligence organisation on the basis of donations, I would probably have snorted derisively:
"Yes, right. Next you will tell me that donations will fund Intel's next fab as well."
Hah! Yes, well, some of the behind the scenes communications that I've had with SIAI people would tend to support your hypothesis that it's primarily a marketing operation.
Perhaps the one redeeming feature of SIAI has been their support of OpenCog. I'm not exactly sure what the relationship between OpenCog and SIAI is, or whether its developers are directly funded by SIAI donations. Thus far OpenCog hasn't done anything very noteworthy, but it does still look like an interesting project and has at least managed to gain the support of Google as part of GSoC.
What probability does the poster place upon the human race going extinct though a violent and non-consensual destruction as a result of human brain emulations or advanced software AI?
We don't yet have WBEs or sufficiently advanced AIs in order to be able to meaningfully characterise their behavior. So trying to assign probabilities to the behaviors of these systems is pretty much an exercise in futility at this point.
On WBEs we may be able to get a better idea at least of the requirements for emulation once we have a structural model for an entire brain.
@Bob: Presumably, in order to object to mitigating a risk, you have to argue that the expected loss from that risk is small, which means that either you think that the probability of the risk occurring is small, or you think that the likely effects of the risk are not consequential.
Posing AI or WBE as a risky business is I think a somewhat grandiose notion, more informed by science fiction than by any real knowledge of the systems involved. Certainly if you've ever done any AI development you'll be aware of just how incredibly modest and limited our current efforts are.
At the moment we're just not far enough down the line of development on these technologies to be able to say authoritatively, or even in more vague probabilistic terms, how they will behave in practice. We can make assumptions, but these aren't informed ones and tend to meander towards biases based upon accounts from popular fiction, which mostly originate from certain fears and prejudices promulgated within western culture.
Our Uncertain Future project is pioneering probabilistic futurism in AI and WBE studies, and has received thumbs-up from several academics including Bela Nagy, who manages the Santa Fe Institute Performance Curve Database.
A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider "nice" or "common sense" is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values. See the Fun Theory sequence on Less Wrong, for instance.
SIAI believes that AGI is an extremely difficult endeavor and deserves far more theory-level work than programming in the dark or working towards narrow AI tasks that drain away our attention at the expense of the Singularity itself.
Basically, if you consider an intelligence explosion plausible, SIAI's activities make sense, and if you don't, they don't. It's not a matter of marketing, just disagreement on which tasks are the most important for humanity to face right now. We consider clarifying decision theory and creating a reflective decision theory to be a major priority, for instance, and spend time on that accordingly.
Someone asked I'm not exactly sure what the relationship between OpenCog and SIAI is, or whether its developers are directly funded by SIAI donations.
The answer is: In 2008 and early 2009, SIAI funded OpenCog to the amount of roughly US$125K, which paid for much the initial OpenCog software development (much of which involved porting some Novamente LLC code into the OSS OpenCog framework, with suitable modification).
Since around March 2009, SIAI donations have not been used to fund OpenCog R&D, due to SIAI deciding to narrow its focus onto other things.
But SIAI is still conceptually supportive of OpenCog and provides the OpenCog project some administrative infrastructure; and the future relationship between SIAI/OpenCog could take a variety of directions.
Calling SIAI's non-OpenCog work "vapourware" as someone did in this thread, isn't really fair. A lot of SIAI's work is frankly and intentionally theoretical, the goal being roughly to come to a conceptual and/or mathematical understanding of some important issues regarding the future of AI. No one accuses Nick Bostrom's Future of Humanity Institute at Cambridge University of being "vapourware" oriented. FHI is pursuing theoretical work about future technology and society from one perspective, and SIAI is (among other things) doing so from a different perspective.
Personally I'm more focused on AGI technology development, but I think there's also significant value in the theoretical and philosophical issues that others within SIAI are pursuing.
(For those who don't know, I'm the leader of the OpenCog project; and I'm also the Director of Research of the SIAI -- although in practice I'm not directly involved with all of SIAI's work, only some of it. SIAI is a distributed and agile organization.)
"Do we know how to build an AI? Why have we failed so frequently in the past?"
The answer to these questions are very clearly "no", and "because it's a really hard problem", respectively.
"What kinds of AI technology can be used to assist the elderly? "
This is not the kind of AI SIAI is trying to build.
http://yudkowsky.net/singularity/power
"Could open source/volunteer enthusiasm be rallied around the problem in a way that's constructive?"
Not really. Adding lots of not-very-well-trained volunteers to a new and confusing field generally just leads to chaos.
"This seems like an impressive vapourware marketing feat to me."
See http://singinst.org/accomplishments2009 for a list of 2009 SIAI accomplishments.
If we currently don't know how to build an AI then the question becomes one of how do we find out how to do it. What are the requirements? Can we devise benchmarks to measure performance against the requirements, and so on. This is an area where theorists can make useful and original contributions.
Though Yudkowsky's comments about the power of intelligence may be interesting they provide little real enlightenment for anyone trying to build an intelligent system, nor any guidelines on how to use related technologies in a manner concordant with progressive values.
It's good to hear that SIAI did directly fund OpenCog development. Funding projects which display ambition and a long term perspective, which would be unlikely to funded through more traditional channels, should I think be the primary role of organisations like SIAI. I hope that they resume funding this and perhaps other comparable projects as soon as it becomes practicably possible. However, not using the Singularity Research Challenge as a vehicle to investigate those apsects of OpenCog which require further work seems like a wasted opportunity to me.
Re: vapourware:
"The Singularity Institute was founded on the theory that in order to get a Friendly artificial intelligence, someone has got to build one. So, there: we’re just going to have an organization whose mission is: build a Friendly AI. That’s us."
- http://www.youtube.com/watch?v=0A9pGhwQbS0
That's Eliezer Yudkowsky, founder of the institute.
Until the proposed "friendly artificial intelligence" exists, "vapourware" seems like a fairly appropriate term to me.
The FHI is rather different - since it doesn't have the same mission.
"...SIAI seems to have become obsessed with "AI risks" and human extinction. Perhaps they always were from the beginning, ..."
Ever since I first encountered Yudkowsky in early 2000 both he and SIAI have been primarily concerned with AI risk. So no, at least as far as I can tell, this is not new.
Tim,
Until the proposed "friendly artificial intelligence" exists, "vapourware" seems like a fairly appropriate term to me.
is an inaccurate use of the term "vapourware", as I have commonly heard it and as defined at
http://en.wikipedia.org/wiki/Vaporware
It seems to me that Eliezer and his close colleagues at SIAI have been quite careful about not promising delivery of any Friendly AI with any particular properties at any particular time. So they really have not offered the world any "vaporware." They are proceeding with a research project -- on which they likely have different estimates of the success odds and the value than you do -- but that's very different than hawking "vaporware."
I myself have been more aggressive in my timing estimates and product delivery claims than Eliezer's group. But I have also never claimed to have a complete human-level AGI, nor a firm delivery date for one. I've stated that with adequate funding I think I have strong odds of creating a beneficial human-level (or greater) AGI within less than a decade. But the "with adequate funding" is a major caveat since my project funding now (though fortunately nonzero) is not adequate. And even in the adequate funding scenario, I don't claim any certainty -- there is research to be done and research is always uncertain to some level.
There's no vaporware at SIAI or OpenCog. There are ambitious research projects, on which different attitudes and success-odds estimates are quite reasonable, given the level of uncertainty involved.
I'm a big believer in the potential of OpenCog ... and also in the value of the sorts of investigations Eliezer and his close collaborators are doing. If you're not that's fine -- but then you should just say you don't think those are the most productive lines of research to be following, rather than throwing around terms like "vaporware".
So IMO Motters' title Singularity research challenge: funding the wrong stuff is in itself perfectly fair.... Motters has a different intuition of what research is most worth funding than SIAI currently does. Eliezer and I also have different intuitions about which directions are most worth attention right now; and I think that's fine....
IMO, those of us who belong to the tiny percentage of the human race that recognizes the potential of AGI and the Singularity and all that, should try to avoid "infighting". Factionalism doesn't get us anywhere. At least, unlike most of the human race, all of us in the futurist community are trying to push in roughly the same direction ;-) ....
Having this kind of argument is a BIG, BIG, BIG improvement over the 1970s when I first started thinking seriously about this stuff -- back then it seemed hard to find anybody else on the planet who didn't consider the idea of transcending the human condition to be absolute lunacy!!!
Ben, it seems as though this is hair-splitting over the minutiae of a particular definition. In support of my usage, I would cite:
"a product, esp. software, that is promoted or marketed while it is still in development and that may never be produced."
- http://dictionary.reference.com/browse/Vaporware>
"An advertised product, often computer software, whose launch has not happened yet and might or might not ever happen."
- http://en.wiktionary.org/wiki/Vapourware>
"A term used to describe a piece of software that doesn't exist, is suspected not to exist, or is incapable of performing as promised."
- http://www.scotsmist.co.uk/glossary_v.html>
Whether it constitutes vapourware or not is a somewhat separate theological debate.
From the video Yudkowsky does make quite clear that his intention is to build a friendly AI, rather than merely write floridly on the topic. He wants to "figure out what to do, and do it".
Things like trying to figure out what human reflectivity is and then finding ways to instantiate similar properties into a machine would appear to be worthy research topics, but at least to me, it's unclear how the challenge grant proposals fit with this kind of ambition.
Tim,
I guess my point about "vaporware" was that Eli has actually been very careful not to make promises about what he can deliver or when he can deliver it.
I'm less careful than him about that (partly because my project is in the engineering phase and his isn't, and partly because of personality and philosophy differences), and other folks in the history of AI have been far less careful than me.
"Vaporware" is about products, and SIAI is about research.
Arguments about research funding priorities are very much appropriate and I have somewhat different view than either Motters or Eliezer.
Although, I applaud Motters' call for increased funding for OpenCog and other similar projects. hear, hear!!
I would also add that Eliezer's statements N years back don't necessarily represent either his current views nor the current position of SIAI. His views on various relevant topics have changed significantly over time, perhaps more so than for most thinkers in the field...
But I will leave it to him or others closer to his own particular project, to give more detail in this direction if they wish.
Oh also, about "Thus far OpenCog hasn't done anything very noteworthy,"
...
OpenCog has a big goal (human-level, roughly human-like beneficial AGI) and a fairly detailed design for achieving it ... and we're proceeding systematically with the plan.
It's going well, but there's a long path till completion.
We have intentionally not diverted substantial energy from our plan to produce demos or interim results that will make people say "Oh! That's noteworthy!"
DARPA Grand Challenge and Deep Blue were noteworthy but not so directly useful for AGI. My own bioinformatics work has been in some cases noteworthy -- but also not so directly useful for AGI. Also, a human baby doesn't do much that's noteworthy -- if we didn't know they were going to grow up, we'd find them pretty useless.
And about "SIAI is primarily a marketing organization"
No. SIAI has put on the Singularity Summits, which IMO have had a large positive impact on the world via spreading the Singularity meme in important social circles. And SIAI has funded OpenCog a little bit; and has funded Eliezer's research on the foundations of AGI and ethics. How does this make it a "marketing organization"?
I don't mean to sound overly defensive.... Constructive criticism is welcome but not all of the critique in this thread has a constructive feel to it.
Re: "a large positive impact on the world via spreading the Singularity meme in important social circles"
Ben, that sort of thing is what I was calling marketing. The summit also has numerous calls for donations. Some lectures are devoted entirely to stimulating contributions.
Re: "How does this make it a "marketing organization"?"
I didn't say "entirely a marketing organization" - I said "primarily a marketing organization". I am not privvy to the SIAI accounting - but essentially they have no saleable products, and rely entirely on marketing and advertising for their revenue. It is hardly surprising that they do so much marketing.
A look at their current projects and they practically all seem devoted to improving their academic credibility - presumably in the hope of being more credible in the eyes of their sponsors.
Tim,
Again there seems to be some profound confusion between research and product development in your posts.
You say
I am not privvy to the SIAI accounting - but essentially they have no saleable products, and rely entirely on marketing and advertising for their revenue.
A research organization is not typically in the business of producing saleable products.
Some research organizations have big initial endowments (e.g. Perimeter Institute, in physics). Those without, need to do marketing and fundraising to support their research.
Criticizing SIAI's particular choice of research directions, as Motters did in his original post, makes sense to me. In fact SIAI places more focus on AGI risks than I would place, if I were sole SIAI decision-maker rather than one among multiple SIAI stakeholders.
But criticizing them for fundraising to support research, instead of being a business and producing saleable products, is just off-base IMO.
Do you think that research organizations, pursuing understanding and science rather than production of saleable products, should not exist? I know that some radical libertarians do believe something like this, but I don't personally ascribe to that creed. I'm an entrepreneur and value business and the profit motive, but I don't consider it all-powerful...
Ben, I am not *criticising* the emphasis on marketing. I am just pointing it out. A large proportion of the SIAI's visible actions look like marketing to me - including most of their proposed future projects.
Marketing is not something which is intrinsically bad. Rather, it is an essential part of most organisations.
A large proportion of the SIAI's visible actions look like marketing to me -
Hmmm.... I think that's partly a result of Eliezer's choice to keep his more technical research unpublished and private....
In terms of budget, SIAI does not spent the bulk of its $$ on marketing or outreach activities.
I think marketing and outreach activities are just more easily observed than research activities (at least until the latter finally, hopefully, yield dramatic results!!)
I don't agree that SIAI is _primarily_ a marketing org, though I suppose I can see that people would think that if most of what they see is the SIAI's media output. Some people may have said the same thing about the LHC, or probably did about the human genome project or other scientific endeavors not yet complete that also had a marketing component.
Vaporware by technical definition could be applied to many projects, like the human genome project's sequencing softwares, and probably some of the software used with Atlas at the LHC etc. My encounters with the term 'vaporware' have left me with the impression that it means something disparaging about a commercial type of software and among other things quite different from research and development at SIAI or Open Cog (which is not to say that this means you were fishing for this type of impression). I'm not sure if its use here is particularly accurate.
How about call it 'Singularityware' instead. I don't know what it means either, perhaps software in a development stage, where it is milestones fade past the horizon of our understandings. Progression towards a horizon yields more landscape and opportunities for mapping. May need to steer around awkward terrain, even turn back temporarily at times, develop new patterns for moving forward. Until one day either developments cease and evaporate, or we have enough data for takeoff, or on horizon might we might discover we have come full circle to the place where we began. Personally I think it is highly probable that a takeoff will happen.
I admire organisations like SIAI, Open Cog, Foresight Institute etc, it's great that they are putting in fair amounts of effort at communicating technical and scientific knowledge to a wide audience, enabling public debate around surrounding issues, and attracting technical and financial resources to help goad further development.
Kind regards,
Adam Ford
Tim,
Academic credibility is also rather helpful if you want to recruit or collaborate with talent from academia.
Tim Tyler said "I looked at Stefan Pernar's material. My conclusion was that he was probably more confused about the issues he was discussing than those at the SIAI are."
Tim, I would appreciate direct and specific criticism instead of such blatant statements. If you have a specific beef with any of my writings - and particularly with the article linked here - please state them clearly. If you do not want to take the time to follow my line of thought please also do not condemn them based on your personal prejudices. Thanks.
Tim,
The bulk of SIAI consists of researchers and writers. We have almost a dozen employees/interns who do things like providing legal advice, writing papers on inference, philosophy, and futurism, putting together the Singularity Summit, and answering donor questions. Only a very small amount of activity is directed towards marketing. If there is any "marketer" in the org, it is me, and I don't spend much time on explicit SIAI marketing. We are primarily interested in gaining knowledge and investigating a variety of technical and philosophical issues. SIAI's cause is so obviously legitimate to so many people (not many, but a few hundred, which is a good start) that we don't need to market that aggressively to keep ourselves afloat. We have a number of loyal supporters because we have such an excellent cause.
Stephan, I already gave you a fair bit of feedback about what I thought was problematical about your ideas: here and here.
So, despite not spending much time on marketing, you just explained to us how "obviously legitimate" the SIAI's "excellent cause" is. It seems ironic to be publicly promoting the virtues of the organisation in the very post where you are explaining how little marketing it does.
Post a Comment