At some point programmers will be able to spin up a fleet of thousands of world class automated researchers—the resulting progress accomplished by this researcher cohort could be immense. Future poorly controlled self-improvement is probably the biggest issue in AI development and has the least amount of work on thinking about how to control it. This research project will use agent-based models to model the process of automated AI R&D / uncontrolled self-improvement (USI). The goal is to better understand potential runaway AI R&D processes and how to make these processes more controllable. Paul Salmon is a highly-cited expert on complex systems, human factors, and sociotechnical systems. He has also previously published work on how human factors can be used to manage the risks associated with AGI. Basically for this problem we don’t need extreme ML knowledge; instead we need knowledge of how to model systems and their dynamics and ways they can go awry.
The goal is to better understand potential runaway AI R&D processes and how to make them more controllable. It will create basic models of the dynamics and interactions between interventions that attempt to make this process more controllable.
This funding will be used to pay graduate students who will help with the modeling.
Paul Salmon is a professor at the University of the Sunshine Coast. He is a highly-cited expert on complex systems, human factors, and sociotechnical systems (google scholar). He has become interested in AGI risk and has previously published work on how human factors can be used to manage the risks associated with AGI.
There is a chance that modeling runaway AI R&D processes requires a different skillset or is more complicated/intractable than initially expected. If that happens, the model may produce less interesting results or explain less than one might hope.
None that I am aware of.
Rachel Weinberg
10 months ago
Hey Paul, unfortunately Manifund won’t be able to fulfill this grant at this time. An unexpected influx of year-end regrants spent down the total pot of regrantor funding, meaning that we don't have enough left to fund a few of the last projects (like this one).
We’re so sorry if this created false expectations. Best of luck applying for funding elsewhere—hopefully Dan’s enthusiasm and support for your project will be of help, even if he couldn’t give you a grant directly.
Dan Hendrycks
11 months ago
Despite AI capabilities quickly progressing towards human or superhuman level, the dynamics of an intelligence explosion or automated AI R&D haven’t been very thoroughly explored. If an intelligence explosion were to happen, humans would likely quickly lose control of the process, by default, unless precautions had been setup beforehand.
Paul Salmon has previously published highly impactful work in safety engineering and is familiar with the type of systems analysis needed to do this research. Paul is also interested in AI safety, having previously published on the topic of AGI risks.
Whether agent-based models is the right approach remains to be seen.
The amount regranted was comparable to other grants in the field.
I will be helping with this project as well.