We wrote a 1-year update which includes the projects we used the Manifund funding for: https://www.apolloresearch.ai/blog/the-first-year-of-apollo-research

Thanks again for the support!

Main points in favor of this grant

I am quite excited about deception evaluations (https://www.lesswrong.com/posts/Km9sHjHTsBdbgwKyi/monitoring-for-deceptive-alignment), transparency and interpretability (https://www.lesswrong.com/posts/nbq2bWLcYmSGup9aF/a-transparency-and-interpretability-tech-tree), and especially the combination of the two (https://www.lesswrong.com/posts/uqAdqrvxqGqeBHjTP/towards-understanding-based-safety-evaluations). If I were crafting my ideal agenda for a new alignment org, it would be pretty close to what Apollo has settled on. Additionally, I mentored Marius, who's one of the co-founders, and I have confidence that he understands what needs to be done for the agenda they're tackling and has the competence to give it a real attempt. I've also met Lee and feel similarly about him.

Donor's main reservations

My main reservations are:

1. It's plausible that Apollo is scaling too quickly. I don't know exactly how many people they've hired so far or plan to hire, but I do think that they should be careful not to overextend themselves and expand too rapidly. I do want Apollo to be well-funded, but I am somewhat wary of that resulting in them expanding their headcount too quickly.

2. As Apollo is a small lab, it might be quite difficult for them to get access to state-of-the-art models, which I think would be likely to slow down their agenda substantially. I'd be worried especially if Apollo was trading off against people going to work directly on safety at large labs (OAI, Anthropic, GDM) where large model access is more available. Though this would also be mitigated substantially if Apollo was able to find a way to work with labs to get approval to use their models for research purposes externally, and I do not know if that will happen or not.

Process for deciding amount

I decided on my $100k amount in conjunction with Tristan Hume, so that we would be together granting $300k. Both of us were excited about Apollo, but Tristan was more relatively excited about Apollo compared to other grants, so he decided to go in for the larger amount. I think $300k is a reasonable amount for Apollo to be able to spin up initial operations, ideally in conjunction with support from other funders as well.

Conflicts of interest

Marius was a mentee of mine in the SERI MATS program.

I'm very excited about Apollo based on a combination of the track record of it's founding employees and the research agenda they've articulated.

[Marius](https://www.alignmentforum.org/posts/KzwB4ovzrZ8DYWgpw/more-findings-on-memorization-and-double-descent) and [Lee](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition) have published work that's [significantly contributed to Anthropic's work on dictionary learning](https://transformer-circuits.pub/2023/may-update/index.html). I've also met both Marius and Lee and have confidence in them to do a good job with Apollo.

Additionally, I'm very much a fan of alignment and dangerous capability evals as an area of research and think there's lots of room for more people to work on them.

In terms of cost-effectiveness I like these research areas because they're ones I think are very tractable to approach from outside a major lab in a helpful way, while not taking large amounts of compute. I also think Apollo existing in London will allow them to hire underutilized talent that would have trouble getting a U.S. visa.

I think Apollo is going to be very hard to beat for a few reasons and I would have written up the grant if Marius didn't.

1. They are a very very talented team. I think several people involved could be working at top AI labs.

2. They are focused on a tractable and "most important problem" of deception in AI systems.

3. They seem well positioned to be a potential centre for an AI safety org in Europe

4. They could grow to absorb a lot more funding, effectively. A key barrier to EA funds right now is productively spending lots of money. Seems possible that they could absorb >20M/year in 4 years time

5. I think the people here are very "alignment-pilled"

6. I think I am a good judge of talent/character and Marius passes the bar.

7. I have a preference that they become a non-profit vs for-profit company that sells auditing and red-teaming. I think funding in the first few years will be pivotal for this.

Reservation:

1. They seem quite expensive on a per person basis compared to other projects Manifund has been funding. That said, there is going to be a lot of "bang per person here". Marius explained to me that they are competing with large AI labs and are already a significant salary cut. I would much rather see someone working within Apollo than doing independent research. I think we should be quite weary of funding independent researchers until orgs like Apollo are fully funded, with rare exceptions.

2. I still don't get why SFF and OP can't just fully fund them. My best guess here is that they are terrified of seeding another "AI accelerator" or wanting to save their cash for other things and thus allow others to donate in their place.

3. I am slightly worried that there isn't a good way to coordinate across the community of "how much they should receive". From the inside, Apollo wants to raise as much as possible. I don't think that is optimal at a movement level and this leads to them spending a lot of time fundraising so they can have more money. There probably is a funding level that Apollo shouldn't exceed in it's first year though I don't know what that number is.

I would bet that if we reviewed Apollo 1.5-2 years down the line, it will outperform a majority of grants on a per dollar basis (very hard to operationalize this though).

My biggest concern is that scaling the org too fast could be detrimental, and I have no idea how to judge it besides deferring to people with similar experiences

If scaling Apollo's output is, in fact, limited only by cash - this seems like an amazing grant. At least as good as funding most of the independent folks.

Thanks for posting this application! I've heard almost universal praise for Apollo, with multiple regrantors expressing strong enthusiasm. I think it's overdetermined that we'll end up funding this, and it's a bit of a question of "how much?"

I'm going to play devil's advocate for a bit here, listing reasons I could imagine our regrantors deciding not to fund this to the full ask:

1. I expect Apollo to have received a lot of funding already and to soon receive further funding from other sources, given widespread enthusiasm and competent fundraising operations. In particular, I would expect Lightspeed/SFF to fund them as well. (@apollo, I'd love to know if you could publicly list at least the total amount raised to date, and any donors who'd be open to being listed; we're big believers in financial transparency at Manifold/Manifund)

2. The comparative advantage of Manifund regranting (among the wider EA funding ecosystem) might lie in smaller dollar grants, to individuals and newly funded orgs. Perhaps regrantors should aim to be the "first check in" or "pre-seed funding" for many projects?

3. I don't know if Apollo can productively spend all that money; it can be hard to find good people to hire, harder yet to manage them all well? (Though this is a heuristic from tech startup land, I'm less sure if it's true for research labs).

By the way @apollo I edited your profile to include your name and a username that's not a long uuid—there was a bug in the sign-in flow at the time that you made your account so it may not have prompted you to do that. Feel free to change those, I just thought it looked bad to have them blank.