Actually, not generated by AI. It was largely handwritten.. but since you found this phrasing objectionable, I removed it.

Agreed. While “Cost per task” is a much better unit than cost per token, it’s still an intermediate metric.

The metric we actually care about is closer to "cost per accepted outcome" which would roughly be "total inference + retries + failures + the human effort required to verify, correct, or resume the work."

The first few are easy to instrument.

But "human effort verification" and “accepted” are much harder to define consistently across different agent workflows.

For our code review product we have a decent proxy because we can measure user reactions and whether findings are actionable and if it's merged/accepted.

For more open-ended agent tasks, I don’t think the industry has a particularly good answer yet... I would love a point if there is one?

The cheaper model isn’t actually cheaper if a human has to spend the savings babysitting it...

That's something I have been working on because I want to decouple work from the agent. In my opinion, work should be treated as a first-class object.

A long running work could have many attempts, failures and retries should be aggregated into the same result.

An acceptance of a work result should not be inferred as "the model said it is done", but be recorded as a human approval.

To calculate the cost of the whole work, the outcome and verified result of each attempt must all be saved.

I don't have a final method to measure human effort, right now I start by recording recovery, review and explicit acceptance events, the at least we have data to analyze.

I'm exploring this in a work-runtime project: https://github.com/kungfu-systems/kungfu

We built something similar an A/B testing framework that measures Unblocked's impact on real AI coding agents.

It spawns agent CLIs (Claude Code, Codex, Cursor, GitHub Copilot) with and without Unblocked's MCP server attached, then statistically compares the results: https://github.com/unblocked/unblocked-harness-compare

We likewise measured token savings, (wall clock) time, # tool calls, and # turns.

Unblocked CEO here, happy to answer any questions!

As one of the Founders of buddybuild, I'd like to perhaps also provide another perspective on this specific thread.

We certainly don't think of buddybuild as a closed ecosystem.

In fact, our users are able to customize buddybuild infinitely to meet the needs of their workflow (details can be found here: https://www.buddybuild.com/blog/customizing-the-build-proces...)

Further, while buddybuild is favored by thousands of because it can can auto-provision iOS devices for you on the fly. Meaning, you never have to deal with provisioning profiles and UDIDs ever again.

We also care deeply about making sure that teams can involve and engage their end users as part of the development process. One of the reasons we built buddybuild is because of the frustration sending builds to our users... AND then never hearing back from them. Phones are great devices for "consuming content", but it became obvious that the physical form factor of the device tends to result in infrequent feedback which was often ambiguous.

So, we build an SDK that gives you high quality and actionable bug reports from beta users. With a simple screenshot, testers can file bug reports that include graphically annotated screenshots, feedback notes and their device metadata.

We also have a feature called "Instant Replay", which lets you watch a video of the exact steps required to reproduce a crash. Instant Replay works in conjunction with our Crash Reporting solution to offer you both the exact lines of code that caused a crash as well as the exact sequence of events that lead up to it (https://www.buddybuild.com/blog/introducing-instant-replay)

Thousands of teams with fairly complex needs use buddybuild every single day. They teams like Slack, Meetup, FireFox, Wordpress and others..

You can read why they made the choice to use buddybuild here: http://www.buddybuild.com/customers

I sure hope so

At buddybuild, we see tens of thousands of mobile projects from mobile dev teams.

Of the teams looking for cross platform solutions (versus native iOS or Android), Xamarin comes up on occasion.

However, React Native tends to be FAR more common as teams are looking for solutions for "write once, run on many"

Could it be that your service is more suited for people working with React?

In a slightly related note, I find it hilarious that people are using a framework made by a company behind what is probably the most hated mobile app ever.

Ah, I should clarify and provide context.

We're a mobile focused continuous integration and deployment solution.

When deciding which platforms to support, we looked at current adoption for Native iOS and Android (this was no brainer), Hybrid (PhoneGap, Ionic, Cordova) and "emerging" platforms (React Native, Xamarin).

Ultimately, we decided to prioritize the others first. CI/CD service is applicable to any mobile development team.. but to your point, buddybuild is better suited to those other platforms, as those are the only ones we support.

In our experience (which also factors inbound requests), Xamarin is still relatively nascent as compared to the others.

That said, you can should expect to see Xamarin support soon :)

Also, fwiw, Microsoft purchased Xamarin only recently.

But this puts your first statement in a completely different light ;)

I should also note that React Native was announced in 2015 while Xamarin was announced in 2011 (with Miguel's work on Mono being much older). Can you then call Xamarin an "emerging" platform?

(disclaimer: I don't use either platform)

As GitHub queues up webhooks, we're bracing for impact at buddybuild! :)

You shouldn't notice a webhook deluge because the site isn't generating events. I'm watching our webhook services though and will let you know if that changes.

Hi Kyle!

It looks like webhooks are wedged.. no?

Everything should be A-OK now. If not, hit up github.com/contact :)

But no one can do anything to generate new webhooks.

pushes have been working for the past 10-15 minutes... pulls for 5-10 minutes.