David Arbour
[ 00:00:00 ]The problem with pure velocity-based testing is that it’s kind of twofold, right? A lot of times you say you should run 100 times more experiments, but unless you fundamentally change the way you’re analyzing the experiments, unless you have 100 times more users, you haven’t really bought yourself anything but frustration, right?

Matt Wright
[ 00:00:16 ] Hi, everyone. Very special, special episode we’ve got for you today. The team behind Adobe’s upcoming Experiment Accelerator and diving into the whole kind of topic that is the potential of AI-powered experimentation.

Matt Wright
[ 00:00:29 ] Two very special guests we’ve got here. One is Brent Kostak from Adobe, and then David Arbour.

Matt Wright
[ 00:00:36 ] I probably can’t do a good enough job actually introducing all that you guys do. So maybe I’ll just kind of tee this up one after the other. Maybe Brent, you can give a quick introduction to yourself and then Dave, we can set you up to fall.

Brent Kostak
[ 00:00:45 ] Yeah, absolutely. So Matt, thank you and the conversion team for having us on. Brent Kostak, I lead product marketing for optimization experimentation at Adobe, and we are discussing some new AI-first application launches here for Experimentation Accelerator. As some of the upcoming innovations. So happy to be here. Thank you for having us today.

David Arbour
[ 00:01:07 ] Yes, and I’m David. Yes, thanks again. This is super fun. I’m a senior research scientist at Adobe Research where I work on experimentation, causal inference, and AI in the service of the first two. So a lot of that’s been working on this new thing. So I’m excited to talk about it.

Matt Wright
[ 00:01:22 ] Absolutely awesome. It’s great to have you both and get you both at the same time. So thank you so much. I do have one curious question before we dive into this, especially to get the Adobe perspective on it. It’s just like use cases really broadly around experimentation optimization. What, from Adobe’s perspective, are like the top use cases for that? I’ll open to the floor to both of you and points for bravery. Who wants to go first?

Brent Kostak
[ 00:01:45 ] I’ll take this first. Then you can add in some context for some of the product innovation. So, currently at Adobe, we’ve been in the market a long time with Adobe Target and Journey Optimizer. And from a use case perspective, we’re starting to branch out beyond just optimization and UI, UX specialists, teams that are using Target or channel marketers, email, lifecycle, journey experts, and NAJO. And these use cases that we’re seeing…

Brent Kostak
[ 00:02:15 ] primarily drive around the evolution of this agentic AI era of experimentation, is now to drive higher impact campaigns and journeys. When we’re thinking about the broader end-to-end customer journey, understanding higher conversion, higher revenue lifts in that context. Second use case: automating a lot of the experimentation analysis so you can get faster prioritization around what you’re testing, why you’re testing it. And then the third is a really… peer use case where we’re seeing a lot of growth teams, whether it be marketing centric, product centric, looking to increase subscription growth, you know, across some of their services and products. And that growth use case is something that we’ve expanded into across some of our work here for this new AI-first product we’re launching. So those are the three main use cases that we’re kind of taking to market. But in the context of optimization, experimentation, and growth, it kind of fits into those three core teams and personas. David, I don’t know if you want to add anything to that.

David Arbour
[ 00:03:16 ] Yeah, no, I think that’s great. I think from my perspective, it seems like everything that Brent talked about is exactly right. And I think really what a lot of this points at is that it’s never really been easier to create content for experiments. And I think increasingly it’s maybe never been harder to get meaningful insights on top of that. And so a lot of this is just saying, like, you know, we know we need to experiment a lot. We know it’s very easy to generate a lot of content. But what’s the best way to sort of make that analysis approachable, but then also reusable in future contexts, right? So you’re not just running an experiment and sort of lighting it on fire after it’s done. You’re like really kind of learning from it over time.

Matt Wright
[ 00:03:53 ] Yeah, totally. Okay, that’s awesome. Thank you. That’s a great way to set the scene too for this. In terms of like experiment accelerator, specifically, and maybe just the topic of AI guided experimentation, like what actually is it? How would you describe it to someone?

Brent Kostak
[ 00:04:08 ] I’ll take this first as a pass too. It’s not my expertise right now. Adobe Journey Optimizer Experimentation Explorer. It’s a new AI-first application we’re launching. It’s coming out September 30th. And this is a new application that we’re initially building for Adobe Target and Journey Optimizer customers to accelerate and automate experimentation analysis.

Brent Kostak
[ 00:04:31 ] Identify high impact opportunities to scale growth, and then scale experimentation programs across the enterprise and those three you know aspects of where this product is tying into our existing customer base and where Adobe has been. But there’s a lot of generative and agentic AI as a foundation for alignment to Adobe’s overall agentic AI innovation right now with the new Agent Orchestrator platform we’ve launched. So this product is a piece of our broader innovation that we’re launching with all AI agents in these new AI-first applications. But specifically for Experimentation Exciter, it’s all about automating some of this analysis, identifying where to test next and identifying new opportunities to drive growth, and then scaling in terms of people, processes, and beyond the technology piece. Scaling maturity of the experimentation programs themselves. And we’ve seen a lot of excitement on that piece too.

Matt Wright
[ 00:05:24 ] Yeah, very cool. And then maybe that kind of segues to the next thing. It’s just like when you think about this, you’re already kind of hitting on some of them, Brent, but then maybe more broadly, like what are the problems or challenges that this will solve or really at least take away some of the pain that people are facing? Maybe David, I’ll start with you, just to get your perspective, because I think it’s fascinating from the scientists’ perspective, like research science perspective as well.

David Arbour
[ 00:05:45 ] Yeah, I think it’s like never been a better time to work on experimentation, which is funny because I don’t know that I would have said that, you know, candidly three years ago. So it’s really great to sit here now.

David Arbour
[ 00:05:53 ] Yeah, I think, you know, like I was pointing at earlier, like, you know, it’s super easy to create content now. I think one of the… ways we sort of came to this project is you have this sort of recurring conversation where people get very excited that they can create a lot of variance. And then they go to run their experiment, and they realize it’s underpowered or they don’t know how to make sense of the results, and all of these things. And so, I think, from our perspective, that indicates a couple, like, a couple key things that we needed to work on. One was just sort of like making plain why you’re running an experiment. Sometimes it’s like not always clear to people what the value gain is. And from my perspective, like, when you run an experiment, the thing you’re really trying to do is two things. You’re trying to make a well-informed decision and you’re trying to sort of build learnings. So your intuitions on future decisions is better. And so the entailed question from that is like, ‘Great,’ so how do we use AI to do that? A lot of this is saying, ‘Well, we have all of this great ways to sort of interpret content, sort of think about the similarities between things.’ These are all things I think, you know, really good product managers already do in places that have these deep experimentation cultures.

David Arbour
[ 00:06:58 ] But I think we have this opportunity now to use AI to sort of put that into product and say, ‘Hey, these are the attributes of things that are working. These are the attributes of things that aren’t.’ And based on that, maybe you should run this next.

Brent Kostak
[ 00:07:10 ] One other piece to that, I’ll add, Matt and Dave, that’s spot on, is one of the, like an organizational cultural shift that we’re seeing. And this is a really exciting part, like even through the beta and some of our customers, you know, that are using multiple different applications across our experimentation stack at Adobe.

Brent Kostak
[ 00:07:29 ] They’re really trying to understand how these new AI agents, agentic workflows are in a way more easily accessible across different teams that they can align those business goals or experimentation efforts or optimization efforts. We’re typically seeing a lot of the problems being solved is now that we are building kind of a centralized integration point across all of our areas where a customer and practitioner can experiment, whether it be different channel teams, different personas, different business units. And it’s really exciting to see a lot of our customers are starting to ask questions on, ‘Hey, instead of just having like a biweekly monthly team meeting where we’re trying to pull these learnings across different teams or we’re trying to, you know, unpack them outside of a backlog of PowerPoint slides or Jira tickets.’ We now have this, you know, automated fashion— a lot more data that we can get from these agentic conversational UIs to put that into a centralized, you know, area where then teams can kind of start to understand how they can contextually work together, that makes sense.

Brent Kostak
[ 00:08:35 ] And we’ve seen that where, you know, typical optimization shop might not be the same business unit within the same brand. That’s owning the channel you know, SMS push channels and they’re running some experiments with landing pages and emails, but there’s no context around what that team might be doing, or you know, if a director of an EP position is owning this whole program they’re now trying to say, ‘Hey, let’s bring our teams together and let’s make some sense across, you know, acquisition, engagement, and retention.’ So that’s an exciting area where we’re seeing this growth perspective. There’s been a lot of trends in the market where that’s an important aspect now in terms of the actual innovations that we’re seeing internally from an organizational cultural perspective.

David Arbour
[ 00:09:20 ] Yeah, just to quickly add on, I think like one of the things, one of the things that like I have learned over and over and over is that like you spend all of this time sort of putting together these really careful experiments and you run them and then you have these beautiful slide decks. And then there’s just point of friction on learning on them because it’s just someone has to recall that slide deck. And that’s like a really unnatural motion because then you have to sort of like ask for it and someone has to recall it. And if someone leaves the organization, God forbid, then you’re basically just lost to the ether. I think the thing that’s cool here is that it’s essentially like a way to remove some of that friction so you can actually like make experimentation be. Something you actually keep around and not just something that you do and learn in the next month, you know?

Matt Wright
[ 00:10:02 ] Yeah, definitely. And maybe this is like a great jumping off point for me to geek out a little bit or dive in a bit of how this actually works. Because I love the kind of preview and knowing a little about it. You’ve got the whole AI experiment insights and that’s kind of learning across experiments.

Matt Wright
[ 00:10:16 ] One of the things I wanted to ask is: I’ve definitely played around with AI’s and maybe it’s just me, but I’ll be vulnerable for a second, which is like, I’ve tried to give lots of context, ask it a question, and then almost had a frustration that it only took two or three sources of my context and then spattle my answer. I’m going, well, what happened here? What did I do wrong? So maybe I would love to be like, ‘Why is this not going to happen in this case?’ And then how does it actually work? Like, how do we solve for that? What does that look like? And maybe being super naive, but I think that if you’re just new to this space, something that like people bump into kind of thing.

David Arbour
[ 00:10:50 ] Yeah, no, that’s a great question. I like to think about, or say, the things that I think about are just ordered in terms of like the things that keep me up at night, like most. Right. And this is like literally the thing that keeps me up most at night. And so we were really careful when thinking about this, and exactly that, that problem. Right. So, like you. One of the most sort of straightforward things you could do is you just dump your experiment into an LLM and say, ‘Hey, tell me what happened,’ right? Why’d this work? And you could do that, and sometimes it looks plausible. I’ve done that, and sometimes it looks plausible. Sometimes you have to play with it a little bit. Because you have, you know, two problems, right? The sort of glaring omissions, and then the problem of hallucinations where things just appear. That’s one of the things that kept me up at night, saying, ‘Hey, we should just sort of launch this out to the masses.’ So what we do is like a little different, which is that we explicitly take representations of the content itself, right? So we use the LM’s ability or the general model’s ability to sort of like take a fixed representation of that. And then we have a set of sort of like attributes and features of that content that we learn from historical experiments. And so under the hood, you can imagine like when you look at two experiments or two things, you say, hey, why are these two pieces of content?

David Arbour
[ 00:11:59 ] behaving similarly. Well, you probably have a set of attributes. You say, oh, you know, this is about jeans and this is about jean jackets, right? So maybe they’re both made out of denim. Maybe they’re both soft. Whatever. So you have all these attributes and you essentially score them in your head. And then, based on that score, you then translate that into how you think the performance is going to do. That’s exactly what we do. And the advantage of that is now we have these fixed scores. So it’s essentially like an audit trail, right? So if you go back in a month, you’re going to get the same answer that you got today because nothing’s changed, right? It’s not based on sort of the stochasticity of like the language model. And the other thing that lets us do is like really anchor against historical experimental results. So that means that we look at the model, it’s a model that’s built on prior experiments and their relationship to attributes. And so when you go to run a new experiment, we use those learnings to say, ‘Hey, this is in the context of your new experiment. This is what matters, right? Like these are the attributes that matter. And then we use a language model to make that more fluent. But the basic facts are all really tethered to what I would call like sort of hard attribute truths rather than sort of like a more flourishy language that you may or may not be able to trust.’

David Arbour
[ 00:13:06 ] It’s the same thing with the opportunities, right? So we look at that same model and we say, ‘Hey, these are the things that we’ve noticed in your other experiments. That are working well.’ So, and we notice it’s not present in your current experiment. So maybe consider, you know, making your tone more empathetic in, in your next, um, your next campaign, maybe consider making it, you know, more humorous, whatever. I’m just making up attributes. Brett’s much better at sort of talking to actual, like, uh, you know, the human aspects of it, but. That’s the basic idea, right? And so it’s a little different, I think, from what you typically hear, because I think typically when you think of LLMs, it’s just like, you know, content, prompt, go. And here we… We intentionally sort of buffer it, and we say, ‘No, actually, there should be a layer of indirection where you have to ground on facts, and you have to ground on replicability, because otherwise it makes it hard to sort of make decisions that you could defend to your boss in three months, right? Because, hey, why’d you do this? And it would be like, well, at the time my language model said it was a good idea, but who knows? Right.

Matt Wright
[ 00:14:06 ] Does that like scale indefinitely? And maybe the way I’m kind of going in my head is like, what if there is even like, like things that are in conflict or something like that? And how will it handle it? That like, Hey, we saw this result at one period of time and that and the other. And there’s like some unknown context that changes it.

David Arbour
[ 00:14:22 ] Yeah, absolutely. I mean, I think the thing that’s nice here is that like, because it’s sort of ground, I’d like to say it’s like a handshake between like AI and like classical statistics, right? Like classical statistics is very good at giving you guarantees and AI is like super good at giving you sort of like interpretable results. And so, all of these like problems that you’re pointing to— things that, you know, things change over time. We know this is true. Anyone who’s sort of like worked in the content space knows that some data from last year is just not valid or things, you know, it was a holiday or, you know, it’s a different brand, right? All of these are things that you can sort of put in the model and like explicitly represent and then control for statistically.

David Arbour
[ 00:15:00 ] The nice thing, the other nice thing about this quickly is that— Um, the more experiments you run, like, just the more powerful the model is, right. And the more sort of tailored it is to your experiment, like your experience and your experiment. So we have sort of like a baseline model that we use. Um, but you know, if Matt, if you’re running like a thousand experiments, your model is going to be much more tuned into the kind of things that you’re running and what’s working specifically for you and, you know, in the context of this year or whatever.

Brent Kostak
[ 00:15:27 ] So it’s interesting, like when we talk about this broader, you know, customer experience orchestration story Adobe, we kind of cover these pillars of like data, content, and journeys. And we had that framing at Summit. I think it’s really great to kind of pillar these practicalities or scalabilities to those three foundational models.

Brent Kostak
[ 00:15:47 ] David’s talking a lot about the data, the context, the LLM models, how we’re building these experiment variants, how we’re suggesting that context on our platform and other agents. The content piece is also… A part of our conversations, like even with beta customers, right? It’s like, you might have this team that is purely focused on experimentation and they’re getting the outputs. We had, you know, a customer ask about like the same thing, like ChatGPT and, you know, what’s the difference? We have teams that are already doing stuff like that. But in terms of on-brand compliance, right, there’s a lot of guardrails and aspects for using AI that brands need. And you get that baked into how this content supply chain story Adobe is connecting into experience delivery activation for how we’re building our applications, like these models. So, like on-brand content, making sure that everything is compliant, making sure that those approval checks, as part of the experimentation process, are, you know, in a handshake or like lockstep with what the optimization teams are doing. That’s one important point that that’s a content pillar.

Brent Kostak
[ 00:16:49 ] The journey’s pillar too. There’s also, you know, the— aspect of like arbitration that comes up sometimes too, where you get this like conflict management, how are things being done? Across the same landing page or across the same channel? And do we have this air traffic control plane to understand what’s happening and where? And that’s where, again, this centralization for bringing this evolution of experimentation to this product. But being built on Adobe Experience Platform and having all of the, you know, additional benefits from a platform play, you get the additional coverage with, you know, what you were talking about with scalability and making sure that these individual teams running experimentation, optimization are in the context of what’s happening across campaigns, across journey variants, and this experimentation agent that’s powering this AI-first application we’re talking about today. In context and orchestration, right? That’s happening behind the scenes too, with all these agents. So you might have like the journey agent extending over to help understand, you know, where to create the experiment.

Brent Kostak
[ 00:17:52 ] And that’s like the vision with why.

Brent Kostak
[ 00:17:55 ] You should have the scalability enterprise play here for the platform that we’ve built this on. And I think there’s a lot of foundational pieces of that beyond just the scalability of the experience variants themselves, aligned to this data content journey story. So hopefully that is foundational for where this fits into kind of the scalability story and where a lot of our experimentation, core personas and practitioners are asking these questions that are like content authoring teams or, you know, the channel email delivery teams. Those are all kind of coming in uniting into one conversation now.

Matt Wright
[ 00:18:33 ] Well, maybe I’ll just have one more question of that for listeners. Cause even myself, I think this would be helpful. I think I hear Adobe talk about like AI capabilities or features, and then talk about Adobe agents specifically. Are those the same thing? Are they totally different? How does Adobe think about those between like a feature and an agent?

Brent Kostak
[ 00:18:53 ] Yeah, you can think about it kind of in three different variations. So Adobe’s launched the AI Assistant. That’s our conversational piece that’s surfaced in different applications. So you’re engaging with a, you know, interface to ask different prompts about, you know, what’s happening with your experiment. For example, or give me a summarization of my experiments. The AI agents that we’ve launched that we announced this year at Adobe Summit are a new era of agentic AI in some of our agents that are running reasoning. Different queries across different data sets and taking actions on different tasks are built into the agent orchestrator platform. So Adobe’s got Adobe Experience Platform. This agent orchestrator platform that we’ve launched at Summit is coordinating all the AI agents that we’re speaking to today. And those are different parts of the overall portfolio of how we’re tying in the AI features into these applications.

Brent Kostak
[ 00:19:55 ] A lot of our partners, clients, and customers are really excited that they now have this agent orchestra platform to build out custom agents and tap into some of the reasoning and data query that these workflows are being surfaced across the applications that we’re talking about too. So I think the AI assistant conversational piece is one part of this. AI features that we’re, you know, talking about for like content generation. And then these new agents we’ve launched— 10, I believe— now. So that’s this, this new kind of, uh, era of, of evolution that we’re speaking to with this experimentation product.

Matt Wright
[ 00:20:33 ] Is there anyone, any of these agents, that would be interesting specifically to call out of the 10 you’ve launched that might be good to share, like particularly relevant for Experiment Accelerator?

Brent Kostak
[ 00:20:41 ] Yeah, it’s primarily the experimentation agent. So if you go to like any of the Adobe product pages talking about AI or our agent orchestrator platform, you’ll see these individual agents. Experimentation agent is powering this AI first application that’s called Experimentation Accelerator. Journey agent and data insights agent, I would say, are like closest to where we’re seeing teams very interested in how we’re kind of bringing together some of these different workflows to users in the product. David, I don’t know if you would have any others, but those are, you know, there’s, there’s a lot of potential. I would say, for the roadmap of where we’re going with the product, but those three agents I would say are the top three we’re getting a lot of questions around from the analyst, experimentation, and optimization teams.

David Arbour
[ 00:21:28 ] Yeah, just quickly, I think the one place or in the future I could imagine this really like playing in is like the audience agent is also sort of like very interesting, right? Sort of like understanding your audiences, sort of thinking about how segments work, because you could imagine. Experimentation and understanding your audiences are sort of like— you know, the part and parcel. Right, so like running a good experiment in part is like who you’re running that experiment on. So I think that’s exactly right.

Matt Wright
[ 00:21:50 ] It may overlap with what I already asked, but just to dive in a little bit deeper, how do the agents actually work, David? Are they very much the same across all of them? Are they different? How does that work in terms of each Adobe agent?

David Arbour
[ 00:22:03 ] That’s a great question. So they’re pretty specialized to use cases, I think. So like Brent sort of indicated, there’s an orchestrator, right? And you can sort of think of this at the middle of the symphony, sort of like going to the different parts of the play. And then each one of the agency or at least, you know, the ones that I’ve sort of like seen in development or like, you know, my close coworkers have worked on are like pretty specialized to the use case. I think a lot of work goes into two things. Like one is making sure that. We have all of the features that you want to see when you go to ask it for certain things. And the second, which I think is maybe not as obvious, but something that we spend a ton of time on. I know folks…

David Arbour
[ 00:22:42 ] put a lot of hard work into making sure that that conversation is translated into the features that you’re actually trying to use to make sure that experience is like not only not frustrating, but actually feels like something that’s natural, right? That is sort of fundamentally different than a UI. And so I think those are sort of like the two components, right? So one is like, you know, experimentation guy thinks really hard about experimentation and like adding a bunch of sort of like nice bells and whistles. But the other piece that I think is surprisingly difficult to do is, you know, user sits down and says, ‘Hey, I want to play with this experimentation thing.’ They’re actually intending is coming back to them. Not the same, like you have this many samples, like, you know, 200 times or whatever, which I think if anyone’s played around with prototypes of some of these orchestrators, that can happen, right?

Matt Wright
[ 00:23:33 ] I might just geek out for a second. So just humble me. I’m just curious. So if like, I go really deep on this. So you want to build your own agent at like Adobe. What is that actually like? Like, how do you evaluate it? What was the process of actually bringing one of these to life? How do you optimize it? And then, maybe also like, what’s the thing that surprised you about how hard it, like, what was really hard to solve for in an agent like this?

David Arbour
[ 00:23:53 ] I think, well, first I’ll jump to like the very last piece, which I think I’m like constantly surprised at how ambiguous language can be. And so I think, sort of, knowing like what the right sort of evaluating context and making sure that you’re thinking about all the use cases, I think, is constantly.

David Arbour
[ 00:24:12 ] It always surprises me. I think, instead of talking to folks and, you know, so working on this stuff, I think, if you think about how you’re connecting it, like the big thing is that, you know, typically when you do what I now call classical machine learning, which is sort of machine learning 2020 and before.

David Arbour
[ 00:24:28 ] It’s like supervised, right? So you have labels that you’re trying to fit to. If you want to know how well it worked, you can just sort of like see how precise it was or how accurate it was. Now everything is really based on human feedback, right? And so there’s this very non-trivial loop of saying, you know. I have these queries. Now I have to annotate 400 examples of real product managers talking to the product. Is this correct? Was this frustrating? If this was an in-scope or out-of-scope question, was that correct? Or was this something we should have answered? Or was this something we shouldn’t have answered and we did anyway? And so it’s a little— I don’t know if this is a satisfying technical answer, but I think in many ways it’s almost like a sociological like endeavor to be, you know, how are people trying to interact with this? And then you see things on live traffic and the questions are, you know, a little bit, a little bit different even. And so you have to go back and sort of like iterate again. Prompts get updated and the examples get updated.

Matt Wright
[ 00:25:22 ] What’s the gist in that? Obviously, this will continue to evolve. How will this continue to adapt as LLMs evolve, all those types of things? It’s more of a general curiosity, but obviously, it’s a very fluid product— I almost think about it as, or it’s a fluid platform, maybe.

David Arbour
[ 00:25:39 ] Yeah, I’ll give, like, maybe the not officially sanctioned by Adobe answer, and then Brent can clean up the mess afterwards. Yeah, no, I think that the thing that’s most exciting to me, right, is that I’ll just use experimentation as an example, right? So right now we have like a set of capabilities that you can sort of interact with. You can ask questions about your experiment. But you can imagine there’s this like really exciting interplay where, like, translating those queries to like increasingly nuanced questions and just executing them using these tools that we’ve built up, right? So, you know, right now we have 10 agents. You can imagine in the future, underneath the hood, that actually could be, you know, hundreds of tools still in the same agent that we can sort of really flexibly compose. So, if you’re asking questions about, hey, find me the best. I ran this experiment. Just find me the best performing segment within here, right? Maybe it’s not one I’ve even defined yet. Maybe then it turns around and characterizes things about the customers. Then we’re talking about campaigns. Maybe you’re talking about sort of journeys that you could lead these people on.

David Arbour
[ 00:26:40 ] But these may not even have been defined a priori, right? Maybe these are like workflows that are literally being defined as you’re interacting with the agent. And that’s the thing that I think we’re going towards because I think our ability to do reasoning is just getting much, much better. And I think it’s getting to the point that if you give it the right sort of Lego blocks, you can actually, you know, build a rocket ship that may not actually take off, but does look super cool.

Brent Kostak
[ 00:27:04 ] One additional piece to that. And I think Matt, it goes back to one of your questions.

Brent Kostak
[ 00:27:09 ] This is kind of, you know, coming from third-party validation beyond, you know, Adobe and then also customers is. The way that we have orchestrated this agent composer to build out custom agents with partners and clients, that’s one option. And you have the extensibility, the infrastructure there from Adobe Experience Platform. You know, all these agents being, you know, grounded in some of the real-time profile data that you have across AP is a huge benefit to our partners and clients. And then you also have the, you know, what we’re talking about here with these applications that are…

Brent Kostak
[ 00:27:46 ] through conversations, through the individual UIs of the products, giving you kind of out-of-the-box workflows and things that you, you know, the non-technical personas and practitioners get all the reasoning and some of the power of these new agents that we’ve launched. So we’re kind of going to market with two. Different pieces, but it’s an important point to know that our agent orchestra platform, because it’s built on top of APEX within Adobe Experience Cloud, you get a lot of the out-of-the-box extensibility. The real-time profiles and all the behavioral signals that are happening across these individual channels or campaigns or customer journeys. So I think that’s a core differentiator that is unique to Adobe right now versus some of the other applications or platforms we’re seeing around these agent workflows. But that’s one additional piece I wanted to call out. Trying to frame up the understanding here of Adobe Experience Platform for those listeners around new to Adobe Experience Cloud and seeing how we’ve launched this orchestrator platform, we’ve launched these individual agents, and now we’re kind of talking about new native AI-first applications on top.

Matt Wright
[ 00:28:56 ] Someone wants to build a custom agent. Can they do that inside of Orchestrator, like with Adobe? Like, what’s the, maybe just open that up. Cause that’d be my immediate follow-up. It’s like, how can I do that if I want to?

Brent Kostak
[ 00:29:08 ] Yeah, like David, I don’t know how deep into developer tooling and some of the practitioner extensibility workflows that we can speak to on this call, but that is part of the agent orchestrator platform, this agent composer. So we’re building that out, Matt, for clients, partners to have a lot more control, right? with where specific use cases they want. And also, like I know a few partners that we’re working with now that we’ve, you know, announced partnerships at Summit, they’re kind of… building out some of these extensibilities with their technical developer teams. They’re customizing them to their business cases or use cases. And then you also get some of the out-of-the-box models that we’re speaking to that you can fine-tune within the application. So there’s two levels to it. From a technical perspective on the developer tooling and how that works. David, I don’t know if you want to add anything to that.

David Arbour
[ 00:30:02 ] Yeah, I guess just to make sure I don’t talk out of school, I’ll just think about internal tools. And I think, like… I think it’s related, though, which is that I think one of the things that excites me about this is that, you know, these tools aren’t siloed, but I think a lot of times, because it’s UI-based, right? Your workflow is based on, you know, how you sit down and use the product over the course of four hours. I think the thing that’s cool about this. are exciting about this is that it really sort of makes workflows a lot more like fluid. And so, you know, now if we’re thinking about interoperability, like, you don’t have to be an experimentation person to like benefit from using the experimentation tool. Which I found has been sort of like a blocker for folks in the past, right? So if you’re someone who likes never thinks about experimentation, you can still sit down and be like, ‘Hey, can you describe, you know, this is a little forward-looking, but like, you know, hey, can you describe the things that have been working over the past year?’ Like, can you just show me the trends of the attributes that seem to be working with this, this segment of people? And that sounds really simple, but underneath the hood, there’s like a ton of things that you have to do, right? You have to know what segment you’re talking about.

David Arbour
[ 00:31:03 ] You have to look at all the historical experiments. You have to look at the trend lines. You have to make sure it’s like not biased. You’re picking up weird signals. Um, but I think, I think we’re moving to a, towards a place where that kind of stuff is like entirely possible. And I could imagine third-party things fitting into that. Although, you know, um, it’s even just with internal sort of tools. It’s cool.

Brent Kostak
[ 00:31:22 ] This composer, you know, we probably have another separate call on just this, you know, Adobe’s agent orchestrator.

Brent Kostak
[ 00:31:29 ] Yeah. And the agent composer, the nice part, too, is like we have an agent SDK, an agent registry. There’s a lot of different tooling that is built in, so you can set these up organizationally. You have different teams— kind of custom building some of these workflows on top of the agents that we’ve. Been speaking to today, so there’s a lot to be you know kind of unpacked with how we’re going to market with the orchestra platform itself and you know like this core team here speaking more to the fine-tuning of the models that are surfaced within that uh application AI first application the experimentation accelerator. So there’s kind of two variations that you can see that we’re partnering with clients and partners and agencies with. And then we’re also launching these new applications.

Brent Kostak
[ 00:32:15 ] Yeah, I think we’ve gotten good feedback, like I mentioned, from third parties, but also from our partners and all the exciting news at Summit this year. We’re starting to deliver even more innovation across other pretty big names in the market that we’re working with. So I think Adobe is in a pretty exciting spot right now when it comes to… Our agentic AI overview and strategy.

Matt Wright
[ 00:32:36 ] Really? Maybe making it real too. I know I think you’ve done initial beta for six months, Brent, or even longer at this point.

Matt Wright
[ 00:32:46 ] I know we’re just in the weeks where this is formally getting announced. So there’s kind of in that awkward, like, ‘Hey, what can be shared? What can’t be shared, but maybe just at a high level, very curious if you can maybe just share some of the stories or trends around like business impact that people who are starting to take on some of these AI capabilities are starting to realize.

Brent Kostak
[ 00:33:04 ] Yeah, we do have a launch blog out, so I can speak to a little bit of that before the official GA date. One of the stories that we took to Summit, which I’ll talk to one internal customer of ours, which is kind of the CustomerZero. com team. And then I’ll talk to some of our customers in the beta. But when you think about, like, business results and proof points, and like how are like, what’s the true value behind some of these new experimentation capabilities?

Brent Kostak
[ 00:33:32 ] Adobe. com was our first partner. I mean, they run Adobe. com. They have their product organization that is managing a massive program for Adobe with a lot of products, analysts, and engineering teams, and marketing teams, kind of underneath the same organization for just managing our public website. So when they were going through some of the testing on Experimentation Accelerator, they were really kind of trying to align their goals and revenue impact. And some of the stats that David took to summit with Paul Alleman and our product organization were really significant. It was not just like faster velocity of testing. But like over 200% increases on the number of variations that they are now able to build into their experiments. Significant increases in terms of win rates and how much consistency they’re seeing for tests, hitting statistical confidence or having a winning test or learning from that. And then, the third was around, again, kind of tying to this growth initiative.

Brent Kostak
[ 00:34:39 ] A lot of these, you know, achieving greater impact in terms of business revenue or growth. And that growth foundational piece was really impactful to see. It was like over 200% increase in terms of ARR impact per individual test. So, when you think about, like, what are we seeing from an individual experiment? How is that aligning to whether it’s an increase in conversion or whatever the success metric might be? So, Adobe. com, that’s basically customer zero.

Brent Kostak
[ 00:35:09 ] AAA Northeast was another beta customer. We have them talking through some of their… you know, insights, opportunity, test ideas, and the value they’ve seen. This is an exciting one that I’ve included in our launch blog too, but when they went back to the context of these insights and how it’s powering use cases. This automation of experimentation analysis, their team was able to get these contextual AI experiment insights that were aligning to a campaign they were launching that was targeting new member benefits, launching a new campaign. Some of their value that they saw were emphasis in terms of engagement pathways that they could potentially embed into the experience that they were testing. So not only, just like, Hey, here’s, you know, an increase, try this messaging, you know, security-based messaging is driving increase in conversions. That was like one aspect of these insights, but the value and the quote from Melissa and the team. It was the intuition of what was working across their customers and what the marketing strategy or testing strategy could be expanded in scope.

Brent Kostak
[ 00:36:15 ] That we’re talking about here in the models, we’re telling the team to, hey, maybe add some different call to actions or online consultations or something embedded within that landing page experiment to open up more of a broader marketing pathway for engagement to then increase. Your revenue and success metrics. So that context, to like the campaign or journey itself, having that embedded in some of those insights, to say, you know, how you should be thinking about the layout, the design of the experiment, the actual experience that they’re embedding into the webpage or mobile app or email channel. That was like, you know, something that was foundationally groundbreaking for that team because they were able to then get some context to the engagement strategy and their marketing overall marketing goals beyond just kind of like increasing conversion of revenue. So those were like two core foundational stories I can say at this point through the beta.

Matt Wright
[ 00:37:08 ] I’m sure there will be a lot more to come too. And then, as this comes out, maybe I’ll make sure that we try to connect some of those articles in the show notes or whatnot as well.

Matt Wright
[ 00:37:16 ] Just, just making sure I also like maximize our time. Cause I know you’re both very busy. I would be curious. This is a very candid question, but there are obviously other platforms that are coming out with agentic AI capabilities. Would love to get like a little bit of like from your guys’ standing. How do you think you’re different or what really sets Adobe apart in this part of like the kind of the landscape? It would be awesome to kind of get some of your perspectives.

Brent Kostak
[ 00:37:47 ] David, you want to talk about the technology foundation? Then I’ll give the kind of market validation.

David Arbour
[ 00:37:51 ] Yeah, that sounds great. I think, you know, it’s funny when, you know, in the last few years, you sort of see this stuff popping up, right? And, you know, at first, everyone, they all, all of these solutions, maybe this is talking out of school a little bit, but they all kind of have the same general, like, you know, we’re going to revolutionize your life kind of claim. And then, when you start to go underneath it, you know, I’ve noticed there’s a few different patterns. And I think the first one is like really a focus on like, I think what folks call it velocity, right? Like, hey, we’re going to get you like a lot of content really fast. And then another one is sort of like, Hey, we’re going to, we’re going to optimize on top of that really quickly. I think the way that we’re a little bit different than that is that, you know, I think a lot of this is very cool and I’m not trying to like put down any other tech, but I think the problem with like pure velocity based testing is that it’s kind of twofold, right? A lot of times you say you should run 100 times more experiments, but unless you fundamentally change the way you’re analyzing the experiments, unless you have 100 times more users, you haven’t really bought yourself anything but frustration. Right. And so the way that you typically will like—like—accompany this is you’ll just get a little looser with your stats testing underneath.

David Arbour
[ 00:38:57 ] Right. Like, oh, whatever. Right. Which, you know, you can have— you can say it’s good. You can say it’s bad, but I can say it’s like harder to verify. And as in terms of someone who’s like— maybe a little bit more anxious about like throwing things up to my boss that, that would, that keeps me up a little up at night. I think the other thing is like, you know, if you put that on top of certain, like a pure optimization standpoint. So, like, a lot of times what you’ll see is like, you know, I have these degenerative content. One of the things you could do is you can just say, ‘I’m just going to purely optimize on top of that.’ There’s definitely use cases for that. I think the thing that you have to sort of be honest with yourself about that is when you just purely optimize for this use case, there’s a natural trade-off between how much I learn about the other variants versus how much I optimize for today. Right. So there’s this like notion of sort of like, you know, if I’m just optimizing for regret right now. So regret is like, hey, how much money would I have made if I just launched this variant?

David Arbour
[ 00:39:48 ] Then you’re not going to learn about anything else. And I think there’s, if we think about what we’re doing with experimentation, you’re really trying to sort of like learn from you. It’s not just optimizing for right now. It’s sort of like learning from everything.

David Arbour
[ 00:40:02 ] In the future, even if something wasn’t the absolute best, maybe it was actually very good for a certain subset of users, right? And if you’re just sort of doing this pure optimization play, you’ll miss that. And so I think the way that at least we’ve been thinking about is saying. Now, actually, the thing you should do is very much ground everything in analysis, right? So you should enable yourself to run a lot of variance, but we should be really careful about changing the way we’re analyzing that content in a way that preserves some statistical rigor. Right. So there’s a couple of things I know it’s like been out for a bit at this point. But like the confidence sequences is part of that sort of like having like those anytime guarantees to say, look, look at your content whenever you want to. But we will do a little bit extra work on the back end to make sure that you still have your seatbelt on. Right. So like you’re not going to sort of shoot yourself in the foot. And then on top of that, saying, ‘Okay, maybe we’ll also change the way we’re looking at what it means to compare two variants by like using the representation of them explicitly in the test to say, ‘Okay.’

David Arbour
[ 00:41:04 ] We can run more variants. Maybe we’re thinking about the way attributes are related, right? And so we get to keep some of the stats that we, honestly, have always been the selling point to folks from my perspective.

David Arbour
[ 00:41:16 ] While hopefully like letting people do some more of the opportunity of that sort of like flexible, generous stuff. I know that was, that was a bit of a mouthful. So hopefully I answered your question.

Brent Kostak
[ 00:41:25 ] I wanted you to go first because David is the, the research brains behind this. Matt, two more, two more points. I think we covered a lot of the platform and agent orchestrator and kind of the delivery piece.

Brent Kostak
[ 00:41:40 ] One thing I’m getting from some of our customers, these are customers too, right?

Brent Kostak
[ 00:41:46 ] Knowledge of this market, where we’ve been with Target for many years, and where other you know, um, experimentation platforms are building out some of these similar agentic or ideation and new test ideas, when we built the Experimentation Accelerator, we wanted some of these experiment insights. So that’s like the core foundational piece of directional guidance before you even get into a new test idea. To be based off of, like, three different components: one was the content itself. That’s typically where you see, like, pattern recognition, some of these embeddings that David’s speaking to. The second is audience and where you’re targeting. So that’s a component that Adobe is building into these insights. And then the third is the test behavior itself. So it’s not user behavior. It’s actually what’s working within the test. How is some of that causal impact analysis being built into those insights? So three different coverage areas for our experimentation insights, getting a lot of traction there. And then the other piece of this was— in terms of um, you know, model flexibility, we’re allowing and enabling customers, partners, to bring their own multi-iron bandit tests in different models to kind of again, fine-tune, adapt to their experimentation shops, which is really exciting.

Brent Kostak
[ 00:43:01 ] Then the final thing I was going to mention was— in terms of the overall delivery component, there is this aspect of, again, mentioning the scalability across maturity of teams. A lot of our organizations and beta customers are like, ‘This is great.’ Give us the new insights, new test ideas. We have a backlog, but we want to know what’s working and then what to do next. The delivery component, I’m looking at a lot of…

Brent Kostak
[ 00:43:29 ] you know, been in market research for the past year and seeing that a lot of customers and folks that are using these new agentic workflows, the hardest part and maybe some of the most trusting part is actually how you execute that test or how you build the test. not just the ideation hypothesis generation, but actually the execution part. And this AI adaptive experiments that we’re launching, it’s a new novel methodology of where a lot of the research that Adobe’s partnered. Where David and his team have done a lot of the innovation here. It’s this human in the loop workload where you can now add and remove different treatments or different variants to active experiments. That’s a novel concept. It’s going to probably cause a lot of debate, which I’m excited about in the market and how Adobe is doing this and where that component of execution and delivery is unique. But that’s really cool because it unlocks this generative AI concept, right? I mentioned this content supply chain narrative.

Brent Kostak
[ 00:44:27 ] Content velocity is no longer the issue. It’s making sure that, as David alluded to, statistical rigor, the sample size ratios, the traffic volumes, like that’s the stall. In the limit that we’re seeing a lot of these customers that now have all this data in agentic workflows at their hands. They’re like, ‘This is great,’ but then how do we actually take action on this in reality across campaign, a journey or a touch point of what their optimization teams are focused on? There’s a lot more to unpack with that. We’re doing some thought leadership in the market separately, but that adaptive experiment type of model that we’re launching is really, really cool. So I wanted to make sure that that’s something that is taken away from this talk where we’re trying to help the build execution delivery phase just as much as we’re trying to help the ideation. Concept testing and like, where to go directionally with your experimentation strategy, overall—yeah, no that that’s awesome.

Matt Wright
[ 00:45:18 ] I actually was going to say all the things that I’ve seen inside of this this adaptive experiments was the number one, if I was going to have like a A follow-up on this, this jumps out right away because I was like, this goes against everything you’re ever taught about, okay, you need to flush the data when you restart an experiment, all that kind of stuff. So I think that’s its own topic. Just coming up on time, I have two closing questions. So a little bit of a rapid fire round for you both.

Matt Wright
[ 00:45:41 ] Two different parts. First part is more like…

Matt Wright
[ 00:45:46 ] We’ll call it like predicting the future a little bit. So everything does seem to be going through some step change here. Very like shifting landscape. I think we just think big picture around like experimentation programs. Even that is a very broad group of people these days doing very different things, but let’s just stay abroad for a second. What do you think that will look like a year from now? What will be different or fundamentally different that will exist then that isn’t now? Again, bonus points for everyone to jump on that first.

David Arbour
[ 00:46:13 ] I’ll jump in quick because Brent does a great job of cleaning up my messes.

David Arbour
[ 00:46:17 ] Yeah, I think the thing that’s like most exciting is that I think that the place we’re moving towards is having experimentation and sort of explicit intervention, like more tightly coupled into the workflow, right? And I think the, if you think about this, like the more sort of like black box stuff that we have, right. The easier it is to do this, like the more important it is to sort of stop, be careful about measurement at some point in the process. Automate that to the extent possible, right? And I think experimentation is like a you know historically a very natural way to do it. We’ve been doing it since we tried to figure out why aspirin worked. Right and so, like, now if we sort of like look forward, it’s like very similar. Right, so this seems to be working. Why and that why question says well, I should run an experiment, right? Here’s the notional idea. And you can imagine a world where even part of these workflows, maybe the agents are literally proposing experiments back to you. Hey, we think this is working. Do you want me to run this experiment for you?

David Arbour
[ 00:47:07 ] That’s where I sort of see it going. Less sort of maybe is like sitting, I think historically it’s sort of like sat out in its own little island. I think now it’s just like a much tighter like coupling into the workflow.

Brent Kostak
[ 00:47:20 ] I’ll give a quick answer too, because I love David’s answer there. I think experimentation is like in the spotlight right now in different facets. And one of the more like future vision conversations is like, as we’re seeing the technology, the usability, how… You know, these agent workflows are being kind of in parallel worked with our, you know, human aspect and the actual experimentation teams. I think programs overall are going to start to ask. Like, how do we make sure that we’re thinking about these different modalities or different ways to optimize? So before, you know, biggest news in the market was how we’re optimizing for LLMs to feature brand content. Within search and different things that we’re talking about now for generative AI and LLMs impacting, you know, SEO. That’s like a huge foundational piece of the market right now. When you think about that, in the context of an experimentation program or governance, now we’re starting to think about questions with teams of like. How do you optimize for prompt conversations or chats? Like, the component in the success metric of optimization will evolve over time too.

Brent Kostak
[ 00:48:24 ] So I think these use cases are expanding beyond just like conversion revenue gains to like the actual engagement piece internally or the operational piece? And how do you think about these different agents running some optimizations or embeddings across the time to value in a conversation? Or in like some universal conversational chat. So that’s a cool piece of this. I think experimentation from an operational standpoint, and then experimentation just in terms of engagement with the user experience or customer experience, is going to be foundational to where we’re at in a year from now.

Matt Wright
[ 00:49:00 ] That’s awesome. And then, my second one, which is an advice one. And I think it’s a McKinsey quote, which is something like roughly three quarters of AI initiatives fail, kind of been passed around quite a bit. I’m curious if you have any kind of guidance that you’re already seeing. I know it’s early days, but just how to successfully adopt and realize value from this to kind of key things or foundational things that people should be thinking about.

David Arbour
[ 00:49:21 ] Yeah, I think a lot of it is like very similar to like failed experiments, right? So I think there’s this notion of velocity, right? Like we have to get something out, right? And I think one of the things that can get lost is that like you really need to keep your use case in mind and define success in an objective way that doesn’t use the word AI. Right. So, and if you can, you can do that and then sort of like build for that use case, AI can help in that. But I think where a lot of folks get lost, right, is that it’s very easy to do the first 85% with AI. And it’s often like very hard to do that last 15%. And I think the difference is that if you have a very sort of clear sense of what that objective is, what you’re actually trying to work towards in a way that doesn’t use the word AI, then it’s much easier to define the path to success.

Brent Kostak
[ 00:50:06 ] Yeah, that’s a great answer, Davin.

Brent Kostak
[ 00:50:09 ] Kind of similar notion, Matt, where we were talking to just where we’re kind of going from this cultural perspective. I think organizational readiness is just as important as this new, you know, era of the technology advancements for generative and agentic AI. So think about like a concept for our beta customers. If you’re just trying to embed automation and some of these agent workflows in terms of what they might be outputting for experimentation insights or generated new test ideas, those teams are kind of like just— they’re just… skimming the potential of success. When you skip out, I think, on foundationally, how will this impact how our teams are working? What is the model for the augmentation piece versus the automated piece? That’s a big concept right now for like… Analyst workflows being augmented with some of these new data points that those teams can then, you know, take in and understand this product, you know, works with Adobe Analytics and Customer Journey Analytics. And we have a ton of teams that are focused on experimentation.

Brent Kostak
[ 00:51:12 ] So the augmentation piece is just as important as some of these automations that we’re now embedding into those user flows. But organizational readiness, like that’s foundational, I think, for success, where in production, if you haven’t thought through what that component will be across different organizations, different business units. You’re just kind of siloing yourself with maybe some increase in revenue and conversions, but you’re not unlocking the potential of where your teams can be operating in a state that’s completely different— maybe. In a year or two, but with good— you know, good difference. I think it’s, it’s challenging brands to understand, you know, what they should and how they should be thinking operationally, just as important as let’s make sure that, you know, our teams are generating some of these embeddings and LLM models on, on what they’re already doing existing today.

Matt Wright
[ 00:52:01 ] Well, I just want to say super fascinated to talk to you both. I’m sure there’s going to be so much more that’s about to come out of this in the next six months in particular. I just want to give you a big thank you. I know that we will just add some stuff into the show notes that were mentioned. Is there anything that either you guys want to suggest on, hey, if people want to keep learning about this, what’s a good thing to check out? Keep kind of taking them down that path.

Brent Kostak
[ 00:52:22 ] We have some upcoming in-person events that we’re running into next year. You know, big news coming at Summit. So our Adobe Summit is always a great place if you’re able to visit one of our main conferences. And then, yeah, we’re running a lot of thought leadership. podcasts on YouTube now. Our teams are doing podcasts like this. So stay in the loop, stay tuned, and we’ll be talking to your listeners very soon. But thank you, Matt.

Matt Wright
[ 00:52:46 ] Thank you. Thank you for joining. Cheers.