Speaker 0: Business, but as everyone here in marketing operations, I think it’s about continuously learning new things and at the current moment on deep diving to how to um, use AI in our marketing activities. Jaime, now when you’re here, should you do a short introduction?

Speaker 1: Absolutely. So thanks, Adele, for for leading of the session. Thanks for holding the fort. My name is Jaime Lopez. I work at Ivan as well as a senior director of marketing ops. Um, I have an unconventional background for ops, just like everybody in ops, probably. I’m a nuclear engineer by training, but I’ve been doing data driven sales, data driven marketing, marketing ops for almost my whole career for of about ten years. Um, right now, I work at Aiven, which will explain you in a second what is Aiven, why are we talking to you, why are we interested in these topics. But, um, my background as an engineer and my background in in data, uh, probably shows through the presentation the way we approach problems, uh, at Ivan and in my teams in a quantitative way. We want to, um, show that marketing can be a science apart from an art. That means that things are measurable, things are provable or disprovable, and all results are provisional until we have validated them with data.

Speaker 0: Yes. Thank you. That was profound. Um, in very short, as I said, really, really want you to have something to take out from repression. Uh, we’re today looking first. Um, what is journey today? I what? Why is it in print the current moment? Then we’re jumping into super, super interesting topic about segmentation. Then after that, we look how Thio, um, use that in a journey optimization. And finally, we’re looking how to personalize content and scale. And, Nohany, maybe you can open a little bit about where we’re coming from.

Speaker 1: Sure. So what do you see there? Ivan, probably the crown in my shirt as well. Uh, very cute logo, but what on earth is this? So we are, um, a company, it’s a former startup now that’s scaled up, about 500 people worldwide, who build the data infrastructure for the services you use, love, and trust on a daily basis from banks, governments, food delivery apps. A lot of their applications that offer you a warm meal or, um, how far away is the next cap from me run on our infra. We build those services that provide you and provide kind of our customers’ customers things of value, and that has to do a lot with storing data, streaming data, analyzing data. Nice. So this is a really good metaphor to start to frame the conversation about, like, why are we here today? Um, a few months ago, Adele and I had, um, an ideation session. We started to to delve into what was possible with AI and whether AI solved any of the problems that had been stuck on our desk for a number of months, quarters, or years. And we little by little started to see that, yes, it actually does solve some of the problems that have been kind of, um, paining us for a while, but also makes things possible that we haven’t even considered. And then to use a historical metaphor, I started to think about, um, the gold rush of California in 1849. And, you know, a lot of people migrated to the West Of The US, uh, fueled by the promise of finding gold, and in the end, they ended up building modern day California. But the people who actually stayed and provided value were not just the ones who went there and, like, went searching, panic for gold, but the ones who built the infra to make it possible, the ones who sold jeans, spades, picks, access, housing. And I thought that that’s very similar to the role we want to play here as Ivan, but also as marketing ops professionals. We build the infra that delivers value to our customers. And I thought that what a beautiful way to illustrate this, that with a AI generated picture of what would the gold rush of 1849 look in modern day San Francisco, which is what you have in front of you. And now moving on to the next slide, and keeping that metaphor present, um, the next question is like, okay, why do we do AI? But now it’s clear, but why now? I’ve been working with machine learning and AI for six, seven years. Um, I have further studies after my master’s in AI, um, and it has always been complex. For the first time, I would say about one year ago, it started to be more useful than it was difficult. So I think at that inflection point is when it’s the right time for us to jump on that wave, to catch this wave now, when we can derive benefits from AI and machine learning without being very, very onerous to to jump on board. Alright. So continuing this, um, why now, why then? Um, now it’s the time of explaining the what. What can you do with generative AI? And we chose four use cases that are probably going to resonate with all of you or who you have had to deal with at some point in your MOPS life. First, brush strokes. So when we started planning how to present our results, we actually asked a generative model, like, what would be a good outline for a presentation on this topic? And we started building from there. Another thing, um, that’s extremely useful in mops is to be able to bridge gaps in your knowledge. You’ll probably hear a lot of people during these days, uh, at Modriming talking about becoming a super generalist or a super powered generalist. It’s exactly what, um, what AI does. This is the calculator for the for the scientific professional or the modern professional. To give you an example, during my analysis, I needed to receive prospects from from Pardot, from account engagements, and I wasn’t sure how to write a call for the the new API version five. So I just told an API to write a call for me that did what I want it done. So from my width of knowledge, I can also now add a lot of depth. Best practices. So, um, if you are hosting webinar and you want that to be repeatable, scalable, and documented, you can ask an AI to give you a list of best practices. Um, and also something that probably all of you have had to do. If your business owners come and ask, like, hey. We need a tool that does this. Um, which one should we buy? You want to drive that process. You want to make sure that the requirements are collected. You can ask an AM model to quickly make a comparison matrix of available, in this case, social media syndication tool. So your competitive analysis is already kind of partially started for you. Alright. So now you also saw a few things kind of on surface level what you can do. That’s something you can already take back home tomorrow, uh, to your to your work. But let’s go a bit deeper. How do we get started? And the focus of this session when it comes to data, it’s going to be marketing cloud account engagement, formerly known as Pardot, um, the pre trained OpenAI models, and a coding tool of your choice. In my case, this is r. I’ve done a lot of statistics in my life, and R is a convenient language for me, which is what you’re gonna see throughout this. But don’t be discouraged. You can also do it with any other programming language of your own. Um, also, if you don’t know how to program, I just told you one slide ago how you can get this done using AI. So, um, code is a useful tool to expand your effectiveness, but it’s not the only way to do this. So now that we have framed the problem and we know that we’re gonna use, um, some code, we’re going to show you what to do with customer segmentation. And this is our literally one minute demo, which I’m going to walk you through as soon as it starts playing. We noticed we had a problem. We had about 60,000 people in our database that had a job title but no persona. We had never asked these people to self describe themselves. So in one minute, what you’re gonna see there is that I am reading data of Pardot, reading basically all my prospects, then filtering the people who are our employees, the people who don’t have a job title, and kind of storing them in a data structure or data frame. Uh, and then now I am actually sending an API request to, uh, GPT model telling them this is how I describe a developer, this is how I describe an executive. Can you please classify these buckets of people that I’m going to send you next? And the model starts kind of well, I format the data a little bit. I rearrange it to send it to the model, and now I’ve sent a request. And you’ll see very soon that, um, in a white arrow, you will get we will get an answer for the model. There it is, which is stored in a variable called answer, which tells me for each of these 10 people or 10 prospects I send them with their prospect ID and job title, what is the correct persona classification. Alright. Um, how does this look like if we move to the next slide? Let’s see. Basically, what we are going to get is a categorized version of the data we gave the model as input, which you see here. Uh, for people whose job that was research director, my model correctly classifies them as a tech executive, but it’s not very certain of it. Okay. I’ll look at that later. For people who are, like, senior software engineers, classified as a developer with 90% certainty. Brilliant. It’s exactly what I what I would want to do. The model also learns to recognize that there are some people who have a bogus job title, like Factotum. So we will leave them unclassified and give me a lot of certainty. So you can I can filter them out later? But, overall, the model understands these job titles as well as or better than a human does, even job titles in different languages, um, and processes this with an efficiency that is at least 20 times that of the fastest human, which you you would, by the way, kill by boredom if you made them classify 60,000 prospects by hand. So now I have a scalable way of classifying my prospects and segmenting them into different buckets that allow me to do things like behavioral clustering. Okay. So now we know who they are. Next, it’s time to look at what they do, how they behave. And I defined from my business strategy, four clusters of behavior. So I’ve taken two variables from account engagement, which are, um, their monthly spend with us, which is a custom field, and their Einstein behavior score, which is a standard field. And depending on whether they are very engaged or unengaged, spend a little or spend a lot, I’ve put them into one of these four buckets. And for each of these four buckets, me, together with my, uh, business, have defined what are the targets. People who spend little but are engaged, I want them to spend more. People who don’t spend and are unengaged, I want to offer them a free trial. People who are very engaged and spending a lot, I want to serve them take care of them. So now in the next slide, you’ll see how I can take these four clusters and teach a machine learning algorithm, um, a clustering algorithm, to feed it prospects and find which are these clusters and assign to each prospect a cluster number. And my idea will be to try to move these people progressively up and to the right to the cluster where they’re engaged and spending. Now you’ve seen how we classify people in personas, how we classify people by behavioral clusters, and I will hand it over to Adele to tell you about how to weave this into the big picture.

Speaker 0: Yes. Thank you, Jaime. This is super interesting. And as Jaime mentioned, it is actually something that you have the tools in place if you’re having for instance account engagement. So check the recording. I hope you took some notes. We’re happy to open it more. Um, that is maybe something you want to take into action. And believe me, um, I’ve been happy on our exercise and done a few experiments already in our systems, which we are going to walk through now. First of all, one more thing to say. I said there is always a person behind your prospect, which has different needs. That said, there is no one flow, not one customer experience that we’re having. But, indeed, we are coming to a world where we are having complex solutions, complex different kind of changing intense activities, needs of each and every individual. And that gets very complex when we’re thinking about the marketing operations word and or how do we respond with the right message at the right time. And AI is here to support us with that. We’re looking now, um, next, a short example on what kind of solutions we can use to, um, make timely messaging for the right audience. Um, as as we just saw one note here though, before we dive into, of course, as we know, if you are having a watch available, we are quite tight on time. So we are focusing only on email examples, Jay. These all methods, as long as you are having an integrated MarTech stack, are available for any of the other activities that you are having individually or as a whole multichannel campaign. So keep that in mind despite we’re looking for one example. Um, and the slides are me. Good. Here we have a very, very, uh, simplified example of a trip campaign made for cost optimization for a certain region. Here we have prospects for certain segments. So as Jaime showed, we have the segment clusters and they all have their different business needs, which requires a different messaging with a different CTA. Of course, we most probably have all used dynamic content. That is a powerful tool that we can personalize the messaging. But that’s not quite enough if we are thinking about the activities and intense. And how do we want to respond on those needs, um, for a certain prospect? That’s where the other things are coming to play. We’re having, for instance, I ai based scoring in use how we’re calculating and reacting with the behavior changes. On top of that, we’re using third party, uh, tools such as qualified and 6¢ in our stack to enrich more intent, uh, changes inside and outside of our environment. For instance, in this case, we can see a third party intent increasing. They are searching for cloud cost optimization as well as we can maybe see that they actually found our site and they engage with our chat, which increases their score. These actions together combined with the profile, we are able to have a right message for them, but also we are moving them in between different programs. So for instance, in this case, we had a higher intent, um, program that was supported with a different kind of scalability. So there were more actions happening, a slightly different actions happening based on the intent and activity changes for a certain segment. As said, the content is king. Uh, Jaime will walk through us how do we use the segmentation also in the content.

Speaker 1: Cool. Thank you very much, Adel. I’m really happy that you can show us kind of the big picture and the whole dimension of this because here we’re showing you one toy example of one tiny case of one tiny channel. And now you can take this, go to town, and build a much bigger castle. But with a much bigger castle comes a demand for a whole lot of content. If each channel, each, uh, persona, each behavior cluster needs different content, how am I going to generate all that at scale? I don’t have more resources. So what we thought could be useful is to actually use generative AI to create that content for us. And we started explaining this time to a text based model, uh, which is GPT 3.5, what we were trying to do, and what were the behavioral clusters we have created and the personas we have created. So there you see some prompts where we are kind of teaching a model, what do we want to get? And then we fed it, um, the the content of one campaign for cost optimization, and next we got output for how this email about cost optimization should be personalized for each persona and each behavioral cluster. So we’re taking, again, one single channel email. But when we move to the next slide, you’ll see again another animation where we have fed this into a pilot via the API, and we are using dynamic content to personalized subject line, email, text, and CTA. You see that the the generic version just has, like, absolutely no content there, um, or just has, like, placeholders, like, generic CTA, generic, uh, body. But as soon as I start picking prospects that I created ad hoc with the correct persona and behavioral cluster, you’ll start to see those subject line, email body, and CTA changing. And it’s important to notice here that the the topic of the email will remain constant. We want this to be a part of a cost optimization campaign, but the tone of voice changes depends on the persona, and the the wording depends on the persona, and the CTA depends on the persona. So here you’re seeing something that we are showing, if I remember correctly, to a developer who’s unengaged that has some spend, and we want them to explore, grow their engagement. If now we show it to a developer who is a lost cause and what I wanted is to offer them a free trial, I’m going to get the CTA and the email subject line to emphasize the free trial, but still talk in language that a developer will understand and resonate with. If I choose a different persona, my model generated me text, CTA, and subject line that’s suitable to a completely different persona. And I think that’s the end of our demo of how to generate content at scale for your campaigns. So I’ll head it I’ll hand it back to Adele for her to take us home.

Speaker 0: Thank you. Thanks. Um, indeed. We are coming to the end of the presentation. So, also, final minute to type any questions that you have in mind, which I hope are multiple. But just final reminders before going there and especially as the market is wild for AI solutions. First of all, especially for your, um, new fellow MarTech guys, um, you own the stack. So you are also the driver of your car. Make sure that everything that you are doing, everything that you are buying is making sense. It has the ultimate bus business case and business impact. As we saw here, you can do a lot of different things without actually having, um, having to invest a lot of money. It’s more over what do you wanna do and then figuring how how you are going to do it. Second thing, also something, um, that came out behind the scenes is that especially if you are doing a multichannel campaign, which I definitely encourage you to do, your data have to be in a readable format so that you can utilize it, play it around, also have integrations between different systems and use the same data point to several purposes. That is how you create this engagement and automatically floating systems, to guide your prospect with a timely messaging in the right time. Understand your business. So everywhere, you have to have a purpose. For instance, if you start now tomorrow or after today’s amazing sessions building your segmentation, you have to yourself understand what do you wanna do, what with for what profiles, and what could be the possible CTA. Also, AI suggest things, and you have to yourself validate those things. That said, also be aware of the biases that AI has, especially if you are building always on programs, uh, or segmentations. Go there time to time. Check what is the content about? Is it still up to date? Does your model support? What is your aim? With that, uh, I want to thank, uh, from both of our side and, of course, our sponsors for this amazing event. Uh, thank you for having us to again this year. I think we have a couple of minutes for any of the questions. So let’s see if we have some PMB session comments.

Speaker 2: Yes. We do have a couple. Um, I’m gonna click share on here. This should pop up the question on but if not, I’ll also read it. So, uh, uh, from Christopher Mead, it looks like it popped across the bottom. So I’ll let you all take over.

Speaker 0: Yes. Amazing. Actually, this is something and Jaime can maybe continue. But, um, in one of his slides, you were able to see the, um, accuracy. So basically, it’s comparing to the historical data. And we are calculating accuracy towards that, which is actually a very, very good, um, identificator if you’re building your programs.

Speaker 1: Yes. Building up on that, um, the only a 100% bullet proof way to check for accuracy is to to do a manual classification and compare the two of them. However, that’s quite onerous. So I would only do it for a small dataset. Let’s say, classify a 100 personas manually, and then tell the model to classify those same 100 personas, see how they match. What you see in in the data field that we are uploading is not exactly accuracy. It’s confidence. So how confident the model is that it’s doing things correctly. And that very often is enough. I I have enough results that kind of lead me to being able to trust the model. Cool. Thanks, Chris, for the question.

Speaker 2: Yeah. Great. Thank you all. Um, and it looks like Chris has another question, so I’ll go ahead and share that as well.

Speaker 1: Notice any difference in results with this methodology compared with existing sensors alternatives? Um, this is the, let’s say, the the backyard experimentation that leads to then Salesforce features being built. Um, you’ve seen up close an example of that, of what nowadays is part of Einstein attribution. It was previously a model that that that we built at Wartsila and that some people from the Salesforce unit in Terra built them into a feature. And you will soon see something similar because, um, when Adele and I went to Dreamforce this year with basically this talk, the afternoon before the talk, we started chatting with some people who, uh, were product managers at Marketing Cloud. And we realized that Salesforce is building a very, very, very similar feature to this for Marketing Cloud, which was bittersweet because on the one hand, damn it. Kind of they are a bit faster than us, uh, in building these things. But on the, like, on the plus side, this is going to be available for everybody. It’s a great feature. And if we are doing something at the same speed as a Salesforce, we might not be too bad. So a lot of it is crowdsourced from the community, and a lot of it is the amazing engineering teams that they have.

Speaker 0: Yep. And I think that was really, really good, Albert. Also to continue for Chris, this is like if we’re talking about Einstein’s same time optimization, for instance, this is slightly more scalable because you can use it also out outside of Salesforce products. So that is