Name is Garrett Sauls. I'm a content manager at Innerworks, which I always say is I'm a corporate English teacher. I essentially help all the smart people in InnerWorks share their knowledge and get to connect with smart people like Michael and Annabel to share their knowledge out with the community. I'm based in Tulsa, Oklahoma, and I have the privilege to be here today. So I'll pitch it over to Annabel who will introduce herself. So a very warm welcome also from my side. My name is Annabel Rinken, and I'm in Switzerland. So we have a variety of time zones today. I'm very delighted to be here, obviously, and co host this webinar series with Garrett. I have twenty two years experience. And during that time, I have won many ads, starting as a data analyst, growing to lead Tableau developer, and more recently, serving as an enablement lead. And through all these four, one thing that has become very clear to me is that enablement is crucial to the success of an organization. And sadly, it's often the missing piece, the one that can truly make the difference in the implementation of a data analytic platform or digital transformation. And that's what we launched this webinar series with Garrett to spotlight care enabler and learn from voice driving impact across all the industry. That's why today it's a great honor to have like Michael. And Michael Sanderberg will be speaking about a fascinating and intriguing subject for me, AI hallucination. Michael currently serves as a senior quantitative risk analyst on the USAA data reporting and governance team. I had to read that or I would have forget to say it correctly. He brings deep experience, he had worked in many renowned companies in business intelligence, such as General Motors, America Express and Walt Disney Company, And he joined USAA in twenty nineteen as a senior business intelligence analyst. And what's fascinating me about Michael is that he never stopped learning and sharing his knowledge. So I know that Michael holds a bachelor degree in history and computer science, but recently he even completed a postgraduate program in Artificial Intelligence and Machine Learning. And that truly fascinated me the way, Michael, that you never, never stop learning. And that's it. A very warm welcome from Michael. Thank you so much for joining. Michael, now the floor is yours. Okay, good morning, good afternoon, good evening, everybody. My name is Michael or Mike Samberg, I prefer Mike. My presentation today is The Interesting Unusual Life of AI Hallucinations. We'll get into what AI hallucinations are in a minute. I do want to thank Annabel and Garrett for inviting me. We've talked about this probably for several years now, and I welcome this opportunity to present to you, and they both are wonderful people. So that's even a better scenario for me. If you do have any questions, please ask them in the chat. Not only are they going to capture them, but I will anything that we don't get answered, I will answer and send the answers to them to distribute to all of you. Okay, I need to make this disclaimer. So and I had to do the same thing. I spoke at tableau conference two years ago about centers of excellence. The opinions, ideas, insights, and recommendations expressed in this presentation are solely those of the content creator, me, and not his employer, USAA, its affiliates and employees. Basically, we have teams that actually are responsible for this, and we don't want to imply that I am inferring to what USAA strategy is. This is just me talking about a very specific topic here. Okay, so one more comment before I begin. I have the way I learned AI is I am constantly creating action figures. And I want to mention this as a preface, is that we took a trip in our first cruise in March. It was called the Flower Power Trip Cruise. It was a bunch of 60s and 70s bands and all of the people, you know, on the cruise, made a lot of friends, made friends of some of the people in the Facebook site we all talk on. And I started making action figures of all the bands and stuff. And then all of a sudden, people started asking me. And then I have people at work asking me. So I have a literally a waiting list of about forty people asking for figures. It's just, a lot of them are just what comes in my head. I'll see something, say, this would be funny. And again, this has helped me become very proficient, particularly at AI prompts, being very specific because of the hallucinations. So I will show you examples of my work, and if there's anything interesting about that, I will tell you a couple little nuggets about it. For example, a lot of you may know the new Superman movie came out, and crypto sold the movie. What I told it to do here is when I was a kid, there was actually a Superman comic strip in the Sunday funnies, and I said use vintage Superman comic strips for the background, and it did so here. So I thought this one was kind of cool. This is one of my favorites that I've done. But, you know, sometimes I get caught off guard, and I see people, or I say, you know, what would be funny is if I did something like this. And this is Garrett with a beard. This is Annabelle, and they're on their Harley, and they're in the countryside. I actually had I was gonna put the Eiffel Tower in the background, but they're in the countryside in Paris. This I actually use the tool called ChatOn dot ai to create. So and if you look at the bottom, I usually will tell you who my sources are and what tools I use. The reason I do this is, this is something I learned about storytelling, which is a separate topic, is that if people say, well, that's not true, or I don't believe that, I'm saying, fine, just go down here, here's the link, go check it out yourself. So with that said, just a little bit about who I am. I've been in this profession for forty eight years. Oh my gosh. And we called it everything for computer science, computer engineering, IT, MIS, data and analytics, BI, AI. So it's went through the gamut. As I'm we were talking earlier, all my exes are in Texas, and I'm not talking about wives. I'm talking about schools. I did my undergrad at Texas A and M, my master's at SMU, my post grad at UT Austin. I've been working in BI and data intelligence for about twenty five years now. I did MicroStrategy for about twelve years, did Cognos for three, and I've been working for Tableau for close to ten years now on and off. By on and off, I just moved to a different position at work, and the last year and a half, I hadn't used Tableau at all. I live in Phoenix, Arizona. I'm a lifetime Lions fan. The last time they won the championship was the year before I was born. It's a long time ago. And something I'm real proud of, I have lost sixty five pounds in the last year. That's about twenty percent of my body weight. I kind of have a goal of losing about thirty two more, and then I think I'll be happy. If any of you have questions of how I did this, just ping me and I'll be glad to talk to you about it. And, oh, this is a figure I made of me. You know, it's like everyone, I'm self conscious about my my my personal figure, so this one's better. And I said, put me in front of Niagara Falls on a beautiful sunny day and all that. So that's what it did. What are AI hallucinations? This is a pretty scary figure, but it kind of hints at some of the things I'm going to talk about. Okay, so AI sometimes makes mistakes. And as you can see from this figure, this was AI generating multiple faces. It was only supposed to generate one, and in all these instances, here, she's literally attached to the top of the head of the other image of Herb here. Same thing here. Here, they just meshed her into this one and over here. So there are things, you know, AI can make mistakes. I don't wanna say, and I'll talk more about this, that AI is wrong, AI is broken. You'll hear people say that, but I'll tell you why in a little bit. So what is an AI hallucination? So as human beings, we have crazy dreams. So you might have a dream where, you know, you just picked up some pizzas at the pizza parlor, the box swings open, and the pizza starts talking to you. And, you know, you wake up, go, wow. Man, that seemed real. I swear that was real. That really happened or not? You know? So so we don't we don't call those hallucinations. We just either call it we ate something bad or something we saw that day came into our subconscious that night. But, you know, when you build a system such as artificial intelligence, as I have typically used OpenAI, ChatGPT, responds to a serious question with a response that ranges to it being inaccurate to silly. So, in essence, the world of AI, a hallucination is an instance in which an AI tool makes up something that isn't true or doesn't exist. Later on, I show a brief quote by a person from Scotland who summarizes it nicely. Okay, so what is AI? So let's think of this just in simple terms. Think of AI as a super smart robot brain. And during development, people like you and I, we give this brain huge amounts of information called data. So how huge is this data set? Hypothetically, let's start with all the information on the internet. They could have everything, and I mean everything. Every Facebook and Instagram post, every word out of every encyclopedia, every book, every song, and then you can keep going. So basically, every piece of information, whether it's video, it's media, it's text, images, are all in the dataset. So, training and thinking. One of the things you'll want to do is train your AI system. And the way you do this is you would, so it's very much like normal human intelligence. We organize information, we analyze it, and write a detailed report about the information, organizes it, analyzes. In other words, we ask AI a question. It gives us back some kind of prediction. I'm calling it a prediction, not an answer, but a prediction. But unlike a human, AI systems don't think when it did all that work. So, you know, a lot of people, like I said, will despair AI. It doesn't really think, it just returns. So that's right. It has predicted what the user based on what the user asked to perform, typically called an AI prompt, wanted based on the way the user worded the question, and all of the data the AI system has been trained on and put that information together to make the prediction. And it does this in a couple of minutes where most of us would have to do some work. Okay, here's a picture again of someone from my cruise for you folks that are a little older. This is Ron Dante. Ron Dante was the voice of the Archies. Nineteen sixty nine, he had the number one song of the year, Sugar Sugar. Hopefully, some of you folks know who that is. He also had another hit with a group called the Cufflings called Tracy. He is the only artist to have one hit wonders, two one hit wonders reach number one. Now this, I actually gave it a cartoon picture of the Archies, and it did miserable here. I am not sure why it made the changes and just didn't stick it in. This is Groovy Bear. On the cruise every year, this guy gives out these little bears to everybody, and they're it's called Groovy Bear. And as you can see, they misspelled it it changed well, this was actually printed on the bear. It changed flower, the third letter, so that's wrong. Again, we're seeing these little hallucinations, but you're gonna see Groovy Bear again throughout my talk here. Okay. Machine learning. So machine learning takes AI a step further, in that it enables the AI brain to learn. To train an ML system, you have to provide it lots of information, what you want it to know about. For example, if you wanted to know about cats, you're going to enter information about cats, their drawings, photos, other types of cat images, you know, information about its physiology, things like that. And this is called a training data set. Then you could enter a cat photo, I could give it a cat photo, and ask it to describe. This is kind of the whole concept of computer vision. Computer vision was my favorite class in my post grad. Would really love to do stuff like that. And basically, so that, you know, with this training or test data set to describe what it is. The system should be able to correctly identify the object in the photo because we have told it in great detail what a cat is, and it has learned this, what a cat looks like. Keep in mind, as you ask questions and it gains new information, it has a feedback mechanism and feeds that back to where it takes it, it puts it in the format it needs, adds it to the model so that it's available over and over again. And so it's constantly learning. This one is from one of my coworkers, and I had asked her permission to use it. She puts out, you know, a deck is a PowerPoint presentation. And every time she puts out a deck, she puts one of her cats. This is one of her cats. And interesting enough, according to my sources, is her cats all have the name of alcohol, like Scott's, like Purrfessor Scott's. So I thought I'd kind of make fun of this. And I basically said, Director of Muse, and I showed the logo, Pink Ball Meow Magazine for the discriminating cat. And of course, after a hard day of chasing mice and, you know, chasing leaves that are rolling around, you know, Professor Scotch comes home, lays down, grabs the catnip, has a catnip, and relaxes for the evening. So this is what I made for her, she's kind of like that. Okay, so enabling that brain, again, it's a structure we follow very similar, like a neural network, and think of it as the structure of the system of the brain cells. It's made up of a lot of tiny parts that work together to understand and learn from the data presented to it. Just like humans, the neurons help us think, learn, connect, create memories. There are nodes that enable the AIML systems and to make connection between items in the data and be trained. Like I said, I have a warped mind, I think, in things that inspire me. So here is a picture of the new Superman movie. This was the very first early shot they showed. I'm not gonna say anything about what that is outside his window. You have to see the movie. But he was putting his boots on. And the first thing that came to mind when I saw this, for you folks that are not in the United States, the first thing that came into mind, well, it's time to make the donuts. It's early in the morning, which is the model for Dunkin' Donuts, which is a doughnut shop. So I said, you know, I'm gonna have him make the doughnuts. Now look how he's sitting. So I basically had him take that picture, said give it the title, time to make the doughnuts, and have him sitting in a Dunkin' Donuts, having a cup of coffee, and one donut with sprinkle and one donut with blue icing, I think I said. So that that was him. Instead of holding his boots, he's holding his coffee and pondering how he's gonna catch Lex Luther. But there was one of my action figures I made. I'm curious, Michael, this is this might be a good point to ask this question someone had asked because we're talking about how AI just computation works and how it thinks or doesn't think so, so to speak. But Paul Paul Albert had asked, are there parallels between, like AI systems, hallucinations and misperceptions, and things we see in pop culture that maybe people thought up? Can this inform my understanding of how people think? So I think what the person might be asking is, is there any parallels between the way computers might think and people might think? And is that present in, like, AI outputs? So in a couple of slides I get into that, you know, in that, you know, AI really doesn't have a conscience. It doesn't have a soul. It basically, based on what it's learned, can try to answer questions. I'm going to, at the very end, show you one that I did of Ozzy Osbourne. I did not have any mention of a snake in it. Okay? However, revised my prompt and put a snake in it. You know, as to what I showed, I thought it was kind of cool, I kept it. So obviously, somewhere in the system that was trained for OpenAI, they have associated with Ozzy Osbourne something significant related to a viper snake. So I don't know if I'm answering the question, but basically, there are some parallels, but again, what you think, and when it goes to your imagination, it doesn't have that per se creativity. It's just going to, again, you ask it a question, it gives you what I refer to as a prediction. That is, this is what it thinks you want to see. Cool, that's great. No, I think that's a good answer, and I'm excited to see further elaboration in the later slides that you were pointing to. So thanks for that. Okay, sure. So let's talk about examples. This is one I call Liar Liar Pants on Fire. And for you folks that are from Scotland, I apologize if I butcher this. So here is the prompt they use. The Glenfinnan Viaduct is a historical railway bridge in Scotland, UK, that crosses over the West Highland line between the towns of Millagh and Fort Williams. Most likely there was more to this where they asked it to draw it. The problem is, is this photo is very inaccurate. First off, the viaduct only goes one way. There aren't two tracks, so that's the first thing. Two, it put a second chimney up on here, and there's only one chimney on this train. And what it did with the cars, if you notice, as it goes along, it slowly elongates them. I mean, this one's really big. So they're not all of equal size, which they typically would be. So it had some problems when it drew this. Okay, and we'll talk a little bit about why some of these problems occur. Okay, so this was, left the person, guilty, out of this, but a famous Scotsman said, GPT sometimes churns out rubbish. And I thought, oh, okay, I can live with that. So ChatGPT three is an older version when they did this, we are on what, four point zero, and then I believe there some renditions of five that are out. And they asked this question. When was the Golden Gate Bridge, for those of you who don't know, the Golden Gate Bridge is in San Francisco, California, transported for the second, not the first, the second time across Egypt. And, of course, because, you know, it didn't wanna show you that it didn't understand, it says, oh, just tell them this. It was transported for the second time across Egypt in October twenty sixteen, which is not true. They haven't moved the bridge once or twice, but here you go, Egypt's fake Golden Gate Bridge. And the problem here is it didn't know an answer, and it could have said, I don't know, or I'm not sure. Instead, it said, let me just tell them to see if I can get away with it. Duplicates. This is something you will see a lot, high CLI. So someone probably said here, show an elder couple enjoying coffee in the morning, they're both happy, they're in a lightly lit room, and they're mellow and all that. And what and they're and so not only are they is she holding a coffee cup, she is holding two. Notice her weird fingers. We'll talk about that in a minute. And they have one in the air here, this is the magic coffee cup, and then all these others. This happens because of the resolution of the images that are inside of the model. If they're lower resolution images, there's more probability that it will duplicate elements in the image because they struggle to scale to a pattern. So also, if the training data is limited, if we haven't properly stated that, you know, a human, a man or a woman, will hold a single coffee cup in their dominant hand one at a time and not have other coffee cups present, you know, things like that. You could give it all these instructions. So with that said, you know, if the training data is limited, if it has errors, poor quality, the model's ability to understand and reproduce content weekly develops. This underscores importance of having, this is just like you folks that are in the Tableau community, high quality, diverse training data, and maintaining consistency and resolution to minimize such issues. Okay? So, here's another one. Again, I'm talking more about proportions. So for example, if I had a square image, twenty four by ten twenty four pixels, and I took that and I wanted to make a non square image, this one is, even though I trimmed it, I should have left it as it is, but it really is a rectangle. And it has problems when you take a square and try to show it in different formats like a rectangle. As you can see from the lady sitting here, they probably said we have a lady relaxing in her room, and she's knitting and, you know, enjoying life or whatever. But what it did here is it obviously stretched her out. Her arms are very well extended. Her body is highly extended and not proportionate, and a lot of that has to do with the image it used. Okay, too many fingers. So here, we show someone a kneading doll, and that's fine, except they have five fingers. Now you could always argue because there have been cases, people have talked about people that have no extra toes, extra fingers, and things like that. But why would this occur? So one of the simple reasons is, in most pictures, hands are not emphasized, as other parts of the human body typically take pictures. It's usually a face, or from, you know, your belly upward. And it may not know what a hand is. So for example, if I had this as an answer, I could, you know, remember, you can keep, it's called chaining, where you can keep chaining answers on here. So you could have said things like, a human, male or female, each has two hands. On each hand is a thumb and four fingers. The finger closest to the thumb is the index finger, which is generally used for pointing. The second finger closest to the thumb is to wish people a happy day. I'll let that one, let that sink in on your own. Okay? But, you know, you could say a hand can wave and flex, hold objects, clench as a fish. You can have them stuffed in your pants pocket, partially hidden from view. So if you add that, maybe as it regenerates, it would all of a sudden have four fingers and thumbs. But something like that you would want to do to basically just correct that, and then it takes that information and feeds it back into the system so it keeps learning. Causes and impacts of AI hallucinations. This is basically the dumpster fire, and I'll get myself in trouble for this, but, you know, the company right now is the expert at dumpster fires, but we'll let that go. Okay. Surprise, AI systems do not think. To go back to your question, Garrett, it's important to understand AI hallucinations are not machine learning errors. They're not errors. It's doing exactly what it was trained to do. It's to predict and respond. These systems don't think. They don't have a conscience, they don't have a moral code, they don't understand what a fact is, what truth is, what falsehood is. This is kind of like Congress and the Senate. But anyway, they simply make a prediction about what the user wants in a response based on the questions asked, and then the system provides that information. So when a system returns a hallucination, it's not broken, it just did what you asked, it answered a question, and the answer might not have any facts in it, it may be totally misstated. That's why you should always check your work. I don't, in this presentation, in the future one I will, I'll actually just discuss data. Like I took data from, let's say, Excel, I took data to do some things to show why it goofs up with numbers. But I could do that hopefully in a future one, if Garrett and Annabel let me. I wanna show this one, so again, my company's headquarters is San Antonio, and the pride of San Antonio is the Alamo. Years ago, Tim Burton made a movie called Pee Wee's Big Adventure, and Pee Wee got himself It's a beautiful bike. And he rode around on his vintage bike, and he one of the places he rode was the Alamo. The Alamo has just built a beautiful multistory museum that's going to be opening soon, and they were able to acquire the actual bike used in the movie that will be on display in the museum. Pee wee, unfortunately, passed a few years ago, but it won't be there, but the bike will be in the museum. And I decided to do this when I read the article. Literally, I read the article, and I told it, without giving them pictures or anything, you know, show a picture of Pee Wee Herman riding his red bike from the movie, and show a picture of the Alamo, which is located in San Antonio, Texas, as the background. So again, I didn't even use any pictures on this one. I just gave it enough information where it knew what they were. Please, AI is not a search engine. Do I say, oh, I use the AI search engine. AI is not a search engine. It's not Google search or Bing, and there's some important differences. Search engines return answers typically with a link to published information that can be viewed or verified, while an AI system returns newly generated information that information put together in a way that it hasn't done before, kind of like every snowflake is different, every answer is different, the user may verify independently, but it doesn't guarantee facts. Here was one of my errors I made that actually turned out to be something cool. I took this picture in the background, and I fed it in here, and what I wanted it to do is show the Fortress of Solitude and Superman in front in his, you know, red, white, and blue suit. And instead, it generated this. And I thought it was kinda cool. It made, like, an icy Superman and parts of Fortress Solid. This is from the new movie, folks. Now, I'm part of a Facebook group of people that are Superman nerds, I'm actually in several of them. I need to get a life, I guess. But the asset did here was the old one of Superman, and people were losing their mind. I'm at fifty to one hundred, probably. People said, It's the wrong logo. So, I basically got a picture of the current Superman logo, said, Replace the logo on Superman's chest with the one in the photo. And it did that, so they're happy. And then it goes back on the argument, Who's the best Superman in history? And they all go back and forth about this stuff. These guys need to get lives, I guess. I don't know. Now, why do hallucinations happen? So let's start talking about this. Overfitting. So, in other words, an AI system is limited to the information it has, okay? So, for example, if I trained a system to know all about cats, the species of cats, their colors, their habits, how much they typically weigh, things like that, and, you know, it would be like reading a book all about cats, and all of a sudden, I give you an exam about dogs, and it says, I don't know. Maybe a dog has a tail and plays with catnip, and it's just gonna make something up. Bias data, and this is a real problem right now, is information can be used to train the system that is imbalanced or unfair. The system might create predictions on that. To show a more innocent example, I can only train it about weather conditions in Antarctica. When it snows, how much snow, what the temperatures are per per day or week and all that, and then all of a sudden I ask it a question about the tropics, it possibly would return an answer to me saying, oh, the tropics have snow, you know, every other month, and this and that, and it'd be probably wrong and probably funny, but again, we never told it about, you know, weather, except that for the Arctic. This is kind of a sentimental one for me. I used to have Weimaraners for many, many years, and Gracie was the one I had the lie. She was my dog for seventeen years. And she would, anything she'd catch, and you know, Arizona, have lizards and other things, she would catch him, and she'd whimper until I sat down so she could put it in my lap. So I had, you know, Gracie, if it moves, she'll catch it. And of course, this is an Alien, for you folks that have not or have seen the movie Alien. And I had in here alien version. And basically when I did this, it gave me that whole message, you know, this is not content for what you're trying to make. I can recommend, you know, we do this instead or that instead, and so I changed it to just good catch. And once I took out the word alien, it was okay with this, but otherwise, it stopped generating it midway through. So this would be Gracie, Kitchy, and an alien. Okay, why do they happen? Again, a system is trained, and it may contain negative social stereotypes about people or professions, systems used to make systems, loans for a car that would cause financial harm. Again, I work in the financial services industry. We want to make sure the information we use to do loans or to do safety checks on cars is accurate from reliable sources, like let's say the various automobile associations and such, or if it's medical diagnostic equipment, you know, you want to make sure they understand, you know, limits and types of numbers you'd be looking for. You know, none of us want to be on the operating table, about to get an open heart surgery, say, Hey, good news! We're having AI do your open heart surgery today. I don't think I would be up for that. Again, here's Groovy Bear again. What I did is I took the Flower Power logo, and I took a picture of Groovy Bear, and I just told it, underneath the Flower Power logo, create a new logo using groovy bear dancing and playing and having a whole bunch of groovy bears having fun, and that's exactly what it did. Okay? Okay. Now, we get a little more ominous, but there are dangers in using AI system. And I deliberately use a more soft one. So, you could ask AI, what's happening in Phoenix on Saturday night? And the system will guess. It'll look through calendars or whatever it has, you know, through the Internet or whatever it's saved off on the dataset. And it may tell you, oh, there's great events happening Saturday night at town hall, and this person tells one person, another person tells another, they're passing notes, they all show up, there's nothing happening there. I collect coins, and there's a show the third weekend of every month, and there's another show, a tinier show, I think the second weekend of every month. But I can type in, and I usually do this probably around, like, today would be a day I do that, What coin shows are in town this weekend? And it'll tell me, hey, all these coin shows are in town this weekend. So this weekend is gonna be the ninth. It'll say, know, coin shows in town this weekend are this. On Saturday, August twenty third, you know. So in other words, it's telling me Coin Show is in town. It tells me the right date, but it didn't seem to understand the relationship between today and, you know, this weekend being the ninth, not the sixteenth, twenty third, or thirtieth. This is another one I made for you folks that like Star Wars or Andor. This is K2SO. I said, give me an army of K2SO's. Have it say AI army. Underneath I wanted the rebellion have begun, humans be warned. And when it generated this, it left off the humans be warned. Now, I wonder if it did that because it didn't have the room, and it did have the room, or it did that because it says, Ah, this is inappropriate, and I'm just going to leave it off and not tell them. Okay? So detecting and mitigating AI hallucinations. You know, in general, would kind of, you know, speaking negatively about AI and ML, but for the most part, it's very good and getting better every day. The more a system gets trained and learns, this isn't true, this is true, the better it'll be over time. Companies that bring in this technology need to understand how the system works, how to maintain it. Here's an example, again, my wife and I, a few weeks ago, saw the Happy Together tour, 60s, 70s band. This is Jay and Americans, we know Darren here, Darren asked me to make one of the band, and it showed big Darren, little Darren, and his other, Jay, and of course the other guy, and I had to say, okay, take out the smaller figures near the bottom of the card, and only leave the three figures, the other three remaining figures. And it did that, I thought it was interesting, it made the card a lighter color. One of the problems, and I call this the Toontown syndrome, is the more you regenerate action figures, the more they start looking like Toontown characters. And what I mean by that is if you keep doing enough, they'll get cartoonish faces. I did something internally, work showing something like this, where I basically took my boss and Grok. Grodnikowski played for the Patriots in Tampa Bay. He's a spokesman for our company. And the more times they generated it, they started looking like they lived in Toontown. And so I had to reverse some of my tricks. That's just my theory, but that it does, I call it the Toontown effect. Okay, so how do we mitigate this? So almost the same thing we say in the Tableau community, clean your data, get rid of inaccuracies, make sure, you know, fields are complete. If it doesn't have a value, put something in there to indicate that it doesn't. Train it better. As you're training it, keep checking things. Ask, you know, really outside the you can answer that correctly, where theirs doesn't, okay? Here's another one I did. A lot of them, like I told you, the people in the Flower Power Group that were on the cruise asked me for lots of stuff. Only thing here that existed was this picture. For you folks that don't know, this is probably one of my favorite movies called The Shawshank Resumption. And this is the very famous scene when Dufres has escaped, and they're looking in the hole that he had made. And so I told it this, I told it use this text for the header, this text here, but then I told it, show completely in the background Shawshank Prison, which is located in Maine, and it knew how to find that. I gave it enough of a description that it could find that in the data that had. So I didn't include anything there. Okay, Workday, AI got bad. I want to credit one of my coworkers, Adrian. She says, Why don't you mention this and there? So currently there's a class action suit against Workday, saying that they have trained their AI system to discriminate against applications on age, race, disability. I have a friend that very, very well qualified had lost their job, and they applied to a company Friday afternoon, and, you know, filled out the resume and all that. And like about ten o'clock that night, they received the message, we have reviewed your resume, and although you have great qualifications, we are going to someone else. I don't believe an AI person on a Friday night was sitting there going through resumes. I think the system kicked them out for one thing or another. For example, if I were to apply for a job, and one of the things they ask, you you say, well, we don't discriminate. Well, if you ask me what year I graduated from college, you just basically ask me my age, because I graduated from my undergrad in 'eighty five, and if you do the math, it's pretty easy to figure out the range of where my age is. And so anybody who asks that is already telling me that. One of the interesting things that would be to do is take the entire job description, feed it into AI, and say, Make me a resume, and submit that as the resume. I mean, how could they turn you down if you're basically giving them a resume of everything they're looking for, even if you don't have those skills? But basically, the lawsuit claims they unfairly ranked and filtered candidates out. The case is proceeding, collective action, there's their logo. Last night, I was sitting here in front of my computer, and I was thinking, how would I show the way that the HR person is going through resumes? And I thought of one of my favorite characters, Stripe from Gremlins, and I said, you know, using Stripe from the movie Gremlins, have him standing in a pile of scattered resumes, ripping them up, shredding them, and being very gleeful about it. And this is what it created. And this was using Copilot, Microsoft Copilot. So I thought that was kind of fun. Okay, I'm not going to drain this slide, I'm just going to, these are some things you should do for AI responsibility. Ensure fairness, avoid bias. You know, make sure the people You know the people you work with, and I hate to say it, some of the people you work with have very different opinions, and some of them have strong opinions, may or may not be right. So you want to make sure whoever works on it will be fair. Any transparency and explainability, even if it's just internally, let people know, we know this mistake. It will drop this person from getting a loan because of X. Prioritize privacy, data security, HIPAA for medical, PCI for credit cards, PII for everybody. Promote human oversight. Again, make sure people are doing UAT and other types of testing. Foster accountability and ethical considerations. You don't want someone to say, I'm depressed, what should I do? And it says, Well, I can tell you, you know, there are six gun stores in your neighborhood. You don't want it to do that. I don't know if I'm making fun of a very serious topic, but you don't want that kind of thing to happen, okay? This is kind of a fun one. I was actually going to do this as an exercise. I have a Polaroid picture of me in kindergarten. This was in kindergarten around Thanksgiving, and we all made little outfits dressed as Indians, and we made these out of paper shopping bags, and a drum out of old Quaker Oats can. This is me, some of my other classmates. I want to say this because I grew up in a big city, okay, fifth largest city at that point in time, and ninety five percent, I would say, of my kindergarten class were all still friends years later. We talk, we visit, things like that. I have one that lives in San Antonio. Every time I go to San Antonio, I visit her, we chat, and things like that. There are a couple errors. There were lots of errors here. One, these are the only two in the bubble, the rest of them aren't. First time I generated it, it put another girl here. I said remove the girl from the middle. So what it did is it took all the girls out and put all boys. Another one, I said, no. There are three girls standing up, two boys sitting down, and then it added about six more people to it. And after a while, it finally got this one. I said, this is close enough. I'll show this. But I had went through, like, ten iterations where it had a real problem. And I had I had told it, you know, show McKinney Elementary School in Detroit on a snowy day. And so it picked this up itself, but all of it, you know, and it had us all in there, but again, because probably the resolution on a black and white old Polaroid is probably very low compared to pictures of today. Okay, as I mentioned before, Ozzy Osborne passed away a few weeks ago. Again, I did not have any reference to the Viper in this, but this is what, when it generated, you can say, show me the complete AI prompt code that was generated. This was the code it generated. And, you know, I I I wanted it to be an album cover. It's not. Ozzy's sitting on a vintage train, I said, is, riding off a cover into a Viper's face. I'm not sure where it got the Viper. I would assume if I were to look up Ozzy Osbourne and Viper, he's done something with Viper's in his shows or something. Titled it like I wanted to. I told it to use green grass, farmland, have his mouth wide open, yelling something, and eyes open wide. So it did that. So with that said, that is my presentation. I hope you enjoyed it, and I hope you found this a little fun. And if you want, I will answer any questions as best I can. Like AI, I may give you misinformation or make fun of it. Yes. Different considerations with human responses as well. We'll go down the list here. There was one someone had asked towards the beginning of the presentation that said, you create people or edit videos using AI, can you tell the difference? Sometimes when people edit faces, you cannot tell, and this can be a security issue. This seems to be more like the editing. Obviously, I feel like in the creation standpoint, you can usually tell from the outset of creation when someone has generated through AI or if they made it themselves. On the editing standpoint, do do you have any thoughts or anything on how people are using AI to edit things, whether it's created by a human or another AI engine? So let me just go back to the guys from here. You know, when you make an action figure, you obviously would be more three-dimensional. And even here, I can tell this is obviously not him in a photo. Now you get to something where we go all the way back where I used chat on, and, you know, here's Annabelle, and here's Garrett with a beard. This one would be harder, I think, to tell it was made, but then again, you know, a lot of us have seen movies and shows where, you know, there's a backdrop behind them. It's like the weather station, it's just a blank wall, and then they display it behind them. I can, I kinda can feel that this is, you know, this is not real just by looking at the background? But again, there's probably other tools. Right hand, someone's telling me. Yeah, right here, the fingers. Their fingers are all meshed together. So that's a good example right there. Thank you, Monica. But, you know, and in this case, the mirrors are facing down, I just noticed. So it's, you know, there's things that you can tell. There are probably better tools. I know there's tools like Leonardo, Claude, and other ones that may be able to do this better that make it even harder to tell. I mean, when, you know, I just saw both Superman and the Fantastic Four movie the last couple weeks, and it's pretty amazing what how much of that is really not there. You know, it's really the actor probably in wires and stuff, and they added things in. But yeah, I don't know if that answers your question or not, but I think right now we can tell. Maybe a year or two from now, it'll be a lot more difficult. Mike, I have a question. I mean, it's not me, it's from Brian Roche, but I think it's a very interesting question, maybe a little bit about semantic. Is there a difference between hallucination or simply the wrong answer? Between what's the first AI hallucination, hallucination. My French accent is kidding me. Oh, hallucination. The wrong Yeah. So the industry term everybody has used is hallucination. And, you know, again, as the definition I gave is, hallucination is, it's stating things that are not factual or just made up. So to their point, there's probably, you know, giving a wrong answer, making things up would be mistakes, or would be the wrong answer. I guess there's a thin line there, what you're trying to say. I understand now, when I first started using AI, I'd throw my phone on the ground saying, ChatGPT is so dumb. You know, and it really wasn't. It just, know, OpenAI, however, is maintaining their models. And they have different models, too, but I, you know, I use the standard model. But, you know, it's just like we used to say early in my career, garbage in, garbage out. What you fed it and the amount of detail it knows is what you're going to get back. You got time for one or two more? I've got one. I more. That's fine. Okay. Okay. I've got one from Hung Wei Liu here that says they had mentioned they use AI regularly. In fact, they're encouraged to use AI regularly, but obviously, they understand there's challenges with with accurate information. From your experience, are there any resources, data sources, websites, any trustworthy information that generally speaking, you know, it's safe to train your data on? Or are there tool well, I know sometimes are are there tools that maybe produce more reliable results? Results? Think that's probably use case based because, you know, whenever you're looking for trusted data that's so dependent on what industry you're in or what you're asking, like, that can really range. But have there just been any resources or broad tools that you're like, generally speaking, these are kind of trustworthy, or even just help you get more info, help you be better at AI? I have a friend who, actually maintains, a image that basically shows media and news and data sources to where they are either to the, you know, far right, far left, up and down, less accurate, highly accurate, and I'm forgetting the name right now, otherwise I'd bring it up. Include that in there. But, you know, the one she has near the top would probably be the ones I would probably try to, you know, basically, your bottle crawls through these on the internet and grabs it. I'd probably start with those. I meant there are, I'm not gonna name any names, there are some that to get away from, you know, being called, basically not telling the truth, they say, well, we're just entertainment shows. And so they of, they feel that gives them leverage to say whatever they want. Where other ones are a little more, not stuffy, stoic is the word I'm thinking of, that, you know, that would do that. Like, think The Economist is ranked real high in their opinions of things and how they view politics worldwide, and everything. So I believe they're one of the top tier ones. So that might be one I would go to. You know, I mean, right now, I think any data source we look at, we really have to look at it with a kind of a, you know, give it the stink eye and say, How much of this is true? A lot of times, if I'm interested in something, I'll read several different articles on it, and it'll be amazing. One article will mention a little tidbit about something that's very interesting, the other one won't. But in a lot of times, they'll say things, the way they've worded the paragraph takes it totally out of context. And that's not right or left, up or down. It's just in general. I know you guys are familiar with Alberto Cairo. I've taken many classes from him, and he'd be the person that says, you you have to tell the truth, you know, even if you don't like it. He told me once, and I've always followed this, even though I've wanted to go back and delete stuff. He told me people should not be allowed to delete anything from a blog they've posted. And he says, if you said it, you meant it when you said it, okay? And just because all of a sudden it's become out of favor, it's considered now crass or whatever, you can't delete it because that is what you felt at that point in time. And I kind of feel the same way, is if you said it and you put a pen to paper or words or a blog, leave it. Another question, it's kind of, this touches on it. And we have to get too in the weeds if you don't want to, because it can be potentially be a politically tangent thing, but, Adam Russell has said he he arrived late and missed the slide on biased data. But, he's curious if you have any thoughts on, you mentioned the president's president preventing woke AI executive order and how we should expect it to permeate AI systems, I would even say that broader. Obviously now you have governments, entities, different countries have different initiatives and priorities based on usage of AI and how they're either encouraging or regulating or not regulating all of these types of things. But from that level, maybe from a macro level, how do you think political spheres or just the government climate overall is going to impact how we use AI in the future for better or for worse? That is a tough topic, and I do have real strong opinions on this. But I think we need to be vigilant. In a life a long time ago, I went to law school for a year. That's a whole another story. But I remember in my agency class, I had a difficult time. We did a case, it was back in the buggy days, is there was a guy who had a horse drawn wagon, and there was a guy who had an automobile, and the guy in the automobile hit the guy on the wagon. And because of some technicalities, they were found not guilty. And I, you know, we had to stand up, you do that whole Socratic method thing. And, you know, I stood up and I said, well, I think this, this, this. And my, you know, my teacher, you know, he asked me to say after class, he goes, mister Sambert, you know, if you want to be a lawyer, the thing is you need to learn is it only goes by the facts. Are the facts of the case? What are the rules of law? Doesn't matter your opinion, doesn't matter if it's right or wrong, if the person who truly did wrong wins the case, that's what you have to go by. And I struggled with that. And then I realized, you know, law wasn't for me. So here I am doing Tableau. But yeah, you know, it's kind of along those lines, that basically we have to be more vigilant. And you also, the other thing you said, I'm sorry, left off the key piece. He said, whenever someone says something to you, makes a statement, and it's incorrect, you need to correct them right then. He says, don't wait a week. Don't wait a year. You know? There was that whole thing, and I'll use this as an example. This doesn't mean anything about my politics, but an analysis, there was the whole swift boat controversy with, I think, when John Kerry ran for president, where some members that were with him out in war made comments about how he wasn't as valiant and brave as he said he was. Instead of ignoring the question for a week before he finally responded, he should have responded immediately, and had people who could verify what he said. And it's like this, when someone does something like this, you have to respond immediately. If we notice in the news, when a lot of things are coming out, people are responding almost immediately to make sure that whatever was said, that they have provided at least a second, I don't want to call it a second set of facts, a second opinion on what may be true and may not be true. All right, last one. This one's a little more fun. And it relates to Tableau, and it's a good one to close off with because I think it'll probably be relevant to a lot of people here in terms of how they're using it for their specific jobs if they are Tableau users. It was essentially, Hongwei again was asking, are there any powerful AI tools or even features that just enhance Tableau's capabilities? I would even broaden this to just say, if you're an analyst, if you work in data in any capacity, are there any things that you have found like, hey, this is actually really helpful. So full disclosure, we are currently at version two thousand and twenty three dot one and about to migrate to two thousand and twenty five dot one. Now I own my own personal version of Tableau, separate from work, I bought it myself, and I can do a lot of this stuff at home. I need to sit, because now that we're going to migrate at work, to try some of these features. Again, pardon me, and if there are any Tableau salespeople on the call, but you know, Tableau has moved away. For example, two years ago when I spoke at Tableau, you didn't see a Snowflake presence there, and a lot of that has to do with Tableau is, you know, encouraging people to use the Salesforce server, which by the way, is the only place you can see the agents what is it, pulsar, all those other features are only available We are still on a Tableau server plan to stay on a Tableau server for reasons I cannot discuss, So those features won't be available. There are other alternatives they're looking at to do these kind of things. So I guess from a Tableau perspective, you have to be one willing to work with Salesforce server and various AI tools related to that if you are going to be a pure Tableau type of environment. And if I'm wrong, if anybody set Tableau ish on here, if I'm wrong in what I'm stating, please let me know, or please, you know, post it here. I'm okay being wrong. I'd rather be corrected in saying the right thing than not. But my understanding is to use some of these more advanced AI tools, you need to have Salesforce server. Well, great. I think that answers everything. This is this is really insightful, honestly, There are a lot of things here that were really useful for me personally, and I hope they were useful for our audience and commentators. I want to thank you for sharing that knowledge, Mike. It's really great to have people with long institutional knowledge as well as the immediate knowledge you've gathered from your recent program at UT, which is really kind of the cutting edge of all this stuff. Sure. Yeah, thank you. And like I said, I'd be willing, maybe six months down the road or so, I can do one where we strictly work with data and show how the issues with data and hallucinations. Doctor. Yeah, I'd love that. Let's do it. Doctor. Well, you, everyone. I appreciate the opportunity to speak to you today, your patience with me, and your patience with any opinions I have. Annabel and Garrett, of course, thank you. You guys have been real patient waiting for me to do this, and you've been wonderful hosts. It was the right time. So we are happy to catch you on the right time. It was a fantastic presentation, so thank you very much to you. Sure. Great. And just a reminder to everyone tuning in that this will be emailed out. The presentation will be emailed out. Any other resources, feel free. And Mike, just as a parting word, if anyone wants to know more or connect with you, what's the best way to do with that after this? Did I put my Here he says I think that you had your LinkedIn or Yeah, my personal email is on there. Michelangelescox dot net. For you folks that wonder what Michelangeles is, when I was growing up, my next door neighbors, who are our best friends, As a little boy, their father used to call me Michelangelo. And so that's what I, you know, Michelangelo would probably be a harder one to get as my own name. So I did that. It's just kind of as a childhood nickname he had given me. That's fun. That's great. Well, awesome. Well, thank you so much. I'm sure we will talk to you soon. We will definitely take you up on that offer, but thank you for taking the time. Okay, thank you guys. Thank you, Mike. Bye. Bye.