Video title: Ambiguity and confirmation bias in reward learning Video URL: https://www.youtube.com/watch?v=Fagr6lo7kNs Ann은 원래 약간 비관적인 성향입니다. 어느 날 파티에 가는데, 별로 기대를 하지 않습니다. 일이 잘 풀릴 거라고 생각하지 않죠. 그래서 분위기를 풀려고 농담을 하나 하지만, 사람들이 크게 웃습니다. 그런데 Ann은 사람들이 자기 _비웃고 있다_고 느끼죠. 결국 상처를 받고, 다시는 파티에 가지 않겠다고 마음먹습니다. 이번엔 Bob입니다. Bob은 타고난 낙관주의자라서, 상황이 대체로 잘 풀릴 거라고 생각합니다. 그도 파티에 가서 농담을 했는데, 모두가 크게 웃었습니다. Bob은 “와, 내 농담이 통했네! 내가 꽤 괜찮잖아!”라고 생각하고 기분이 좋아집니다. 다음에도 또 파티에 가겠다고 결심하죠. 즉, _Ann_과 _Bob_은 정확히 같은 경험을 했음에도, 완전히 다르게 해석했습니다. 이건 여러 이유로 중요합니다. 그들이 앞으로 어떤 _위험을 감수할지_—즉, 또다시 시도할지—에 영향을 미칩니다. 그리고 초기 기대가 서로 달랐기 때문에, 같은 사실을 각자 다르게 받아들였던 겁니다. 이런 현상이 바로 **확증 편향(confirmation bias)** 입니다. 인터넷에서 ‘confirmation bias’를 검색해 보면, 대부분 이런 그림을 볼 수 있습니다 — ‘객관적 사실’과 ‘기존 신념’을 겹쳐서 본 결과가 곧 우리가 _보는 세계_라는 뜻이죠. --- 확증 편향은 보통 나쁜 평판을 듣습니다. 인간 추론에서 발생하는 가장 유명하고 널리 알려진 오류로 여겨지고, 과학·의학·정치·법률 등 중요한 분야에서의 **편협함(closed‑mindedness)** 의 원인으로 비판받습니다. 그래서 사람들은 “확증 편향은 위험하고 악성인 오류”라고 생각하죠. 하지만, 만약 이 편향이 _적응적(adaptive)_ 일 수 있다면 어떨까요? _통계적(Bayesian)_ 관점에서 보면, 그것은 오히려 합리적일 수 있습니다. 즉, 불확실하거나 모호한 상황에 직면했을 때, 기존 믿음을 바탕으로 그것을 해석하는 것이 합리적일 가능성이 있다는 것입니다. 컴퓨터 과학에서는 이를 “귀납적 편향(inductive bias)”이라 부릅니다 — 학습 과정을 효율적으로 시작하게 도와주는 일종의 ‘사전 방향성’이라는 뜻이죠. --- 그래서 우리는 _확증 편향이 실제로 적응적일 수 있는가_를 살펴봤습니다. 이 주제는 여러 영역에서 이미 다뤄졌는데, 예컨대 사람들이 자신이 이미 가진 가설을 확증하는 정보를 더 찾아보는 정보탐색 과정, 또는 _지각(perception)_ 과정에서 애매한 자극을 기존 신념에 따라 어떻게 해석하는가 등이 있습니다. 하지만 **보상 피드백(reward feedback)** 처리에서의 확증 편향 연구는 상대적으로 적었습니다. 특히, 우리가 집중한 것은 **선택한 행동의 결과와 선택하지 않은 행동의 비교(선택확인 비대칭, choice‑confirming asymmetry)** 가 아니라, 그냥 **애매한 결과 그 자체를 어떻게 평가하느냐**였습니다. 즉, _Ann_과 _Bob_의 사례처럼, 동일한 사건을 낙관적이거나 비관적으로 해석하는 현상을 관찰했습니다. --- 우리의 연구는 두 가지 목표를 갖습니다. 1. 애매한 긍정적 결과를 해석하는 방식이 확증 편향을 반영한다면, 그것이 **적응적 기능을 갖는가**? 2. 이러한 편향이 **개인의 낙관적 성향(optimistic disposition)** 과 연결되는가? 이 현상은 여러 기존 연구와 맞닿아 있습니다. 우리가 주목한 것은 **정서적 불확실성(valence ambiguity)** — 결과의 ‘크기’는 알지만, 그것이 _좋은지 나쁜지_는 모르는 상황입니다. 다시 말해, 파티에서 사람들이 웃었는데, 그게 _비웃음_인지 _공감의 웃음_인지 모르는 상황과 같죠. --- 이와 관련하여, 뇌는 보상의 크기(강도)와 정서적 방향(좋음/나쁨)을 따로 인코딩한다는 신경과학 연구가 있습니다. 임상적으로도, 애매한 정서 자극(예: 놀란 얼굴)이 _기쁜 놀람인지, 놀라운 불행인지_를 어떻게 해석하는지가 불안이나 우울 같은 임상적 특성과 연결된다는 결과가 있습니다. 불안한 사람들은 일반적으로 **부정적 해석 편향(negative interpretation bias)** 을 보입니다 — 즉, 애매한 자극을 더 부정적으로 해석하는 경향이 있죠. --- 우리의 연구는 두 가지 기여를 합니다. 1. **정서적 불확실성(valence ambiguity)을 포함하는 새로운 다중선택 과제 (multi‑armed bandit task)** 를 설계했습니다. 2. **베이지안 학습 모델**을 새로 구축하여, 이런 상황에서 인간이 어떻게 학습하는지 계산적으로 설명했습니다. 이는 다음과 같은 사고를 기반으로 합니다. 한 결과의 일부 정보(예: 크기)는 알지만, 나머지(정서적 가치)는 모를 때, 사전 믿음(prior belief)을 활용해 그 ‘빈칸’을 베이지안 방식으로 추정한다는 것입니다. --- ### 🪙 실험 개요: “황금 채굴자 과제” 참가자들은 서부 시대의 금광 채굴자로 설정됩니다. 두 개의 금광 중 하나를 선택해 채굴을 하면, 보상이 주어집니다. 예를 들어 왼쪽 금광을 선택했더니 **+10**이라는 결과를 얻었다면, 이는 “금 10단위”를 얻었다는 뜻입니다. 또 다른 금광을 택했을 경우의 _반사실적 결과(counterfactual outcome)_ 도 함께 보여줍니다. 예를 들어 “다른 금광을 택했으면 −11점을 얻었을 것이다”라고 하죠. 이때 참가자들은 여러 시도를 반복하면서 어떤 금광이 평균적으로 더 유리한지 학습합니다. 여기까지는 일반적인 강화학습(paradigm) 절차입니다. --- 하지만 일부 시도에서는 **애매한 결과(모호한 광석, dirty ore)** 가 등장합니다. 예를 들어 “6단위의 무언가를 얻었다”는 정보는 주어지지만, 그것이 금인지 돌인지 알 수 없습니다. 게임이 끝나면 실제로 어떤 것이었는지가 결산되어 보상에 반영되지만, 학습 과정에서는 알려주지 않습니다. 그래서 참가자는 **자신의 사전 믿음과 현재까지의 정보**를 바탕으로 이 결과가 _금인지 돌인지_ 추정해야 합니다. 이때도 “다른 금광을 골랐으면 무엇을 얻었을까?” 하는 반사실적 정보는 같이 보입니다. --- 참가자의 **사전 믿음**을 유도하기 위해, 각 블록 시작 시에는 **가상 술집(saloon)** 장면이 나옵니다. 그곳에서 카우보이가 이렇게 말하죠: - “이 지역엔 금이 아주 많대!” (긍정적 조건) - “이 지역은 거의 돌뿐이라네.” (부정적 조건) - “모르겠어, 잘 모르겠네.” (중립 조건) 즉, 실제 확률은 바꾸지 않고 단지 **기대(prior)** 만 조작하는 겁니다. 실제 환경은 항상 한 금광이 평균 +10, 다른 쪽이 평균 −10으로 고정되어 있습니다. --- ### ⚙️ 모델링 이 연구는 **베이지안 강화학습(Bayesian reinforcement learning)** 모델을 기반으로 합니다. 기본이 되는 _Rescorla‑Wagner 모델_은 ‘보상예측오차(reward prediction error)’에 따라 보상 추정치를 갱신합니다. 베이지안 버전에서는 이 갱신 크기(학습률)가 불확실성에 따라 자동으로 조정됩니다. 즉, 사전 믿음이 모호할수록 실제 데이터에 더 큰 가중치를 두는 식이죠. 하지만 우리 과제에서는 **애매한 결과**가 존재합니다. 예를 들어 “더러운 광석 10단위”라는 결과를 받았을 때, 플러스인지 마이너스인지 확실히 모르는 경우, 모델은 그 결과를 베이지안 평균치(예상 보상)로 보정(impute)합니다. 즉, 기존 신념이 긍정적이면 그 결과를 좀 더 +10 쪽으로, 비관적이면 −10 쪽으로 해석하도록 합니다. 또, 그러한 불확실성을 반영해 **추가적인 분산(uncertainty)** 을 계산에 포함시킵니다. 50대50으로 모호할수록 불확실성이 커지고, 결과의 크기가 클수록 그 영향도 커집니다. 수학적으로는 계산 편의를 위해 “가우시안 근사(assumed density filtering)”를 적용했습니다. --- ### 📊 결과 요약 - 참가자들은 실제로 학습을 잘 했고, 모델이 학습곡선을 잘 예측했습니다. - “이 결과가 금이었을까 돌이었을까?” 라는 **신념 보고(stated belief)** 를 보면, - “금이 많은 지역”이라고 들은 조건에서는 금으로 해석하는 비율이 높았습니다. - “돌이 많은 지역” 조건에서는 반대로 낮았습니다. - 중립 조건은 대체로 금 조건에 가까웠습니다. - 베이지안 모델은 이런 조건 간 신념 차이를 정확히 재현했습니다. --- 더 흥미로운 것은, **중립 조건의 신념 곡선**은 실제 금의 확률과 거의 일치했다는 점입니다. 즉, 조작으로 긍·부정 기대를 줬을 때는 초반에는 편향이 있었으나, 학습이 진행됨에 따라 점차 실제 확률에 근접해 갔습니다. 이것은 “초기 낙관/비관적 믿음 → 경험 데이터로 점차 교정”이라는 확증 편향의 _적응적_ 작동 방식을 잘 보여줍니다. --- ### 🌞 개인차와 낙관성 이제 개인차를 살펴봤습니다. 참가자들의 **낙관성(optimism)** 은 _Life Orientation Test (LOT‑R)_ 로 측정했습니다. 이 점수는 행복감, 건강 등과도 밀접하게 연관된 심리척도입니다. 분석 결과, 낙관성이 높은 사람일수록 **애매한 결과를 “금일 가능성이 높다”** 고 해석했습니다. 하지만 객관적 정확성에서는 차이가 없었습니다. 즉, _낙관적인 사람들은 동일한 애매한 정보를 더 긍정적으로 해석했다_는 것입니다. --- ### 🧭 결론 1. **정서적 불확실성 하에서의 학습 과제와 계산 모델**을 제시함. 2. **확증 편향이 단순히 오류가 아니라, 적응적 역할을 할 수 있음**을 보임. - 베이지안 모델은 참가자의 실제 신념 변화를 잘 설명함. - 시간 경과에 따라 사전 신념과 실제 경험이 일치해감. 3. **낙관적 성향은 긍정적 해석의 강도와 상관**이 있음. 즉, 낙관주의는 일종의 **“사전 확률(prior)”** 로 작용해, 모호한 경험을 더 긍정적으로 해석하게 만들고, 이는 곧 학습과 행동 선택에 영향을 미칠 수 있습니다. 따라서, 확증 편향은 완전히 비합리적인 오류가 아니라 “베이지안적 학습이 작동하는 자연스러운 형태”일 수도 있습니다. 우리 실험과 모델은 이 가설을 탐색할 수 있는 새로운 틀을 제공합니다. as you mentioned this has worked with the Sam gershman and Haley Dorfman who is now a postdoc with Liz Phelps at Harvard so let's see if I can move my slides meet and and is going to a party and she is a little pessimistic by nature and so she doesn't expect things to generally go well but she decides to tell a joke and everybody laughs and they laugh and things very cruelly at her and Anna's sad and she decides to never go to another party again meet Bob goes to a party Bob is kind of optimistic by Nature so he thinks that things generally go fairly well so he decides to tell a joke and everybody laughs and Bob thinks wow they're laughing at my joke Bob is cool I love this and I'm going to the next party again now Ann and Bob basically had the exact same experience they saw the same data and yet they interpreted that in totally different ways this is important for a number of reasons right it clearly affects their propensity to take risks in the future it affects whether they're going to go out and have that experience again and possibly disconfirm their initial beliefs but because they had these different expectations to begin with they end up interpreting that data in different ways now what happens here is basically a confirmation bias and if you Google confirmation bias you'll see a lot of figures that look like this where it means what you see is basically the intersection between the objective facts and what confirms your prior beliefs here so confirmation bias it gets a very bad rap some say it's the best known and most widely accepted notion of inferential error to come out of the literature on human reasoning and it's been blamed for closed-mindedness in Fields as hopefully important and Broad as science medicine politics law so people people hate confirmation bias right they think this is this terrible nefarious pernicious kind of bias that we should never have but what if confirmation bias could actually be adaptive what if it could have some useful value to it and if you think of it from a statistical perspective from a Bayesian perspective it could actually make sense that is to say when you're faced with some uncertain or ambiguous cities it makes sense that you should interpret those in light of your prior beliefs so if you think of it in computer science terms they might call it an inductive bias right something that helps you to jump start your learning process so we have some reason to think that actually it could be adaptive and we've seen this in a few different domains so far some have studied confirmation bias that might be adaptive in the way we acquire information that is do we seek out information that confirms our pre-existing hypotheses or in the interpretation of perceptual data so when you have ambiguous perceptual stimuli how do we interpret those given our prior beliefs uh we're looking at the processing of reward feedback now so far there's only been a little work on what are called Choice confirming asymmetries and so that reflects how people seem to place greater weight on positive outcomes for actions that they chose and less weight on positive outcomes for actions that they didn't shoot so the sort of counterfactual outcomes here we're looking at something different we're looking at how you appraise these ambiguous outcomes themselves uh irrespective of whether that's what you chose or what you didn't choose so more like our Ann Bob situation to begin with and we're going to provide two things here we're going to have a new model and we're going to have a new task so we have two goals here first we want to figure out is the way that we interpret ambiguous rewarding outcomes if that's confirmation biased is that potentially adaptive and second can we connect these biases to individual differences and in particular we'll look at differences in one's optimistic disposition now this relates to a number of literatures and we're particularly looking at a setting where you have what we call valence ambiguity and that means the magnitude of the outcome is something that you know but you're not sure about the valence you don't know whether it was good or bad and this is just like that and Bob situation where you saw uh there was some response some event that happened where everybody left but you weren't sure whether that was in a good way or a bad way and so this links with a few different literatures uh one that has to do with the way that we encode valence and magnitude perhaps separately in the brain uh also there's there's neuroscientific work uh and clinical work on the way that we process ambiguous rewards and Valence ambiguity in particular has been studied in the context of emotion processing so if uh I show you a face of somebody who's surprised was that a good surprise or a bad surprise right were they surprised because of some positive experience or a negative experience and the way that we interpret ambiguous outcomes has also been linked to uh clinical traits and clinical outcomes so for example people who are anxious tend to have what's called a negative interpretation bias where they tend to read into these negative stimuli negative faces negative experience or ambiguous stimuli experiences in a more negative fashion so we have two contributions here first is we are providing a new uh a multi-armed bandit task a new experimental Paradigm uh to incorporate valence ambiguity and second we're proposing a new Basin computational model of how you maybe should learn under this valence ambiguity and it captures this idea that you have almost a Bayesian missing data imputation that is to say if you know part of something like the magnitude of some outcome and you have some prior beliefs you should use that to actually guide your assessment of what the valence is so the way that you fill in the blanks should be consistent with uh Bayes and statistical principles here so in our experimental Paradigm we turn to the wild west as we've done before in a previous paper and in this task people are in the role of gold miners in the wild west this is a very uh an interesting way of presenting a classic multi-armed Bandit Paradigm that's been used in a number of different fields here so participants are in the role of this minor where they have two different gold mines that they can choose to dig from so in a given trial they have to pick one of these once they pick one they get an outcome here so I picked the one on the left and we show them you got a reward of plus 10 which translates into a positive amount of money for them and we show that here as you got 10 units of gold so gold is good we also show them the counter factual outcome here so that's in this case if you picked the other mind you would have gotten minus 11 points we did this so that we can shut off any exploration exploitation and just focus on interpretation of these outcomes and learning here so you took the action you got the outcome and then you go back through this again and again right classic reinforcement learning type approach so far that's pretty standard here's the modification we're making on some trials when you dig you get something that looks like this and this is dirty ore it's an ambiguous outcome where you know that this was six units of something but you don't know whether it was gold or rocks because it's covered in dirt and you'll still get paid at the end of the task based on whether it was in fact gold or rocks but you will not be told which of those it is and that means you have to make your best guess based on any information you have to date any prior uh pre-existing beliefs as well as the magnitude of what's given to you here and again you get the counter factual outcome it was about 50 of the trials where we give them these ambiguous outcomes and always for the option that they chose and then we asked them do you think that this was Rockstar gold so we had a stated belief question here and that tells us more specifically what actually were your beliefs about this ambiguous stimulus now in order to uh try and influence their prior beliefs going into a particular round we started off each block by putting them into this sort of virtual salute and in the saloon they encountered a cowboy who told them that he'd heard a rumor that maybe there's a lot of gold in this terrain right it might be a very rich environment or he said well I hear a rumor that there were a lot of rocks not really a great place or he didn't really hear anything he's not really sure here so this is a neutral condition so we can try and influence them a bit to say it's a rich a poor or a neutral environment we didn't actually change the reward statistics here it was always the case that one mine had an average value of plus 10 when if the other ones had a value of -10 and then there was just some common noise across the actual outcome that they would receive in any given round so we held the context itself or the environment fixed but we just tried to change their pre-existing beliefs a little bit so just to reiterate we said that uh in each block first they were in the saloon they were given some pre-existing belief a little bit then they were given a choice if the minds to dig from they got the feedback and made a judgment about whether they thought the outcome was rocks or gold and then they cycled back to Part B the choice here and went through that again and then after about uh you know a handful of trials they would go back into another condition where they were given a different pre-existing belief here so in our model this is built on a basin reinforcement learning model so the standard rescorla Wagner model this is the most basic version of model free reinforcement learning it says that you update your estimate of rewards by taking your initial estimate and then you increase that based on this reward prediction error which is the difference between the outcome that you actually got minus your pre-existing estimate and scale that by some learning rate that says well how much are you adjusting based on that new data now if you have a Bayesian version of this then that learning rate is itself derived from optimal statistical principles and so the the weight that you place on your prior beliefs versus the data is basically a reflection of how much uncertainty there is in each of these relative to one another so if you have very very big pre-existing beliefs that means your estimate variance here is quite high you're going to update more towards the data because you didn't really know much to start with now to this point this is a standard Basin reinforcement learning model this is the the Kalman filter from engineering but when we look at our Paradigm with ambiguous outcomes well now there are some points here which are actually not so clear right what does it mean to have a given outcome that's ambiguous if I told you you got 10 units of this Derby or should you count that as plus 10 should you count that as minus 10 should it be zero should you ignore it should it be somewhere in between where exactly if so and so we say you're going to impute that ambiguous outcome with basically your best guess your mean uh our expectation of that reward given the magnitude of it so if you think that this magnitude given you know you got A plus 10 if you think that tends to be more likely to be positive than negative then you're going to treat it as somewhere in the middle of minus 10 and plus 10 but skewed more towards the plus 10 side and you can calculate that more precisely given the basic Machinery which I just won't go into very much here the other thing that you need to do is to change the uncertainty about the outcome here as well because it's not only the signal noise that says how different each outcome is in a given trial from the true mean you also have some further uncertainty caused by the um the ambiguity from the coarsening of whether you whether it was positive or negative here as well and so you can see that that's a maximally uncertain when you're totally unsure 50 50 guess as to whether it was a positive or negative outcome here um and it happens to be scaled by the reward because you can think how if you've got an outcome of like zero or one you know that it's basically plus one or minus one you know that it's almost the same it doesn't really matter but if it's plus 100 or minus 100 now you're really really unsure and you should factor that in and I should mention this model is actually approximately based and we did what's called assumed density filtering which basically projects everything onto a gaussian space because it makes the math work out really nicely and that's a lot easier than doing it other ways so what did we actually find here well first the Bayesian model 50s learning curves fairly well so people were learning in the task and this was captured by you know any of these kinds of models that we would use here [Music] we also looked at the stated beliefs right remember that was when we asked them do you think that this outcome was rocks for gold and of course uh can you see my mouse cursor here cool so of course if there's gold this is a nice attention check we know that everybody said it was gold and it was rocks barring Some Noise you said it was rocks but in those dirty outcomes people basically said well if they were in that rich condition where they got the rumor of gold in this environment they had higher stated beliefs in gold and if they were in the poor condition then they had lower stated beliefs neutral oddly seemed to be pretty close to the rich condition actually and these little circles here represent the model predictions of a bayesin model where we fit condition specific prior beliefs here so the Basin model was able to capture these differences in stated beliefs across conditions as well now this is not model uh predictions here this is just the data itself and this is pretty neat if you look at these dotted lines here these dotted lines here reflect the true prevalence of gold in each uh trial here that is to say given the option you chose How likely was that to actually be gold and this is increasing with experience because you're just getting better at the task right you're you're actually learning this is the better mine to pick I'm going to pick that more often that's why this increases here what's cool is that these solid lines represent the stated beliefs that people had and in this neutral condition this Gray Line their stated beliefs track the actual true probabilities really closely that's to say if we didn't kind of push them in one direction or another their stated beliefs are pretty accurate compared to the True Values now the other neat thing is well we did see differences between these conditions so when we said well it's a bit Rich then they they are at a higher belief State here and when it's poor it was somewhat lower but in particular that obtains more at the start rather than the finish so it's almost as if they have different prior beliefs induced by this uh manipulation but with experience they're actually converging to the truth here so that's kind of neat right this is consistent with some idea of uh confirmation bias and optimism as being prior beliefs that are then uh influenced by the data as well and it turns out that if we fit that Bayesian model with different prior means for each condition you could actually capture those differences across conditions that is to say you end up with uh differences in those beliefs at the very beginning but then those end up washed out with the experience that you're getting because that experience is pretty uh comparable of that so the Bayesian model is capturing these dated belief Dynamics fairly well here it turns out the best fitting model does seem to vary somewhat across individuals but across the board the Bayesian version of the model fit better than a more traditional riskola Wagner reinforcement learning model that just skipped those ambiguous outcomes here comes here now can we look at differences across individuals right we saw that there are these pre uh these prior differences in beliefs when we very experimentally people's beliefs but what about individual trait differences and so we looked at optimism using this well-known life orientation test provides the lotr and this is pretty well known and commonly used in Clinical Psychology to capture optimism and it's been linked to a lot of uh well-known positive life and health outcomes so it's been linked to uh you know how healthy you are in terms of certain kinds of diseases and in terms of how you know happy you tend to be so you can see how this has a link between well and things like uncertain times I usually expect the best this is a very clear link uh in at least intuitively to how you're going to interpret these ambiguous outcomes and so we found in fact that if we take the average stated belief in gold on those ambiguous trials that is positively correlated with their optimism level that is to say optimists believe that these ambiguous outcomes tend to be positively rewarding here and it's not that there was any difference across them in terms of their objective accuracy that was pretty much uncorrelated only in terms of their subjective beliefs here so optimists and facts were attributing these ambiguous outcomes with more positive valence in effect so there's more that we're digging into still but in summary we showed this or download this task that measured learning and beliefs about these ambiguous outcomes in this context of valence ambiguity uh and we found that this observed confirmation bias seemed to be consistent with Bayesian mechanisms which is important for demonstrating that's adaptive in the sense that this data beliefs seem to be accurate they were connected closely to the true probabilities we also found that the Bayesian model could provide an adequate fit of this data and we're still trying to look into this form but it could be consistent with this idea that's been proposed of optimism as priors right so your your optimism determines your prior beliefs here which could then be uh minimized with the data that you're getting here or we're still looking into this but it provides some support for the idea that confirmation bias in these reinforcement learning type settings could actually be adapted and we provide a paradigm and a model that helps us to explore this idea so that's uh my talk and I'm happy to take any questions to the extent we have time for it