24_conference1_note - amoon.world🌙

[[1BEC_24.pdf]] Photos taken shared [here](https://drive.google.com/drive/folders/1JmGs4o6uZbWMOiu0TN52xevAsMmoOy3D?usp=sharing). # Day1 ## Session 1 - What is Bayesian Entrepreneurship ### Intro: Alfonso Gambardella Objective of this conference: experiment to create 3 things 1. A community 2. A common language 1. Problem: field of management is incredibly fragmented 1. Lost of papers that do not scale 2. Papers that could amount to scale (talk about similar stuff) but do not share a common language 3. Testable implications in general, attempt to reorganize efforts Why “entrepreneurship”? * Characterized by choice Why “Bayesian”? Bayesian thinking can be the language, the standard Upsides: * Testable - Objective implications Downsides: - The lens of the tool distorts the worlds – then “offsetting” tools can give deeper description of phenomena - Myth 1: Bayes doesn’t work with events with probabilities=0 - Myth 2: People do not think in terms of probabilities High-level format 1) What is Bayesian Entrepreneurship? 2) Priors 3) Experimentation and Learning 4) Persuasion 5) Pedagogy & Practice Micro-level format: - Kick-off - Discussants - Discussion ** ** ### Lead 1: Scott Stern Zappos: * Key elements: Variety of SKUs, issues with suppliers/ logistics, bulky * First hypothesis: to sell & deliver shoes from an online store * Feeling that they had a unique view on the shoe industry (of founders) > Sequoia turned them down (dot.com bubble years) * Experiment: every dollar spent on inventory to be able to speed up delivery > to be used for persuasion in case of confirmation of subjective belief RQ: how does the fact that entrepreneurs _choose_ the opportunity they pursue impact entrepreneurial experimentation, persuasion, and strategy? * Ideas → Experiments → Strategy Problem: vast majority of papers treats each of these 3 topics differently * Ideation & Opportunity Identification * Experimentation and Learning * Strategy: go-to-market strategy, scaling [...] Choice: entrepreneurs _choose_ → Bayesian entrepreneurs _hold contrarian optimism _about the idea which (s)he chooses to pursue - ideas on which they believe deeply while other people don’t We often abstract away from other elements of the process in order to make it more tractable. Baysian lens gives the opportunity to unify these ideas Starting point: Contrarian optimism - entrepreneurs are not just optimistic about their ideas, they have reasonable beliefs that others are not pursuing/positive about this idea. What Bayesian entrepreneurship is _not_ 1. _Not_ Knightian Uncertainty 1. Cannot form neither probability distributions nor envision states 2. Cannot group events together based on their likelihood of occurring 2. _Not_ simply based on shared environmental observables 3. Econ/Finance literature: impact of external environment on ent choice & performance 4. Here: focus on choice of entrepreneur 5. Different industries make meaningfully different decisions in the face of the same decision set 6. You don’t observe the subjective beliefs that people hold 3. _Not _shared priors 7. Bayesian persuasion: focus on equilibrium agreements 8. Heterogeneous priors based on individual experience → possibility to “agree to disagree” 4. _Not _Effectuation 9. Idea of effectuation: ideas are results of local learning feedback loops (which lead to new goals) → ideas don’t exist 10. Bayesian entrepreneurship → ideas matter 11. This mindset makes ideas inseparable from the learning process (no lightbulb moment) 12. In contrast, a Baysian approach builds on a idea selection 5. _Not _Simply Biases and Heuristics 13. Behavioral economics (Kahneman & tversky) highlights biases and heuristics 14. Bayesian ENT: biases as adaptive responses to a unique challenge faced by entrepreneur (who has heterogeneous and idiosyncratic priors) 6. _Not _the “Lean” Start-up Tripsas & Murray - “The exploratory processes of entrepreneurial firms: The role of purposeful experimentation” Primer on Bayesian Entrepreneurship Heterogeneous beliefs = subjective prior is not the mean of the distribution → “optimism”: as an entrepreneur, if you choose to realize one opportunity, you are rather on the right tail (even the max) * Super confident (optimistic on value + confident on ability to assess value) Demand for experiments is shaped by the prior you have Persuasion Experiments run will shape how beliefs are updated * The self * Others: ### Talk 1 Todd Zenger - “Comments on Bayesian Entrepreneurship” It’s about strategy, entrepreneurship, innovation Q: Is Bayesian the right umbrella term to describe all of what entrepreneurs do? A challenge: the tenets of this “Bayesian” umbrella were initially born as non-Bayesian 2 caricatures of world: 1. Evidence shapes beliefs (ent tries to fit the data) 2. Beliefs shape evidence (causal logic used to imagine a state about which there’s no data) → What is Bayesian? What is not? Focus is on what entrepreneurs _should _do → agenda is normative Process of prior - data - posterior typical of bayesian thinking involves dealing with very sparse data → what helps in this case is causal logic - learning is accelerated How to move from Bayesan networks to causal logic? The Theory-Based View [Felin & Zenger, 2009; 2017] * Entrepreneurial theories: e.g., Airbnb: we can unleash this hotel capacity by solving 4 problems (assumptions / subproblems): * Matching algo * Facilitate payments among strangers * Facilitate trust between strangers * Facilitate informationally efficient professional listings * Test weakest premise: if this premise is not true, need to revise theory / pick a new theory * Causal logic is not fully testable without launching a Strategic Learning [Ehrig and Zenger, 2024] * Type 1 confidence: assumptions are true or subproblems are solvable * Type 2 confidence: my sub-problems are necessary and sufficient conditions for my theory to be true Belief Formation - generating knowledge at the edge between the known and the unknown [Ehrig and Schmidt, 2023] * Learning within theories, adding and removing premises as search happens Messy connection between theory and data - entrepreneurs cannot gather large amounts of data, + the experiment is not repeatable (as it entails “launching” the product / service / firm) Learning from surprises [Ortoleva, 2019] Theories allow to see surprises (be the revision process Bayesian or non-Bayesian) Bayesian vs. Scientific - it doesn’t matter what label we use, the idea is to have a process that is normative and that allows entrepreneurs to make better decisions ### Talk 2 Elena Novelli - What is Bayesian Entrepreneurship? Bayesian entrepreneurship & the Scientific Approach An application of bayesian entrepreneurship: scientific approach * Theory * Hypotheses * Evidence * Evaluation This is a Bayesian approach because what entrepreneurs do when acting scientific is: * Developing a Prior (Theory + Hypotheses) * Testing a prior (Hypotheses + Evidence) * Forming a posterior, given the prior and the data collected (Evidence + Evaluation) 16 RCTs <table> <tr> <td>RCT # </td> <td>RQ </td> <td>Treatment </td> <td>Arms </td> <td>DV </td> </tr> <tr> <td>1-4 </td> <td>What are the key implications of Bayesian ENT? </td> <td>Scientific (Baysian) vs. Control (traditional business training) </td> <td>2 </td> <td>Pivoting Termination </td> </tr> <tr> <td>5-8 </td> <td>Bayesian vs. non-bayesian approaches? </td> <td>Scientific Effectuation Control </td> <td>3 </td> <td>Pivoting Termination Revenues Funding </td> </tr> <tr> <td>9-16 </td> <td>To what extent does theory-based information add value? </td> <td>Full scientific vs. Hypothesis testing vs. Pure control </td> <td>3 </td> <td>Pivoting Termination Revenues </td> </tr> </table> Key Findings: * More frequent and and quicker termination * More focused pivots * Higher revenue on ### Talk 3 Ashish Arora On white paper, things that resonated about the approach: * It clears out this underbrush represented by shared / common priors → idea that there is value in heterogeneity * Going forward: principled way to relax this idea of common priors * Type of experiments with high priors? * Persuasion - how does it apply to contextual factors? E.g., funding, type of investor,... * What is the correct amount of funding to raise? Going forward: * Theory is normative - to what extent can we use it positively / descriptively? * Applications: high-tech, innovation-based entrepreneurship? * How to analyze secondary / observational data to “fit” the model / understand how applicable it is, and under which conditions? * To what extent can normative data effectively be used to think about individual entrepreneurs (when you can’t rely on the market to do the pruning for you) * How does this theory explain differences across sectors? * Idea of effectuation: entrepreneurs are not just scientists, they also believe that they are shaping the environment around them → how do we include the effectuation framework? * Does the Baysian framework apply to things such as deep-tech innovation * Connections to effectuation - how do we use that and put ### Q&A Moderated by Alfonso Gambardella Mi-Jeung Yang: why exclude knightian uncertainty? Why excluding effectuation? Inconsistencies on non-bayesian approach & bayesian normative view Brent Goldfarb: 2 comments 1. Idea of a surprise feels like a tautology → literature in abduction is very closely related to this 2. Need to create conversations with effectuation, as there’s many things where causal thinking is not necessarily needed (e.g., finding a lawyer) Luca Berchicci * Helpful to think about what kind of probabilities we are talking about - Bayesian vs. frequentist? Is Knight uncertainty conciliable? Mercedes Delgado * Did the model work better when experimenting about customer vs. technology? * When should we be using a model of persuasion in entrepreneurship ** ** Amisha Miller * Market failures in entrepreneurship - when do markets actually remove firms * what kinds of experiments can entrepreneurs run, given their contexts? Rem Koning * There’s a class of entrepreneurs that we don’t see in our experiments, that just go by themselves on the market - i.e., overconfident entrepreneurs * It feels that for these groups of individuals, Bayesian logic could play in very well * How does overconfidence interact with the choice to run (or not) experiments? Dan Elfenbein * Heterogeneous priors: where do they come from? Do they arise because you are born optimistic? Do they arise because you have more information? * Behavior & temperament is something worth exploring Danilo Messinese * Bayesian entrepreneurship can still be applied in cases of hypothesis-based testing (Ortoleva) Daniel Kim * What about people who are contrarian and are generally dissuaded? → peer effects might dampen the contrarian beliefs * How about those who didn’t become entrepreneurs because they were not optimists? Joshua Gans * Does the optimistic entrepreneur who is contrarian want to convince others in the early stages anyways Natalia Wright * Whether & how to include effectuation in framework → in reality, effectuation happens alongside more focused experimentation - is it possible to model / illustrate this interplay? * Effectuation happens in parallel to experimentation, and it might change the prior that you are updating Alfonso Gambardella: * Non-Bayesian = cannot update probabilities for events where probabilities = 0 → move to “priors of priors” and update * Effectuation = I don’t care about probabilities → will just engage in actions so that certain events realize → with actions change the world * Truth: we are both Bayesian and Effectuators Todd Zenger * “Cognitive effectuation” - it’s about selecting new state spaces - Susan Cohen * Need to discuss boundary conditions - there’s probably more gray areas, need to be explicit about this ** ** Thomas Astebro * Both effectuation and scientific can coexist in a continuum George Chondrakis * Shall we see it as a lifecycle - maybe you start as a Bayesian, then become an effectuator? * You start more flexible and willing to update in a Bayesian way, but after a while you become more stubborn and entrenched and act in an effectual way? Shannon Liu * Think about cases where effectuation doesn’t work, and how bayesian thinking can help Rem Koning * Maybe we could each write down what we teach in our entrepreneurship courses and see what is the variance and shared points? Alfonso Gambardella * We should do much more about real observational data (Ashish’s point) * Better explore link with effectuation Scott Stern * Fundamentally in entrepreneurship there is a value in being really clear that for opportunity oriented entrepreneurs, the ability to provide tools and make predictions about those tools is valuable * Most econ and finance literature doesn’t deal with effectuation, nor have they ever heard of it * Entrepreneurs make choices and that can be a central point for linking all of these different disciplines * The reason why they’re optimistic matters - are they correct about * Optimism is an important point: should we tell entrepreneurs to be less optimist or are they accurate and about to make a lot of money? ## Session 2 - Entrepreneurial Priors and Their Role ### Intro: Arnaldo Camuffo George Bell (1999 Excite missed acquisition of Google) Baysian view of situations: different probability distributions for the value of Google - which were grounded in different theories of where value came from (“stickiness vs. speed”) Bell had a relatively strong prior on the value of Google and did next to no updating based on the experiment itself. <table> <tr> <td>Excite (George Bell) </td> <td>Google (Larry Page) </td> </tr> <tr> <td colspan="2" >Same information </td> </tr> <tr> <td>Different priors: strong prior on lower value of google </td> <td>Strong prior on high value </td> </tr> <tr> <td>Experiment was uninformative </td> <td> </td> </tr> </table> Common knowledge / info is not the focus, focus is on heterogeneity of priors - different ways of organizing or understanding information Requires sparse evidence to make this work Sources of priors’ heterogeneity 1. Individual differences 2. Preferences 3. Human capital 4. Context 5. Cognitive processes: imagination, intuition, logical and causal reasoning,... Optimism - Relatively strong prior (i.e., relative to others) Sources of optimism - biases, private information Private information about success How do entrepreneurs form and choose “relatively stronger” priors? * Start by asking a question: will there be a new market? * Continues by defining a state space: Xm=[yes, no], where the state of interest is Xm=[Y] * The entrepreneur doesn’t know neither the outcome, nor the probability distribution (summarized by parameter theta) * Since there’s no data, it’s necessary to think about a potential (subjective) probability distribution → the objective is to concentrate probabilities around a given parameter (theta) * Then, think about antecedents of the end state to be “envisioned”: Xtechnology=[Y;N] - _if_ technology is developed, _then_ the new market will arise * End up with a state space * Causal structure restricts the possible set of probability distributions, strengthening the priors * By defining a theory (P(Xm=Y|Xt=Y)>P(Xm=Y|Xt=N)) allows to restrict the probabilities ENTs could be aware of unknown events / wrong model specifications: 1. Foregoing of attributes because due to selective attention 2. Unforeseen contingencies - evolvable priors → Reverse Bayesianism: change theory altogether 3. “Wrong” priors = faulty causal structures - should then lower the “prior on prior” (shall believe less in the theory) → lower prior on prior (methodic doubt) Why are priors important? They affect the type and amount of experiments that ENTs can run * Optimal experimentation strategy * Optimal experiment choices for “self-persuasion” - “biased” experiments * Bayesian, reverse-bayesian and “theory-switching” → More on this in the experimentation session Key research questions for future: * Where do priors come from? * How do good theories look like? Which and how many attributes? * Is causal reasoning the only way to restrict probability distributions? * Do we need multiple theories? * Optimism: biases vs theories ### Talk 1 Thomas Astebro: Entrepreneurs - clueless, biased, or Bayesian machines? Uncertainty? * Knightian uncertainty: probabilities cannot be formed → possibilities can (“perceived” probabilities) * Attitudes & beliefs then factor in, and may make all the difference * Attitudes = motivational factors * Beliefs = biases in assessing that certain events will happen → bias in this setting = deviation from true mean Optimism = to perceive probabilities of “good” outcomes as higher than they actually are Vs. Overconfidence = to hold higher beliefs on one’s own skills / abilities Forming priors necessitates a tolerance to ambiguity: * Ambiguity aversion (attitude) → people may have more or less utility from ambiguous settings * Ambiguity insensitivity (closer to a bias) → people tend to not recognize the difference between different levels of ambiguity Alternative formal theory - “case-based reasoning” * Idea: DM does not know all antecedents, all cause and effects, she can only recognize certain instances by recalling events occurred in the past * “Similarity function” is key because people reason by analogy * As you cumulate info (data), can go back to expected utility as have so much info that can compute probabilities ### Talk 2 Susan Cohen - Priors in ENT decision-making (the role of advice) Advice impacts priors, but not only Advice = information provided drawn from prior knowledge, networks, experience * Mentors → more helpful on strategy (theories) * Peers → more helpful on implementation Advice can: * Change priors, change priors on priors * Increase the number of theories * Increase experimentation * Broaden hypotheses “Bayesian” advice * Mentors are heterogeneous in their knowledge and expertise vs. ENTs are heterogeneous in their ability to extract relevant advice ### Q&A Moderated by Joshua Gans on Priors in ENT decision-making and the role of advice How do people get to the decisions they are going to make? Different things emerged Luca Berchicci: our brains are Bayesian-like prediction machines: they generate a model of reality and use it to predict Jacqueline Kirtley: entrepreneurs can have different priors but also different goals and outcome measures * AG: here, we are rejecting the idea that there’s one way to do this + we don’t know the counterfactual, there’s not a “real truth” → heterogeneity of outcomes comes from heterogeneity of priors, and we don’t observe other outcomes Chiara Spina: contrarian entrepreneur wants negative feedback from advisors/mentors (but when is it wrongly negative advice and when is it correctly negative advice?) Sarada: sometimes priors are informed by the endowment of resources/wealth → it affects risk aversion and outside options Brent Goldfarb: really important and insightful to think that entrepreneurs reduce the complexity of the world (limit decision space) → whatever we find is very context-specific Matt Marx: founders are not created equal - there’s a lot of privilege (private wealth) behind them, we should take into account for these differences Amisha Miller: idea of updating is true for some entrepreneurs, but there are choices that people make for persuasion (e.g., to persuade investors). Entrepreneurs take advice & update + run experiments in a performative way (“mentor theater”) Scott Stern: at Creative Destructive Lab, the idea is - get advice from so many people, the question is who do you pay attention to? How do you sort through different advice? How does it fit within the world that you are trying to understand? Do mentors/advisors really have your best interest in mind? Thomas Astebro: there is an issue of trust - trust of the expertise of that mentor (i.e., how many cases that mentor has gone through, what is the value of that knowledge) ## Session 3 - Entrepreneurial Experimentation and Learning ### Intro: Alfonso Gambardella Two options for entrepreneurial experimentation: 1. Assess hypotheses within a state space * Mimoto: test hypothesis Young → Market 2. Test theories across state spaces * OSense: * Sustainability → Airbnb of things * Sustainability → corporate fleets Awareness of unawareness - different cases 1. Reverse Bayesianism (à la Karni & Viero) → do not change theory 2. HTM (à la Ortoleva) → change end state and theory ### Talk 1 Josh Krieger - Bayesian experimentation Value of learning comes from: * Learning about viability - test of whether an MVP works * Learning about optimal path of development * Update of views about alternative options Potential way to visualize the problem: <table> <tr> <td> </td> <td>Viability experiments </td> <td>Optimal path experiments </td> </tr> <tr> <td>Strong prior </td> <td>Persuasion </td> <td>Updating development path </td> </tr> <tr> <td>Weak priors </td> <td>Learning to kill / commit </td> <td>Messing around? (Science acc. to Joshua) </td> </tr> </table> ### Talk 2 Hyunjin Kim - Bayesian Experimentation Process of experimentation: Step 1: Idea generation (or “alternative theories”) Step 2: Testing and learning (or learning from experiments) Step 3: Idea Selection The white paper very well explains and details steps 1 and 3, but is less developed with respect to step 2. Experiments are treated as a black box, without describing or defining the process for experimenting. Why is learning from experiments not so straightforward * May not be an informative signal * Noise * Bias * Theories shape the demand for experimentation and also the type of experiments that you choose. * May not lead to learning * Attention problem - may not be paying attention to the data * Streetlight effect - have the wrong data but believe it to be right * Updating problem - do not rightly translate insights into correct updates ### **Q&A (**Moderator: Erin Scott) Rem Koning: key to tie experimentation with financial results / valuation Mi-Jeung Yang: learning from experiments - I want to understand what experiment I am running wrong. Experimentation reduces bias Andrea Contigiani: how do these experiments concretely look in practice? We should think about what types of experiments entrepreneurs should run - landing pages, MVPs? Advice and experiments are different ways of getting signals - it’s interesting to think about how they differ, when one can benefit best from one vs. another Susan Cohen: experiments are one way of learning, but org learning literature talks about a thousand ways to learn - experiments are costly and time consuming, is there a place in this new direction where there are different ways of learning (e.g., vicarious learning)? * Erin Scott: true, in fact sometimes you can learn by experimenting but also competitors can learn what you do (and you might signal to your customers that your product sucks) Arnaldo Camuffo: there is sparse information, it turns out that entrepreneurs may not be as Bayesian as they should be - there are some things that may be easier to do. Seems as if the data are there and we may need to better use it ## Session 4 - Fireside Chat with Andrea Pignataro **Scott Stern & Andrea Pignataro** # **DAY 2** ### Fireside chat with Flagship Ventures ## Session 4 Entrepreneurial Persuasion **Lead: Joshua Gans** ### Talk 1 Joshua Gans - Whom and How to Persuade ### Talk 2 Ramana Nanda **Q&A** Erin Scott (Ajay Agrawal): CDL data will be available to us researchers (Amir Sariri is the person most expert about these data) Mi-Jeung Yang: I would encourage people to think a bit broader - there’s a reason lots of people invest into AI right now (legitimacy) - we need to build legitimacy for this research program Scott Stern: * There has been a long literature that highlighted the challenge of the field - need to shift focus from only ex post (e.g., “these people failed because they were wrong) to what actions were in place to persuade stakeholders * There is a first order challenge - being one step away from having some guidance. Eric Ries gave a language to startups - to Josh Gans’ point: the experiment you use to persuade an investor is the experiment you use to persuade yourself * Reasonable program: when do you attract resources based on your belief vs based on shared beliefs? Sarada: * Do investors base your decisions based on conditional probabilities (i.e., it’s likely that entrepreneur will be effective given that they had run an experiment?) Andrea Pignataro: * To what extent does multi-stage financing require multiple rounds vs. a different contractual structure? * To what extent could persuasion be a robust way to analyze the way contracts have been done in the past years? * Carlos Serrano: there are investment vehicles (continuation funds) so that investors can exploit these types of opportunities * US universities have a mechanism by which they are able to share participation rights from the e Amisha Miller * Control vs. high fidelity in experiments? What happens when an entrepreneur has committed to a given experiment? * Scott Stern: angel investors are homophilic Susan Cohen: * Idea of moral hazard in experiments is very interesting - entrepreneurs in accelerators / incubators do not realize that they are running high bar experiments * Joshua Gans: the nature of the experiment itself is a commitment Hyunjin Kim * What are optimal experiments? If we think about the behavioral economics series, what entrepreneurs and investors know, who is selecting in to optimal experiments and who is not? Are they choosing the optimal experiment because they are naive or because they are sophisticated? Brent Goldfarb: * Entrepreneurs are often teams → we need to think what teams do and how they identify their reservation wages Dan Elfenbein: * Computational simulations - design depends on # periods of learning, # measures to move from one draw to the next [...] - this way of testing constraints the set of things that you can do, you need to tune every parameter (priors, …) > this methodology allows to give constraints Alfonso Gambardella: * Hierarchical organizations are more likely to make high-bar exp (= everyone has to say yes), whereas informal organizations make low-bar (= only one yes is sufficient to pursue decision) * Ramana Nanda: this also applies to decision rules for VCs: “champion” model vs. “complete agreement” Charles Fine: As organizations become larger they gain additional levels of complexity and hierarchy which require more high-bar experiments. Scott Stern: * Running experiments can inform people, but actually learning from experiments and properly conducting them requires a lot of investment into statistical methods, capabilities, tech, etc. Arnaldo Camuffo: * Importance of data, of experimental design (choice of power, choice of parameters) * Csto of experimentation is very important, because they need to be Blackwell-efficient * Very important to focus on sequencing, as it’s a key component of bayesian learning ## Session 5 - Entrepreneurial Pedagogy and Practice ### Intro: Erin Scott Goal: make sure we have a common language so that we can more effectively disagree. This is also important to bring our research in the classroom and teach it to students and entrepreneurs. If entrepreneurs know that experimentation is important, why does it often lead to few actionable insights? → it’s really important to focus on designing better experiments, not simply on whether to experiment or not. There is a trade-off in designing experiments between: * Criticality (= testing a critical hypothesis) * Fidelity * Opportunity cost Is it possible to run experiments that don’t lock you up with commitment? → the key is to sequence learning and escalate strategic commitment. Taxie example: experiments well done that also led to negative revision of prior (and idea termination). “Good” ideas have multiple paths to success * Pill pack: applied the “choose 2, pick 1” paradigma - experimented with 2 different business models * This is what entrepreneurial strategy is about: the _choice_ among alternative paths for an idea and company Experimentation informs the choices that entrepreneurs need to take and disregard some others. ### Talk 1 Chiara Spina Teaching Entrepreneurship Many problems hamper the success of entrepreneurial experimentation: * Wrong theory of value * Missing key hypotheses * Vanity sample * Sampling bias * Selective attention to data * Failure to consider context when analyzing data. Open questions: * How to effectively teach the scientific approach using new technologies? * How can we address the declining use of the scientific approach among entrepreneurs over time? * What happens when we theorize and analyze entrepreneurial teams rather than individual entrepreneurs? ### Talk 2 Andrea Coali Entrepreneurial pedagogy & practice How do we teach people to frame theories in an easy-to-process way? Simulations platform, available at [https://imsl.unibocconi.it/](https://imsl.unibocconi.it/) * Player plays the role of the CEO / founder / owner that has to envision future * Can frame a theory, add prior / belief in the theory * Run an experiment Besides simulations, what if someone came in and asked how to create a good theory? * Need validated & common methodologies ## Closing remarks <table> <tr> <td>Take-aways </td> <td>Open points </td> </tr> <tr> <td>What </td> <td>What is an experiment? Different people may have different understandings of it </td> </tr> <tr> <td>Now Bayesian statistics is in my mind </td> <td>Who validates the experimental design? Connection between validation and the governance structure of a company </td> </tr> <tr> <td>High-bar vs. low-bar categorization is very interesting </td> <td>No clarity on what an experiment is - need to ask someone who is an expert in it (e.g., someone running experiments in medical school) </td> </tr> <tr> <td>Having practitioners in the room is very valuable </td> <td>IMSL platform is key to teach decision-makers, but it would be ideal to include decisions that actual entrepreneurs may have (e.g., go get a patent vs. not) + then have videos with experts that show how a “good” way to think about it would be </td> </tr> <tr> <td>Common language is fundamental </td> <td>What is a management practice that entrepreneurs can get out of this research agenda? </td> </tr> <tr> <td>It’s very difficult for people to conceptualize their ideas into theories → there needs to be some training in graduate courses </td> <td>Individual entrepreneurs are focused on path forward - more on the investor side would be interested </td> </tr> <tr> <td>We may want posterior formulation / learning to be somewhat biased (i.e., in line with initial preferences) </td> <td>Key to understand the heterogeneities in this field - there’s many contextual factors that impact what experiments entrepreneurs run and how they update information </td> </tr> <tr> <td>Kahneman System 1 vs. System 2 thinking - if you are in the “leap of faith” story, need to have complex cognitive processes → the more you invest in cognitive costly effort, the more you are prepared for a leap of faith </td> <td>The audience of experiments is also something we should focus on - entrepreneurs may be building on each others’ experiments </td> </tr> <tr> <td>An experiment = any acquisition of relevant information </td> <td>Going forward, we need to teach people how to reason, how to use basic causal knowledge </td> </tr> <tr> <td>The objective is to help entrepreneurs learn about some future state spaces. There’s 2 broad categories: 1) think and theories, 2) experiment - what’s the right learning production function depends on whether 1) and 2) are complements or substitutes + Depends on whether we are in adjacent or distance state spaces </td> <td>Ask entrepreneurs whether what is being done is useful </td> </tr> <tr> <td>It seems that we want theories that are far away from current space - but we want learning that’s dispassionate. How do we connect these two? </td> <td>Confusion on what an experiment is. Don’t use words that have different meanings for other people, such as optimism (it’s more about information) and persuasion (again, you could call it information) </td> </tr> <tr> <td> </td> <td>Learning can come from the agency of others - others can give signals. Cheapest way to learn is from failures of others. </td> </tr> </table>