Implementation of Stimulus Control in a Computational Model Open Access

Berg, John (2011)

Permanent URL: https://etd.library.emory.edu/concern/etds/8910jv394?locale=en
Published

Abstract

Reacting appropriately under different stimulus conditions is crucial for live organisms to improve their chances of survival. A computational model of behavior based on selection by consequences originally proposed been successful in producing a variety of behaviors consistent with live organisms (McDowell, 2004; McDowell & Caron, 2006; Kulubekova & McDowell, 2007; McDowell, Caron, Kulubekova, & Berg, 2008). However, previous versions of this model have been limited in that there is no method to change behavior in reaction to different stimulus conditions. The purpose of the current project was to modify the McDowell model to enable it to react differently under different stimulus conditions. Changes were made to the model that enabled variation in behavior across stimulus conditions using a conditioned reinforcement approach. Implementing the Rescorla-Wagner model to determine conditioned stimulus strength and therefore the degree of conditioned reinforcement, two-link, single alternative chained schedules of behavior were arranged in a series of experiments. Correspondence with live organism behavior was determined using qualitative and quantitative methods. Specifically, equilibrium values of the conditioned stimulus strength were evaluated for correspondence with the original and exponentiated versions of the Mazur hyperbolic discounting function, modified for use with variable interval schedules. Behavior on the chained schedules was evaluated qualitatively for consistency with previously published live organism data and was evaluated quantitatively in a replication of an experiment Williams, and Fantino (1987), which implemented a pre-reinforcement delay. Results indicated correspondence between the Rescorla-Wagner model and the exponentiated Mazur function. Chained schedules of behavior were largely consistent with live organism data. However, the Royalty et al. experiment was not successfully replicated. The results indicated that the Rescorla-Wagner model and the Mazur function provide a complete model of conditioned reinforcement. Using this model, chained schedules of behavior were successfully produced using the McDowell computational model. However, some behavioral phenomenon with pre- reinforcement delays could not be produced using the currently proposed computational methodology.

Table of Contents

Table of Contents

Introduction...1

How the McDowell Model Works...2

A comprehensive account of adaptive behavior: from neurons to behavior...4

Chained Schedules of Reinforcement...6
Chained Behavior in Live Organisms and Associated Theories...7
The Modified McDowell Computational Model...9
Conditioned Reinforcement...12

Dynamic theories of conditioned reinforcement...12
Static theories of conditioned reinforcement...14
Neurobiological bases of conditioned reinforcement...16

Correspondence of R-W and Mazur Models...17

A modified R-W function...18

Purpose of the Current Project...18

General Methods...20

Subject and Environment...20
Apparatus and Materials...21
Computational Procedures for the Modified McDowell Model...22
Project Structure...24

Experiment Series I...24
Experiment Series II...25
Experiment Series III...25

Experiment Series I: Rescorla-Wagner Model and Mazur Function Correspondence...25

Methods...25
Results...26

Effects of traditional Rescorla-Wagner parameters on conditioned stimulus strength...26
Effect of the currently proposed Rescorla-Wagner exponent, a, on conditioned reinforcer strength...29
Correspondence of Rescorla-Wagner and Mazur functions...30

Discussion...34

Effects of Rescorla-Wagner parameters on conditioned stimulus strength, V...34
The association between the Rescorla-Wagner and Mazur models...35

Experiment Series II: Chained Schedules in the Computational Environment...41

Methods...41

Varying the proportion of behaviors produced with a fitness function...41
Initial link correspondence with matching theory...42
Simple behavior chains...43

Results...44

Effect of varying the proportion of behaviors produced using a fitness function after reinforcement...44
Initial link correspondence with matching theory...45
Simple behavior chains...46

Cumulative records...46
Initial and terminal response-reinforcement rate dependency...48
Initial versus terminal link response rates...48

Discussion...49

Effect of varying the proportion of behaviors produced using a fitness function after reinforcement...49
Matching analyses...50
Simple behavior chains...50

Experiment Series III: Replication of the Royalty et al. Experiments...51

Methods...52
Results...52
Discussion...53

General Discussion...55

Problems with Delay of Reinforcement: Failure to Replicate Royalty et al. (1987)...55
Correspondence of the Rescorla-Wagner and Mazur Models...58
Neurobiological Correlates with the Computational Model...60
Future Directions for Research...62
Conclusion...63

References...65


Tables
Table 1. Parameter descriptions...71
Table 2. Model parameters varied in Experiment Series I...72
Table 3. Best-fit parameter values and goodness-of-fit statistics for fits of classic and exponentiated Mazur functions to conditioned stimulus strength values, V, for different values of β1 under several Rescorla-Wagner parameter conditions...73
Table 4. Best-fit parameter values and goodness-of-fit statistics for fits of classic and exponentiated Mazur functions to conditioned stimulus strength values, V, for different β0 Rescorla-Wagner parameter values...74
Table 5. Best-fit parameter values and goodness-of-fit statistics for fits of classic and exponentiated Mazur functions to conditioned stimulus strength values, V, for different values of α under several Rescorla-Wagner parameter conditions...75
Table 6. Best-fit parameter values and goodness-of-fit statistics for fits of classic and exponentiated Mazur functions to conditioned stimulus strength values, V, for different a...76
Table 7. Rescorla-Wagner parameters, terminal link RI schedule values, parameters of the best-fitting hyperbola, and fit statistics (R2) for fits of the classic and modern matching functions to initial link response-reinforcement data...77
Table 8. Best-fit parameter values from fits of classic and modern matching functions to model response-reinforcement rate data for different values of proportion of behaviors produced using a fitness function following reinforcement...78


Figures
Figure 1. Schematic illustrating development of primary and conditioned reinforcement in modified McDowell computational model implemented in a chained schedule...83
Figure 2. Plot of the exponentiated Mazur function for values of s and r with parameters a and b set to 1.1 and 5, respectively...84
Figure 3. The effect of reinforcement rate (r) on conditioned stimulus strength (V) for 3 different values of α under the model conditions β0: 0.01, β1: 0.5, a: 1.0...85
Figure 4. Instantaneous values of conditioned stimulus strength, V, for the first 500 time ticks. Top panel shows results for α = 0.5, middle panel for α = 0.7, and bottom panel for α = 0.9...86
Figure 5. Mean and standard deviation of the conditioned stimulus strength, V, for three experiments using α = 0.5, α = 0.7, and α = 0.9...87
Figure 6. The effect of reinforcement rate on conditioned stimulus strength (V) for 5 different values of β0 under the model conditions α: 0.7, β1: 0.25, a: 1.0...88
Figure 7. The effect of reinforcement rate on conditioned stimulus strength (V) for 5 different values of β1 under the model conditions α: 0.7, β0: 0.05, a: 1.0...89
Figure 8. The effect of reinforcement rate on conditioned stimulus strength (V) for 5 different values of β1 under the model conditions α: 0.7, β0: 0.01, a: 1.0...90
Figure 9. The effect of reinforcement rate on conditioned stimulus strength (V) for 5 different values of a under the model conditions β0: 0.05, β1: 1.0, α: 0.7...91
Figure 10. Instantaneous values of conditioned stimulus strength, V, for the first 500 time ticks. Top panel shows results for a = 1, middle panel for a = 1.5, and bottom panel for a = 2 (β0: 0.05, β1: 1.0, α: 0.7)...92
Figure 11. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the classic Mazur function for random interval schedules. Model conditions are detailed below each plot...93
Figure 12. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the exponentiated Mazur function for random interval schedules. Model conditions are detailed below each plot...94
Figure 13. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the Mazur function for random interval schedules. Model conditions are detailed below each plot...95
Figure 14. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the exponentiated Mazur function for random interval schedules. Model conditions are detailed below each plot...96
Figure 15. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the Mazur function for random interval schedules. Model conditions are detailed below each plot...97
Figure 16. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the exponentiated Mazur function for random interval schedules. Model conditions are detailed below each plot...98
Figure 17. Standardized residuals (vertical axis) versus predicted values of V (horizontal axis) from fits of the Mazur and exponentiated Mazur functions for random interval schedules under 4 different values of the exponentiated Rescorla-Wagner exponent, a. For all plots the Rescorla-Wagner parameters used were: β0: 0.05, β1: 1, α: 0.7...99
Figure 18. Response and reinforcement rate plots for different proportions of behaviors produced for the next time-step using the parental fitness function after reinforcement (as opposed to production of all new behaviors using the fitness function to select parent behaviors). The percentage of behavior replaced using the linear fitness function is denoted by the legend on the right...100
Figure 19. Standardized residuals (vertical axis) versus predicted values of B (horizontal axis) from fits of the classic (top two rows) and modern (bottom two rows) matching functions to reinforcement-response rate data...101
Figure 20. Cumulative distribution plots showing responses and reinforcements for the first 1000 time-steps of both the initial (top) and terminal (bottom) links for a chain RI1-RI1 schedule...102
Figure 21. Cumulative distribution plot showing responses and reinforcements for the first 1000 time-steps of both the initial (top) and terminal (bottom) links for a chain RI5-RI5 schedule...103
Figure 22. Cumulative distribution plots showing responses and reinforcements for the first 1000 time-steps of both the initial (top) and terminal (bottom) links for a chain RI25-RI25 schedule...104
Figure 23. Cumulative distribution plots showing responses and reinforcements for the first 1000 time-steps of both the initial (top) and terminal (bottom) links for a chain RI112-RI112 schedule...105
Figure 24. The initial link reinforcement and response rates for 4 constant terminal link reinforcement rates under the Rescorla-Wagner conditions β0: 0.05, β1: 0.5, α: 0.7, a: 1.0...106
Figure 25. Initial to terminal link response rate ratios. Model conditions: β0: 0.05, β1: 0.5, α: 0.7, a: 1.0...107
Figure 26. Initial link response rates (B) for 22 experiments varying the initial and terminal link schedules from RI1-RI1 to RI22-RI22 while maintaining a 1 time-tick delay to reinforcement in the initial link. Operant level of response rate for the computational model is denoted for comparison...108
Figure A1. Random distribution of 6 datapoints fit with quadratic (top panel), cubic (middle panel), and a 6th order polynomial (bottom panel) illustrating the increasing percent variance accounted for (pVAF) as polynomial order approaches the number of datapoints...109

About this Dissertation

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Degree
Submission
Language
  • English
Research Field
Keyword
Committee Chair / Thesis Advisor
Committee Members
Last modified

Primary PDF

Supplemental Files