Implementation of Stimulus Control in a Computational Model Open Access
Berg, John (2011)
Abstract
Reacting appropriately under different stimulus conditions is crucial for live organisms to improve their chances of survival. A computational model of behavior based on selection by consequences originally proposed been successful in producing a variety of behaviors consistent with live organisms (McDowell, 2004; McDowell & Caron, 2006; Kulubekova & McDowell, 2007; McDowell, Caron, Kulubekova, & Berg, 2008). However, previous versions of this model have been limited in that there is no method to change behavior in reaction to different stimulus conditions. The purpose of the current project was to modify the McDowell model to enable it to react differently under different stimulus conditions. Changes were made to the model that enabled variation in behavior across stimulus conditions using a conditioned reinforcement approach. Implementing the Rescorla-Wagner model to determine conditioned stimulus strength and therefore the degree of conditioned reinforcement, two-link, single alternative chained schedules of behavior were arranged in a series of experiments. Correspondence with live organism behavior was determined using qualitative and quantitative methods. Specifically, equilibrium values of the conditioned stimulus strength were evaluated for correspondence with the original and exponentiated versions of the Mazur hyperbolic discounting function, modified for use with variable interval schedules. Behavior on the chained schedules was evaluated qualitatively for consistency with previously published live organism data and was evaluated quantitatively in a replication of an experiment Williams, and Fantino (1987), which implemented a pre-reinforcement delay. Results indicated correspondence between the Rescorla-Wagner model and the exponentiated Mazur function. Chained schedules of behavior were largely consistent with live organism data. However, the Royalty et al. experiment was not successfully replicated. The results indicated that the Rescorla-Wagner model and the Mazur function provide a complete model of conditioned reinforcement. Using this model, chained schedules of behavior were successfully produced using the McDowell computational model. However, some behavioral phenomenon with pre- reinforcement delays could not be produced using the currently proposed computational methodology.
Table of Contents
Table of Contents
Introduction...1
How the McDowell Model Works...2
A comprehensive account of adaptive behavior: from neurons to behavior...4
Chained Schedules of Reinforcement...6
Chained Behavior in Live Organisms and Associated
Theories...7
The Modified McDowell Computational Model...9
Conditioned Reinforcement...12
Dynamic theories of conditioned
reinforcement...12
Static theories of conditioned reinforcement...14
Neurobiological bases of conditioned reinforcement...16
Correspondence of R-W and Mazur Models...17
A modified R-W function...18
Purpose of the Current Project...18
General Methods...20
Subject and Environment...20
Apparatus and Materials...21
Computational Procedures for the Modified McDowell Model...22
Project Structure...24
Experiment Series I...24
Experiment Series II...25
Experiment Series III...25
Experiment Series I: Rescorla-Wagner Model and Mazur Function Correspondence...25
Methods...25
Results...26
Effects of traditional Rescorla-Wagner
parameters on conditioned stimulus strength...26
Effect of the currently proposed Rescorla-Wagner exponent,
a, on conditioned reinforcer strength...29
Correspondence of Rescorla-Wagner and Mazur functions...30
Discussion...34
Effects of Rescorla-Wagner parameters on
conditioned stimulus strength, V...34
The association between the Rescorla-Wagner and Mazur
models...35
Experiment Series II: Chained Schedules in the Computational Environment...41
Methods...41
Varying the proportion of behaviors produced
with a fitness function...41
Initial link correspondence with matching theory...42
Simple behavior chains...43
Results...44
Effect of varying the proportion of behaviors
produced using a fitness function after reinforcement...44
Initial link correspondence with matching theory...45
Simple behavior chains...46
Cumulative records...46
Initial and terminal response-reinforcement rate
dependency...48
Initial versus terminal link response rates...48
Discussion...49
Effect of varying the proportion of behaviors
produced using a fitness function after reinforcement...49
Matching analyses...50
Simple behavior chains...50
Experiment Series III: Replication of the Royalty et al. Experiments...51
Methods...52
Results...52
Discussion...53
General Discussion...55
Problems with Delay of Reinforcement: Failure to
Replicate Royalty et al. (1987)...55
Correspondence of the Rescorla-Wagner and Mazur Models...58
Neurobiological Correlates with the Computational Model...60
Future Directions for Research...62
Conclusion...63
References...65
Tables
Table 1. Parameter descriptions...71
Table 2. Model parameters varied in Experiment Series
I...72
Table 3. Best-fit parameter values and goodness-of-fit
statistics for fits of classic and exponentiated Mazur functions to
conditioned stimulus strength values, V, for different values of
β1 under several Rescorla-Wagner parameter
conditions...73
Table 4. Best-fit parameter values and goodness-of-fit
statistics for fits of classic and exponentiated Mazur functions to
conditioned stimulus strength values, V, for different
β0 Rescorla-Wagner parameter values...74
Table 5. Best-fit parameter values and goodness-of-fit
statistics for fits of classic and exponentiated Mazur functions to
conditioned stimulus strength values, V, for different values of
α under several Rescorla-Wagner parameter
conditions...75
Table 6. Best-fit parameter values and goodness-of-fit
statistics for fits of classic and exponentiated Mazur functions to
conditioned stimulus strength values, V, for different
a...76
Table 7. Rescorla-Wagner parameters, terminal link RI schedule
values, parameters of the best-fitting hyperbola, and fit
statistics (R2) for fits of the classic and modern
matching functions to initial link response-reinforcement
data...77
Table 8. Best-fit parameter values from fits of classic and
modern matching functions to model response-reinforcement rate data
for different values of proportion of behaviors produced using a
fitness function following reinforcement...78
Figures
Figure 1. Schematic illustrating development of primary and
conditioned reinforcement in modified McDowell computational model
implemented in a chained schedule...83
Figure 2. Plot of the exponentiated Mazur function for
values of s and r with parameters a and
b set to 1.1 and 5, respectively...84
Figure 3. The effect of reinforcement rate (r) on
conditioned stimulus strength (V) for 3 different values of
α under the model conditions β0: 0.01,
β1: 0.5, a: 1.0...85
Figure 4. Instantaneous values of conditioned stimulus
strength, V, for the first 500 time ticks. Top panel shows
results for α = 0.5, middle panel for α = 0.7, and bottom
panel for α = 0.9...86
Figure 5. Mean and standard deviation of the conditioned
stimulus strength, V, for three experiments using α =
0.5, α = 0.7, and α = 0.9...87
Figure 6. The effect of reinforcement rate on conditioned
stimulus strength (V) for 5 different values of
β0 under the model conditions α: 0.7,
β1: 0.25, a: 1.0...88
Figure 7. The effect of reinforcement rate on conditioned
stimulus strength (V) for 5 different values of
β1 under the model conditions α: 0.7,
β0: 0.05, a: 1.0...89
Figure 8. The effect of reinforcement rate on conditioned
stimulus strength (V) for 5 different values of
β1 under the model conditions α: 0.7,
β0: 0.01, a: 1.0...90
Figure 9. The effect of reinforcement rate on conditioned
stimulus strength (V) for 5 different values of a
under the model conditions β0: 0.05, β1: 1.0, α:
0.7...91
Figure 10. Instantaneous values of conditioned stimulus
strength, V, for the first 500 time ticks. Top panel shows
results for a = 1, middle panel for a = 1.5, and
bottom panel for a = 2 (β0: 0.05,
β1: 1.0, α: 0.7)...92
Figure 11. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
classic Mazur function for random interval schedules. Model
conditions are detailed below each plot...93
Figure 12. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
exponentiated Mazur function for random interval schedules. Model
conditions are detailed below each plot...94
Figure 13. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
Mazur function for random interval schedules. Model conditions are
detailed below each plot...95
Figure 14. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
exponentiated Mazur function for random interval schedules. Model
conditions are detailed below each plot...96
Figure 15. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
Mazur function for random interval schedules. Model conditions are
detailed below each plot...97
Figure 16. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
exponentiated Mazur function for random interval schedules. Model
conditions are detailed below each plot...98
Figure 17. Standardized residuals (vertical axis) versus
predicted values of V (horizontal axis) from fits of the
Mazur and exponentiated Mazur functions for random interval
schedules under 4 different values of the exponentiated
Rescorla-Wagner exponent, a. For all plots the
Rescorla-Wagner parameters used were: β0: 0.05,
β1: 1, α: 0.7...99
Figure 18. Response and reinforcement rate plots for
different proportions of behaviors produced for the next time-step
using the parental fitness function after reinforcement (as opposed
to production of all new behaviors using the fitness function to
select parent behaviors). The percentage of behavior replaced using
the linear fitness function is denoted by the legend on the
right...100
Figure 19. Standardized residuals (vertical axis) versus
predicted values of B (horizontal axis) from fits of the
classic (top two rows) and modern (bottom two rows) matching
functions to reinforcement-response rate data...101
Figure 20. Cumulative distribution plots showing responses
and reinforcements for the first 1000 time-steps of both the
initial (top) and terminal (bottom) links for a chain RI1-RI1
schedule...102
Figure 21. Cumulative distribution plot showing responses
and reinforcements for the first 1000 time-steps of both the
initial (top) and terminal (bottom) links for a chain RI5-RI5
schedule...103
Figure 22. Cumulative distribution plots showing responses
and reinforcements for the first 1000 time-steps of both the
initial (top) and terminal (bottom) links for a chain RI25-RI25
schedule...104
Figure 23. Cumulative distribution plots showing responses
and reinforcements for the first 1000 time-steps of both the
initial (top) and terminal (bottom) links for a chain RI112-RI112
schedule...105
Figure 24. The initial link reinforcement and response rates
for 4 constant terminal link reinforcement rates under the
Rescorla-Wagner conditions β0: 0.05,
β1: 0.5, α: 0.7, a: 1.0...106
Figure 25. Initial to terminal link response rate ratios.
Model conditions: β0: 0.05, β1:
0.5, α: 0.7, a: 1.0...107
Figure 26. Initial link response rates (B) for 22
experiments varying the initial and terminal link schedules from
RI1-RI1 to RI22-RI22 while maintaining a 1 time-tick delay to
reinforcement in the initial link. Operant level of response rate
for the computational model is denoted for comparison...108
Figure A1. Random distribution of 6 datapoints fit with
quadratic (top panel), cubic (middle panel), and a 6th order
polynomial (bottom panel) illustrating the increasing percent
variance accounted for (pVAF) as polynomial order approaches the
number of datapoints...109
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Keyword | |
| Committee Chair / Thesis Advisor | |
| Committee Members |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Implementation of Stimulus Control in a Computational Model () | 2018-08-28 15:53:48 -0400 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|