Generative Symbol Systems in Colored Clay

Generative Symbol Systems in Colored Clay

Craig Graci and Randy Odendahl
Computer Science Department
SUNY at Oswego
Oswego, New York 13126

Abstract

A very simple computational formalism with which people appear to interact very naturally will be described. This formalism exploits an action-oriented knowledge representation to facilitate creative cognitive acts in a wide range of domains, including visual, musical, mathathematical and linguistic domains. A "simplicity metric" which crudely but effectively measures a notion of "simplicity of expression" will feature in discussions surrounding the application of this formalism to problems in Cognitive Science. The role of this formalism in a learning model which embraces the idea of cognitve scaffolding will be suggested.

Introduction
In this paper we describe a symbol system inspired computational formalism grounded in action-oriented semantics that we have been using at SUNY Oswego in a variety of cognitively flavored courses to introduce and explore a number of ideas pertaining to cognition and the study of Cognitive Science. We discuss the organization and application of the formalism in terms of "Generative Symbol Systems" housed in a family of very simple languages, collectively referred to as "Colored Clay." We identify the nature of our use of these systems for illuminating a number of issues pertaining to the structured representation of sequential processes.
Background
In his cognitive approach to creativity, Howard Gardner eloquently summarizes certain ideas pertaining to symbol systems put forth by the German philosopher Ernst Cassirer. "These symbols systems -- these codes of meaning -- are the vehicles through which thought takes place: by their very nature they are creative, open systems. Through the use of symbols the human mind, operating according to structuralist principles, can create, revise, transform, and re-create wholly fresh products, systems, and even worlds of meaning" [Gardner, 1982]. We use the term "generative symbol system" to refer to a particular computational formalism which appears to capture some of the essential cognitive power of Cassirer's cognitive conception.
Andy Clark suggests that a profound shift is taking place in how we think about mind and cognition -- a representational shift from mind as mirror to mind as controller. "The idea here is that the brain should not be seen as primarily a locus of inner descriptions of external states of affairs; rather, it should be seen as a locus of inner structures that act as operators upon the world via their role in determining actions" [Clark, 1997]. Essential to this point of view is the idea of action-oriented representations. Clark describes these as representations that simultaneously describe aspects of the world and prescribe courses of action, representations which are poised between pure control structures and passive representations of external reality. The knowledge representation featured in "generative symbol systems" can be viewed precisely as an action-oriented representation.

Generative Symbol Systems

A generative symbol system is a computational system composed of the following three entities: (1) A small number of featured objects, each of which is a "global object with local state." (2) A vocabulary of symbols, each of which is bound to a command which operates on one or more of the global objects. (3) An extensible language defined over the vocabulary symbols which implements a particular set of generally recognizable cognitive concepts (e.g., chunking, repetition).

The notion of generative symbol system as defined can computationally be realized in any number of ways. One realization of the notion that we have developed at SUNY Oswego is rendered in the "Colored Clay" medium. Colored Clay is family of programming languages which is hierarchically ordered by the subset relation. Each successor language in this family incorporates all of the functionality in its parent language, and adds additional functionality which is inspired by one or more commonly acknowledged aspects of cognition. These languages are implemented in Java and currently run in the form of Java applets on the World Wide Web. The colors of Clay include:

Green Clay which may be characterized as a language of parameterless commands. That is all. It corresponds to an agent with a vocabulary but no mechanisms for relating its vocabulary elements.
Red Clay which adds to Green Clay a capacity for sequencing commands.
Blue Clay which adds to Red Clay a chunking mechanism by means of procedural abstraction.
Indigo Clay which adds to Blue Clay mechanisms of repetition and disembodied chunking which can be composed in various ways.
Grey Clay which adds to Indigo Clay an element of nondeterminism.
Fuchsia Clay which adds to Indigo Clay a concept of process fusion which enables one to explicitly conceive of a process as separate subprocesses and then unifies the subprocesses into the desired process for execution.
Saphire Clay which adds to Indigo Clay a concept of structural refinement which facilitates structural composition via stepwise refinement.
Each of these colors of clay provides avenues of opportunity through which to study various aspects of cognition.
It appears natural to refer to the coupling of a featured object set and a related vocabulary of symbols as a world. Among the worlds we have created are graphics worlds (shape drawing/painting/inversion worlds, tiling worlds), mathematical worlds (chance, combinatorial, number theory worlds), music worlds (melody, harmony, rhythm in abstract and instrumental forms), and linguistic worlds (strict form poetry, word play worlds).
By way of example, consider a very primitive melody world, one which has served as a workhorse for introducing generative symbol systems to hundreds of students at SUNY Oswego.

Featured Objects

A note with a number of properties including degree, duration and register.
A scale which is defined in terms if a key and a mode.

Vocabulary of Symbols

play - Sonically and/or textually render the underlying note.
rest - Silently "play" the underlying note.
rp - Raise the pitch of the underlying note 1 degree within the underlying scale "modulo" the length of the scale.
lp - Lower the pitch of the underlying note 1 degree within the underlying scale "modulo" the length of the scale.
xd(2) - Double the duration of the underlying note.
xd(3) - Triple the duration of the underlying note.
sd(2) - Halve the duration of the underlying note.
sd(3) - Third the duration of the underlying note.

The Blue Clay language is extremely simple. The notion of "symbol" is central to the language, where a Blue Clay symbol is a denotation for an action which is applicable to a global object. A Blue Clay program is either a sequence of symbols or a definition of a new symbol in terms of a sequence of symbols.
blue>melody> play, rp, play, lp.
      1  ------------
      2  ------------
blue>melody> triplet = sd(3), play, play, play, xd(3).
blue>melody> triplet.
      1  ----
      1  ----
      1  ----
blue>melody> bolerobit = play, triplet, play, triplet, play, play.
blue>melody> bolerobit.
      1  ------------
      1  ----
      1  ----
      1  ----
      1  ------------
      1  ----
      1  ----
      1  ----
      1  ------------
      1  ------------
The Indigo Clay language is almost as simple as the Blue Clay language. As mentioned previously, it is Blue Clay augmented by disembodied chunks and repetition.
indigo>melody> triplet = sd(3), play*3, xd(3).
indigo>melody> bolerobit = play+triplet*2, play*2.
As a more representative example, consider DemitriKabalevski's "Little Tune".
Here is relatively unstructured program Indigo Clay Melody World program to play Kabalevsky's "Little Tune."
tune(unstructured) = rp*2, play+lp*2, rp*2, play, 
   lp*2, play, rp, play, lp, play, rp, xd(2), play, 
   sd(2), lp, rp*2, play+lp*2, rp*2, play, lp*2, 
   play, rp, xd(2), play*2, sd(2), lp, rp*2, play+lp*2, 
   rp*2, play, lp*2, play, rp, play, lp, play, rp, 
   xd(2), play, sd(2), lp, rp*2, play+lp*2, rp*2, 
   play+lp*2, xd(2), play*2, sd(2).
In contrast, here is a fairly well structured Indigo Clay Melody World program to play Kabalevsky's "Little Tune."
tune = tune(1), tune(2).
tune(1) = tune(1a), tune(1b).
tune(2) = tune(2a), tune(2b).
tune(1a) = tune(m1), tune(m2).
tune(1b) = tune(m3), tune(m4).
tune(2a) = tune(1a).
tune(2b) = tune(m7), tune(m8).
tune(m1) = play(ss)(s), play(ss)().
tune(m2) = play(s)(), play(s)(ht).
tune(m3) = tune(m1).
tune(m4) = play(s)(s)(ht).
tune(m7) = rp, play(s)()*2, lp.
tune(m8) = play()()(ht).                     "schema"
play(ss)(s) = rp*2, play+lp*2.               "vocabulary"
play(ss)() = rp*2, play, lp*2, play.
play(s)() = rp, play, lp, play.
play(s)(ht) = rp, xd(2), play, sd(2), lp.
play(s)(s)(ht) = rp, play()()(ht), lp.
play()()(ht) = xd(2), play*2, sd(2).
Simplicity and the Colored Clay Sigma Metrics
Generative Symbol Systems can be viewed as mechanisms for structurally describing a sequence of tasks whose net executional effect is to explicitly render visual, sonic, linguistic, mathematical, or other artifacts. Two basic representational challenges associated with Generative Symbol Systems are:

Given a particular Generative Symbol System and a particular artifact realizable within the system, write a program which produces the artifact.
Given a particular set of phenomena, design a Generative Symbol Systems capable of generating the phenomena.
These challenges become cognitively meaningful only in the light of measures of simplicity. The relavence of simplicity to cognition is evidenced in any number of compelling quotes. For example, Charlse Mingus defined creativity in terms of simplicity: "Making the simple complicated is commonplace; making the complicated simple, awesomely simple, that's creativity." George Sand identifies genius in light of simplicity: "Simplicity is the most difficult thing to secure in this world; it is the last limit of experience and the last effort of genius."
The Colored Clay "sigma" metrics are an attempt to quantitatively measure "conceptual simplicity," "readability," "modifiability," etc. of a program. As a rule, if two programs are functionally equivalent the one with the lower value of sigma will be prefered on the grounds of "conceptual simplicity." Each color of Clay has its own sigma metric, which are (by design) easy to compute (by machine). They have two main areas of application.

Given N expressions of an artifact, determine which is simpler.
Given N artifacts, order them by simplicity.

The session below suggests the role for the sigma metric "in action" as the relative simplicities of the (extreme) conceptions of "Little Tune" is measured. Of course, the idea of iteratively decreasing the value of sigma as one conceives and reconceives an expression of a computation is not captured with this illustration. The challange on the small scale is to minimize sigma for each task. The challenge on the large scale is to craft "utilities" which are conceptually sound in that their use tends to minimize sigma. These utilities form a "vocabulary," a "private language" of sorts which can ultimatedly be explicitly shared with others and which explicitly awaits inspection by oneself as well as others.
indigo>melody> sigma.
command name? tune(unstructured).
2222
indigo>melody> sigma.
command name? tune(unstructured).
205
Some questions naturally arise in cosideration of the sigma metric. Is the idea of endeavoring to computationally measure the conceptual simplicty of an idea valuable? What is the relationship between simplicty of computational expression and effectiveness of human conceptualization? How can "sigma" metrics be used in educational research and practice? How good is sigma as currently defined How can we measure how good sigma is?
Personal Vocabulary Development and Scaffolded Intervention
Suppose, in the course of solving several problems, you identify a collection of programs which are generally useful. Call this collection V1. Once these become second nature to you, you craft and identify as being very useful another, higher level collections of programs as being generally useful. Call this collection V2. Continue.
V1: { short, long, shorter, longer, dot, undot }
V2: { doublet, triplet, double, triple }
V3: { asc(2), dsc(2), asc(2)(dt), dsc(2)(dt) }
...
Different people will undoubtedly develop different sequence of vocabularies if left to themselves. A few "geniouses" will develop incredibly powerful vocabularies, ones with which they can produce beautiful, richely textured artifacts relatively simple terms. Often people will get myered in a web of detail and make little progress on their own. At these times some "external scaffolding" may prove very useful. Provide a vocabulary with which a person can meet their goals. After a while, they will have made this vocabulary their own and will move on to more personal vocabulary building. These are vocabularies of ideas.
According to the Russian psychologist L. S. Vygotsky, interaction with ones environment within the confines of a "zone of proximal development" facilitates cognitive development by maximizing opportunities to internalize culturally transmited cognitive artifacts and constructs. Operating within a "zone" we can achieve goals and obtain insights which are otherwise beyond our reach.
Andy Clark [Clark, 1997] champions the notion of "cognitive scaffolding" as an important component of cognitive theory. He adroitly defines this notion as a generalization of Vygotsky's human-oriented zone of proximal developoment which "encompasses all kinds of external aid and support whether provided by adults or by the inanimate environment." The basic idea is that external scaffolding enhances the likelihood of success in whatever cognitive arena is being supported. In our courses at SUNY Oswego we have found that Colored Clay realizations of Generative Symbol Systems are a valuable form of "external scaffolding"! Along these same lines, Clark suggests that one important role that linguistic artifacts play in cognition is captured by the image of "trading spaces." In Clark's words, "The agent who exploits external symbol structures is trading culturally achieved representation against what would otherwise be (at best) time-intensive and labor-intensive internal computation." Colored Clay scaffolding permits students to use others' vocabularies as support and guidance while they strive to build their own personal vocabularies.
Why Build Artificial Scaffolding? The careful crafting of artificial scaffolding (e.g., Colored Clay lanbuage and related orlds) to augment natural scaffolding (people) in support of a person's "vocabulary" development is rewarding on two counts. First, it is enjoyable to have a hand, albeit an indirect hand, in helping people to build useful structures of ideas and skills. Secondly, artificial scaffolding provides a potentially valuable tool for studying various aspects of cognition.
Syntactic Structures, Nondeterminism and Grey Clay

The highest levels of a musical structure are sometimes referred to as schema, widely applicable frameworks within which details can be varied. This idea of schema, coupled with the notion of a vocabulary suggests a crude notion of musical composition which is Jabberwockian in nature. It is possible that refinements of this form of crude composition which embrace nondeterminism and systems of constraints may illuminate processes of creativity.
For certain theoretical purposes Indigo Clay can be interpreted as a cosmetically convenient Blue Clay. A Blue/Indigo Clay program, subject to a simple "procedural to declarative" mapping at its lowest levels, can be viewed as a phrase structure, a context free grammar, which generates exactly one sentence. By introducing a simple mechanism of nondeterminism into these languages one can craft programs which correspond to phrase structure grammars which can generate a number of sentences. Grey Clay is Indigo Clay with nondeterminism.
The interplay between deterministic and nondeterministic definitions in Grey Clay provides an opportunity to study constraint-based language generation in a very naive manner. Questions which might well be investigated with the help of Grey Clay include: How do schemas, vocabularies, and nondeterministic constraints which yield cognitively interesting behavior develop? To what degree does this notion of this one aspect of cognition (based on schemas, vocabularies, nondeterminism and constraints) have psycological validity?
Partially Separable Processes and Fuchsia Clay
A controversial issue in the music cognition literature is whether the melody and lyrics of a song form an integrated representation in memory. A related issue concerns the nature of the representation of melody and rhythm (i.e., is the rhythm of a familiar melody stored separately from the pattern of pitches that make up that melody? [] Consider for example the pitch, rhythmic and melodic Processes for first measure of "Little Tune."
m1(p).                 m1(d)                 n1(m)

   3  ------------        1  ------------       3  ------------
   2  ------------        1  ------             2  ------
   3  ------------        1  ------------       3  ------------
   1  ------------        1  ------             2  ------
Fuschia Clay is an extension of Indigo Clay which supports a simple form of fusion of subprocesses into processes. This form bases the fusion on a common command (in this case "play"), provided preconditions which make the processes fusable are met. One can measure the complexities of the pitch and rhythmic subprocesses in isolation and compare them to the complexity of their integration, the melodic process, via the sigma metric. The following equation serves to illuminate the comparison.
sigma(M) = sigma(P) + sigma(D) + sigma(X)
In words, the simplicity of the melody equals the simplicity of the "pitch" subprocess plus the simplicity of the "duration" subprocess plus the simplicity of the process which fuses these latter two processes together. Equations like this one help to focus the important issues of knowledge representation and manipulation. Empirical data gathered by considering such equations in the context of Fuchsia Clay (or other languages tailored for this purpose) shed insight into these issues.
Stepwise Structural Refinement and Saphire Clay
A computational construct which is conceptually consistent with Deutsch's formal representation of an hierarchically organized tone sequence [Deutsch, 1980] is featured in Saphire Clay. This construct enables one to structurally refine processes via ">>", the structural refinement operator. For example, fragment(level1) below simply plays a a note four times. The fragment(level2) program plays 16 notes which take the form of an arpegio played four times.
fragment(level1) = play*4
fragment(level2) = fragment(level1)>>pattern
pattern = sd(4), arpegio, xd(4).
arpegio = play, rp*2, play, rp*2, play, rp*3, play, lp*7.
This modelling mechanism provides an explicit rendering the notion of melodic foreground and background, of the idea so persuasively put forth by Henrich Schenker [Schenker, 1954] that some notes are more important than others. Modelling melodies with this construct places front and center basic architectural principles, including principles of "line", and principles of invariance, which are inherent in the construction of musical melodies.
Conclusion
The Generative Symbol System model described in this paper is strictly procedural in nature. It provides an exercise in the exploration of what can be done in a procdural mode of conception. Generally speaking a model of composition will embrace declarative representations, procedural-to-declarative and procedural-to-declarative knowledge converters, and rich sets of extensible operations and mappings based on the various knowledge representations. The focus on procedural phenomena in the context of the Colored Clay based Generative Symbol Systems discussed in this paper suggests the diversity of rich possibilities for restricted studies of musical modelling.
References
Balaban, Mira, Kemal Ebcioglu, and Otto Laske. Understanding Music with AI: Perspectives on Music Cognition. Cambridge, Massachusetts: The MIT Press, 1992.
Clark, Andy. Being There: Putting Brain, Body, and World Together Again. Cambridge, Massachusetts: A Bradford Book, The MIT Press, 1997.

Deutsch, D. "The processing of structured and unstructured tonal sequences." Percept. and Psychophys. Gardner, Howard. Art, Mind & Brain: A Cognitive Approach to Creativity. New York: Basic Books, Inc., 1982.
Jackendoff, Ray. Consciousness and the Computational Mind. Cambridge, Massachusetts: A Bradford Book, The MIT Press, 1992.
Schenker, Heinrich, Harmony. Edited and annotated by Oswald Jonas and translated by Elisabet Mann Borgese. University of Chicago Press, 1954.
Sloboda, John. The Musical Mind: The cognitive psychology of music. Oxford, Oxford University Press, 1985.
Stevens, Kate, J. Devin McAuley, & Michael S. Humphreys. "Relational Information in Memory for Music: The Interaction of Melody, Rhythm, Text and Instrument." Australasian Cognitive Science Conference, University of Newcastle, September 1997