News & Events

Computer Assisted Music-Making Systems

Christodoulos Benetatos

Supervised by Zhiyao Duan

601 Computer Studies Building

Computer-assisted music-making (CAMM) systems range from real-time musical partners to offline score-processing assistants, making systems with different purposes difficult to compare. This dissertation develops a taxonomy along seven dimensions: type, intelligence, interactivity, role, input and output, score dependency, and required user proficiency. It examines three systems that occupy distinct regions of this taxonomy and one framework that supports the deployment of interactive music systems.

BachDuet is a real-time partner agent for Baroque-style counterpoint improvisation. A recurrent neural network responds to a human keyboard performer under strict timing constraints, producing simultaneous duet improvisations without a predetermined score or chord progression. User studies, listening tests, and objective evaluations demonstrate the feasibility of this balanced human–machine interaction.

Draw&Listen is an interactive music-inpainting assistant designed for users without formal musical training. Users guide melody generation by drawing pitch and note-density curves. A variational autoencoder disentangles relative pitch, relative rhythm, and musical context, translating intuitive visual input into melodies that match surrounding measures. Objective evaluations and user studies show that the interaction is intuitive and that the model compares favorably with rule-based and genetic-algorithm baselines.

The third system performs guitar tablature and left-hand fingering assignment for symbolic scores. A graph-aware Transformer predicts string choices, while a deterministic ergonomic decoder constructs stable left-hand configurations and uses Viterbi decoding to produce playable fingering trajectories. Supervised and reinforcement-learning objectives support complementary goals, and editable ergonomic rewards, user-provided anchors, and single-score adaptation enable personalization to individual players. Expert evaluations sup-port the decoder’s feasibility judgments and indicate ergonomic benefits from reward-driven optimization.

Finally, Euterpe addresses supporting infrastructure rather than musical behavior. This browser framework abstracts real-time audio and MIDI input, scheduling, concurrent agent execution, communication, and visualization. BachDuet and JazzImprov case studies demonstrate support for symbolic and audio agents, while user and developer feedback indicate accessible deployment and reduced implementation effort.

Together, these contributions show how musical structure, interaction design, user expertise, and supporting infrastructure can be jointly considered when building intelligent systems that assist music making without adopting a single model of human–computer collaboration.