Introduction

One of the goals of this research is the analysis and modeling of left hand articulations in a guitar performance. Here, we describe the gesture analysis performed from collected data, using our proposed gesture acquisition system. The study, is divided in two parts:

1. Study of collected left hand articulations
2. Fingering analysis for the most commonly played chords.

All the recordings have been performed using our augmented guitar 0.1 prototype.

Study of collected left hand articulations

Here is a list of some of the left hand articulations recorded. They are ordered from a macro-level to a micro-level moving scale (i.e. moving the whole hand or slightly moving one finger, respectively).

Finger bars

All strings are pressed by a finger, starting at the first fret; then, ascending fret by fret until the 10th fret; next a pause of a beat; and finally, going down to the first fret. The change of fret occurs every 4 beats at 60[bpm].

Figure 1: Relative capacitance for ascending and descending bar positions from frets 1 to 10 and 10 to 1. High values correspond to the targeted frets, and low values correspond to the other ones.

Chromatic scales

set of chromatic scales, one for each string, starting with the open string and playing an ascending scale until the 10th. fret. The change of fret occurs every 4 beats at 60[bpm].

Figure 2: Relative capacitance for chromatic scales, for frets from 1 to 10, playing the 6 strings independently. High values correspond to the targeted frets, and low values correspond to the other ones.

Grace notes

Ascending grace notes start at the second fret, playing with an ascending grace note from the previous fret until the 10th fret. In an analogous way, recordings of descending grace notes start at the 9th fret and continued to the 1st one. The change of fret occurs every 4 beats at 60[bpm].

Figure 3: Relative capacitance for an ascending grace note between the 4th (gray) and 5th (black) fret at 1st string.

Diatonic scale

2 octaves of a descending A major scale played at first position. The played frets follow the sequence: 5-4-7-5-7-6-4-7-6-4-7-5-4-7-5. The change of note occurs every beat at 60[bpm].

Figure 4: Measured relative capacitance for relevant frets during 2 octaves in a descendent A major scale, and the pitch estimation extracted from the audio. The active frets follow the sequence: 5-4-7-5-7-6-4-7-6-4-7-5-4-7-5.

Basic arpeggios

The proposed arpeggio was played at the first position (strings 3-2-1-1) and with a bar at the fifth fret (strings 4-3-2-2). The change of note occurs every beat at 60[bpm].

Figure 5: Measured relative capacitance for relevant frets during arpeggio playing, and the pitch estimation extracted from the audio. The upper graph corresponds to the performance at the first position. The lower one corresponds to the performance using the bar at the fifth fret.

Slurs

we played a chromatic scale at 60[bpm], changing to the next semitone every 4 beats. We apply the hammer-on articulation at the beginning of the 2nd beat and the hammer-off at the beginning of the 3rd beat.

Figure 6: Measured relative capacitance for relevant frets playing at the 8th fret and applying the hammer-on and hammer-off at the 9th fret, on the 3rd string.

Vibrato

each note is played twice: first, it is played normally and after 2 beats, it is repeated but applying a vibrato. The change of fret occurs every 4 beats at 60[bpm].

Figure 7: Relative capacitance for a 1st string and 10th fret. The first two beats are played without vibrato and, after a new attack given by the right hand, the note is played again but applying vibrato.

Fingering analysis for the most commonly played chords

Here, we present the methodology to recognize the most common finger positions while playing chords.

Acquisition

Table 1 shows finger activation combinations for each default fingering position. Each digit corresponds to the number of fingers pressing at the same fret. These positions can be played in different hand positions and in different strings. 6 refers to bar activation, 1 refers to 1 finger activation at any string, 2 refers to 2 finger activation at the same fret at any strings, and 3 refers to 3 finger activation at the same fret at any strings. The highlighted combinations represent the recorded cases.

Table 2 shows a detailed list of all the recorded positions, specifying the played strings. Each recording includes the hand position moving from fret 1 to 7. The s1..s6 stands for the played string. Each string specification follows an ascending order from finger 1 to 4.

Analysis

Once the gesture data is collected, we create, for each position, a pattern for frets 1 to 4 (relative to the position of the index finger) moving the position horizontally on the fretboard (moving the hand from low pitches to high pitches) and vertically (moving the strings from low pitch to high pitch). This pattern is created by computing the mean for all the equivalent positions (i.e. each row in Table 2). Figure 8 shows patterns for finger positions 1, 2, 3 and 4 with respect to the fret position of the index finger, collected for the 1010b position. Each column corresponds to the same pattern played at difference reference frets (from 1 to 7), and each row corresponds to the same pattern played at different strings (s4s6, s3s5, s2s4, and s1s3).

Figure 8: patterns for finger positions 1, 2, 3 and 4 with respect to the fret position of the index finger, collected for the 1010b position.

Repeating this process for each of the proposed positions in Table 2, we can compute the position patterns, by computing the means, as shown in Figure 9. This means that the absolute values and slopes are equivalent for each row, that is, the same pattern is obtained by playing at different reference frets by moving the hand horizontally on the fretboard, and for each column, that is, the same pattern is obtained by playing at different strings.

Figure 9: Patterns obtained from means and standard deviations for all the recordings at different finger positions.

Automatic recognition

We have 22 categories (including 75 possible finger combinations) recorded at 7 reference fret positions, that is, a data-set with 525 recordings. We analyzed whether an automatic classifier might identify them. For simplicity, we use a K-nearest neighbours classifier (with K=3) and evaluate using 10-fold cross validation.Results provide an overall accuracy of 44,6% (weighted averaged precision = 0.449, weighted averaged recall = 0.446, weighted averaged f-measure = 0.435). Figure 10 shows results of automatic for collapsed categories classification using K-nearest neighbours with K=3. Rows indicate categories that should be classified and columns indicate automatically classified categories. Indexes follow these categories: (1)1000, (2) 1010, (3) 1100, (4) 1110, (5) 1200, (6) 2000, (7) 2100, (8) 2200, (9) 6000, (10) 6010, (11) 6020, (12) 6100, (13) 6110, (14) 6120, and (15) 6210.

Figure 10: Results of automatic classification