Gesture Analysis

Introduction

One of the goals of this research is the analysis and modeling of left hand articulations in a guitar performance. Here, we describe the gesture analysis performed from collected data, using our proposed gesture acquisition system. The study, is divided in two parts:

  1. Study of collected left hand articulations
  2. Fingering analysis for the most commonly played chords.

All the recordings have been performed using our augmented guitar 0.1 prototype.

Study of collected left hand articulations

Here is a list of some of the left hand articulations recorded. They are ordered from a macro-level to a micro-level moving scale (i.e. moving the whole hand or slightly moving one finger, respectively).

Finger bars

All strings are pressed by a finger, starting at the first fret; then, ascending fret by fret until the 10th fret; next a pause of a beat; and finally, going down to the first fret. The change of fret occurs every 4 beats at 60[bpm].

Figure 1: Relative capacitance for ascending and descending bar positions from frets 1 to 10 and 10 to 1. High values correspond to the targeted frets, and low values correspond to the other ones.

Chromatic scales

set of chromatic scales, one for each string, starting with the open string and playing an ascending scale until the 10th. fret. The change of fret occurs every 4 beats at 60[bpm].

Figure 2: Relative capacitance for chromatic scales, for frets from 1 to 10, playing the 6 strings independently. High values correspond to the targeted frets, and low values correspond to the other ones.

Grace notes

Ascending grace notes start at the second fret, playing with an ascending grace note from the previous fret until the 10th fret. In an analogous way, recordings of descending grace notes start at the 9th fret and continued to the 1st one. The change of fret occurs every 4 beats at 60[bpm].

Figure 3: Relative capacitance for an ascending grace note between the 4th (gray) and 5th (black) fret at 1st string.

Diatonic scale

2 octaves of a descending A major scale played at first position. The played frets follow the sequence: 5-4-7-5-7-6-4-7-6-4-7-5-4-7-5. The change of note occurs every beat at 60[bpm].

Figure 4: Measured relative capacitance for relevant frets during 2 octaves in a descendent A major scale, and the pitch estimation extracted from the audio. The active frets follow the sequence: 5-4-7-5-7-6-4-7-6-4-7-5-4-7-5.

Basic arpeggios

The proposed arpeggio was played at the first position (strings 3-2-1-1) and with a bar at the fifth fret (strings 4-3-2-2). The change of note occurs every beat at 60[bpm].

Figure 5: Measured relative capacitance for relevant frets during arpeggio playing, and the pitch estimation extracted from the audio. The upper graph corresponds to the performance at the first position. The lower one corresponds to the performance using the bar at the fifth fret.

Slurs

we played a chromatic scale at 60[bpm], changing to the next semitone every 4 beats. We apply the hammer-on articulation at the beginning of the 2nd beat and the hammer-off at the beginning of the 3rd beat.

Figure 6: Measured relative capacitance for relevant frets playing at the 8th fret and applying the hammer-on and hammer-off at the 9th fret, on the 3rd string.

Vibrato

each note is played twice: first, it is played normally and after 2 beats, it is repeated but applying a vibrato. The change of fret occurs every 4 beats at 60[bpm].

Figure 7: Relative capacitance for a 1st string and 10th fret. The first two beats are played without vibrato and, after a new attack given by the right hand, the note is played again but applying vibrato.

Fingering analysis for the most commonly played chords

Here, we present the methodology to recognize the most common finger positions while playing chords.

Acquisition

Table 1 shows finger activation combinations for each default fingering position. Each digit corresponds to the number of fingers pressing at the same fret. These positions can be played in different hand positions and in different strings. 6 refers to bar activation, 1 refers to 1 finger activation at any string, 2 refers to 2 finger activation at the same fret at any strings, and 3 refers to 3 finger activation at the same fret at any strings. The highlighted combinations represent the recorded cases.

Table 1: Finger activation combinations.
1st. fret 1finger/fret 2fingers/fret 3fingers/fret
bar
6000
6001
6010
6011
6100
6101
6110
6111
6200
6201
6210
6020
6021
6120
6300
6030
6003
1 finger
1000
1001
1010
1011
1100
1101
1110
1111
1200
1201
1210
6020
6021
6120
1300
1030
1003
2 fingers  
2000
2100
2200
 

Table 2 shows a detailed list of all the recorded positions, specifying the played strings. Each recording includes the hand position moving from fret 1 to 7. The s1..s6 stands for the played string. Each string specification follows an ascending order from finger 1 to 4.

Table 2: Detailed list of recorded positions.
Position Played strings
1000 s1, s2, s3, s4, s5, s6
1010(a) s5s6, s4s5, s3s4, s2s3, s1s2
1010(b) s4s6, s3s5, s2s4, s1s3
1010(c) s3s6, s2s5,  s1s4
1100(a) s5s6, s4s5, s3s4, s2s3, s1s2
1100(b) s4s6, s3s5, s2s4, s1s3
1100(c) s3s6, s2s5,  s1s4
1110(a) s5s4s6, s4s3s5, s3s2s4, s2s1s3
1110(b) s4s5s6, s3s4s5, s2s3s4, s1s2s3
1200(a) s5s6s4, s4s5s3, s3s4s2, s2s3s1
1200(b) s4s6s5, s3s5s4, s2s4s3, s1s3s2
2000 s6s5, s5s4, s4s3, s3s2, s2s1
2100(a) s6s4s5, s5s3s4, s4s2s3, s3s1s2
2100(b) s5s4s6, s4s3s5, s3s2s4, s2s1s3
2200 s5s3s4s2, s4s2s3s1
6000 full, half
6010 s5, s2
6020 s5s4, s4s3
6100 s5, s2 
6110 s3s5, s2s4 
6120 s3s5s4, s2s4s3
6210 s4s3s5, s3s2s4

 

Analysis

Once the gesture data is collected, we create, for each position, a pattern for frets 1 to 4 (relative to the position of the index finger) moving the position horizontally on the fretboard (moving the hand from low pitches to high pitches) and vertically (moving the strings from low pitch to high pitch). This pattern is created by computing the mean for all the equivalent positions (i.e. each row in Table 2). Figure 8 shows patterns for finger positions 1, 2, 3 and 4 with respect to the fret position of the index finger, collected for the 1010b position. Each column corresponds to the same pattern played at difference reference frets (from 1 to 7), and each row corresponds to the same pattern played at different strings (s4s6, s3s5, s2s4, and s1s3).

Figure 8: patterns for finger positions 1, 2, 3 and 4 with respect to the fret position of the index finger, collected for the 1010b position.

Repeating this process for each of the proposed positions in Table 2, we can compute the position patterns, by computing the means, as shown in Figure 9. This means that the absolute values and slopes are equivalent for each row, that is, the same pattern is obtained by playing at different reference frets by moving the hand horizontally on the fretboard, and for each column, that is, the same pattern is obtained by playing at different strings.

Figure 9: Patterns obtained from means and standard deviations for all the recordings at different finger positions.

Automatic recognition

We have 22 categories (including 75 possible finger combinations) recorded at 7 reference fret positions, that is, a data-set with 525 recordings. We analyzed whether an automatic classifier might identify them. For simplicity, we use a K-nearest neighbours classifier (with K=3) and evaluate using 10-fold cross validation.Results provide an overall accuracy of 44,6% (weighted averaged precision = 0.449, weighted averaged recall = 0.446, weighted averaged f-measure = 0.435). Figure 10 shows results of automatic for collapsed categories classification using K-nearest neighbours with K=3. Rows indicate categories that should be classified and columns indicate automatically classified categories. Indexes follow these categories: (1)1000, (2) 1010, (3) 1100, (4) 1110, (5) 1200, (6) 2000, (7) 2100, (8) 2200, (9) 6000, (10) 6010, (11) 6020, (12) 6100, (13) 6110, (14) 6120, and (15) 6210.

Figure 10: Results of automatic classification