Understanding Smart Piano Systems: Introduction and Analysis
-
This article introduces pianos available on the market and analyzes the software design implementation of one type of smart piano. The next article will elaborate on the application of smart pianos in teaching and their future development.
Smart pianos are a novel concept in the internet era, connecting with apps via smart devices to guide learners in self-study, lowering the entry barrier to learning, and making piano learning more engaging compared to traditional pianos. Beyond learning, students require extensive practice after lessons, and smart pianos perfectly address the rigid need for professional guidance during practice.
Currently, there are two types of intelligent products related to piano teaching on the market. The first is software-based intelligent piano products, which utilize music recognition technology for intelligence, represented by apps like AI Music Academy. The second is hardware-based intelligent piano products, which collect user performance data through hardware and analyze and present the data via matching software, represented by apps like The One Smart Piano, Piano Master, and Find Piano.
Due to the limitations of music recognition technology, software-based intelligent products are not yet suitable for professional teaching. In practical piano teaching, hardware-based intelligent products are commonly used.
Hardware-based smart pianos are divided into two categories: one connects to smart devices, while the other integrates smart devices seamlessly with the piano. This article focuses on analyzing the first type of smart piano system, providing only a brief introduction to the second type.
This type of smart piano connects digital pianos to devices like smartphones and tablets, allowing users to practice via apps, creating human-machine interaction, as shown in the figure below.
Currently, apps like The One Smart Piano App, Hailun Piano's Piano Master App, and Roland Piano's Rainbow Piano App are available on the market. Users can learn, practice, enjoy music, search for sheet music, and receive intelligent accompaniment through these apps.
This type of smart piano enables users to learn music theory and piano skills by watching instructional videos in the app. These videos are recorded by piano professors or instructors from major music academies. For example, The One's instructional videos are recorded by renowned pianist Lang Lang and teachers from the Central Conservatory of Music, allowing students to attend masterclasses at home.
In the accompaniment function, the most commonly used is the follow-light mode. When a note is played correctly, the indicator light turns blue or green and advances to the next note. If played incorrectly, the light turns red and remains on the current note. The follow-light mode helps learners familiarize themselves with key positions during practice, enabling correct practice without supervision.
Game features enhance learners' interest in piano. For example, The One Smart Piano App includes music games similar to Rhythm Master, while Hailun Piano's App features games like tower defense and spot-the-difference. These games use a waterfall flow mode to guide users in playing, helping them standardize fingering and familiarize themselves with key positions while having fun.
The second type of smart piano integrates digital pianos with traditional pianos, such as Find Smart Piano and BOE Smart Piano. These pianos are modified versions of traditional pianos, replacing the lid and music stand above the keys with a high-definition screen capable of displaying four pages of sheet music simultaneously, as shown below.
These pianos are easy to operate. Users can perform tasks like searching for sheet music or watching performance or instructional videos by tapping the keys corresponding to function icons.
Functionally, they offer remote control via apps to track learning progress, automatic demonstration performances, big data analysis of playing to provide accurate guidance, and a cloud platform offering vast resources like masterclasses and performance videos.
These pianos have distinct teaching features:
This section introduces the software design approach of the first type of smart piano.
The primary challenge in smart piano software systems is the display and synchronized playback of sheet music. To meet these needs, digital sheet music created with computer music typesetting technology is required, along with corresponding parsing and rendering technologies. There are many platforms for creating digital sheet music, such as Sibelius, Finale, Vexflow, and Guido Engine, each with unique features.
Among these tools, the most commonly used and suitable universal exchange format for sheet music is MusicXML. MusicXML is an open XML-based music file format for music exchange and distribution, designed to create a universal standard for Western musical notation.
MusicXML has two types of elements: one identifies how music should be played, often used to export MIDI files, and the other identifies the position of notes on staves, which can be exported as XML files and rendered into readable sheet music by software like Finale.
For displaying digital sheet music, there are single-layer and double-layer modes.
1) Single-Layer Mode
The single-layer mode uses specialized stave font libraries or musical symbol images to display sheet music and playback markers on the same layer. Sheet music information can be parsed from MusicXML.
2) Double-Layer Mode
The double-layer mode displays the entire sheet music in two layers: a background layer with the original stave file (image or PDF) and a foreground layer for other markers.
A major advantage of digital sheet music is its playability, allowing synchronized accompaniment or other parts, with adjustable playback speed. There are two playback methods: MIDI and audio.
1) MIDI Playback
MIDI files are small, saving storage space and data traffic. They are also easy to edit, allowing playback speed changes without quality loss.
However, MIDI files do not produce sound on their own and require software sound sources. Implementation varies by platform, operating system, and programming language, making development challenging.
2) Audio Playback
Audio playback does not require sound sources and can use real instrument recordings, offering better realism and sound quality than MIDI. However, audio files (e.g., WAV, MP3) consume more storage and data. Speed adjustments degrade audio quality and may cause delays, hindering synchronization with display.
The most intelligent feature of smart pianos is their ability to provide automatic error correction and scoring during play. The evaluation system focuses on accuracy, dynamics, rhythm, and duration.
1) Accuracy
Accuracy evaluation includes missed notes, extra notes, and wrong notes. The process is as follows:
There is currently no theoretical support for determining weights, making it a challenge in quantifying music evaluation standards.
2) Rhythm
Rhythm evaluation faces two issues. First, performance timing is never perfectly accurate, requiring redundancy calculations based on note duration and tempo. Second, rhythm cannot be fully quantified in computer language.
3) Duration
Actual note duration = key release time - key press time.
Like rhythm, duration can only be measured with rigid standards.
4) Dynamics
Dynamics cannot be measured with specific values to determine correctness or quality, so they are often ignored in evaluation.
In summary, pitch accuracy is the most reasonable and heavily weighted aspect of evaluation, while rhythm and duration have minor weights, and dynamics are negligible.
Sheet music in software can be displayed in two layout modes: line-breaking and non-line-breaking.
1) Line-Breaking Mode
Using a layout approach close to traditional paper sheet music standards, when the musical notation exceeds the page width, it automatically wraps to the next line, as shown in the figure below.
The biggest advantage of this mode is that it closely resembles the layout of paper sheet music, helping learners develop proper score-reading habits. However, when using a piano connected to a smart device (usually a tablet or smartphone), the screen size is typically smaller than that of paper sheet music. To allow users to preview the next line of music, at least two lines must be displayed on one screen, resulting in relatively smaller notation that can be challenging to read while playing.
To facilitate page-turning for players, pianos connected to smart devices generally use automatic scrolling. Some smart pianos even incorporate two additional pedals for page-turning, freeing up both hands for uninterrupted playing.
2) Non-line-breaking Mode
The entire musical score is arranged in a single line, and during playback, the notation automatically scrolls to the left to achieve the final display effect, as shown below.
Since this mode only requires one line of notation on the screen, its biggest advantage is that the notation can be displayed in a larger size. However, this mode has several drawbacks. First, it requires keeping the clef, time signature, and key signature constantly at the beginning of the notation, disrupting the score's integrity. When repeat signs or other jump marks appear in the score, the program needs to jump accordingly, making it difficult for players to follow the display. Additionally, this display method is not conducive to developing proper score-reading habits.
Currently, intelligent piano software commonly employs two main display modes: Follow-Score Mode and Performance Mode.
1) Follow-Score Mode
Based on the traditional staff notation, this mode adds visual indicators (usually a playback line or other markers) to highlight the notes to be played, as shown below. When the player correctly plays the notes at the corresponding time, the "marker" moves to the next set of notes to be played. In this mode, there is no metronome restriction, and if a note is played incorrectly, the marker remains in place.
Follow-Score Mode is also known as "Correct-to-Advance" or "Step-by-Step" mode. In this mode, players don't need to worry much about timing and rhythm, as they have ample time to read the notation. This mode helps learners familiarize themselves with the score gradually.
2) Performance Mode
Performance Mode can also be called "Correction Mode" or "Follow-Music Mode." The correction and scoring mechanisms are primarily applied in this mode. Like Follow-Score Mode, it is based on staff notation. Once the metronome starts, the player must try to play the correct notes at the corresponding times. The software displays the performance results in real-time on the score for correction and scoring purposes. This mode helps learners adhere to the performance standards of each piece and continuously improve their playing skills.
The above image shows the evaluation screenshot of this mode. A green note head indicates that the player performed the note with good accuracy, rhythm, and timing control. A yellow note head indicates slightly off rhythm or timing. A red note head indicates extra or wrong notes. An unchanged note head indicates a missed note.
In the Waterfall Flow mode, blocks or bars fall from the top of the screen like a waterfall, while the keyboard is displayed at the bottom. The player completes the performance by touching the corresponding keys on the screen or playing them on a connected piano, as shown below.
Waterfall Flow games can help familiarize players with key positions during play. Learners can complete a simple piano piece by memorizing key positions, and even those with minimal piano skills can play a complete piece with prompts, enhancing the fun and sense of achievement in playing. The line chart below compares the performance scores of students using traditional staff notation versus the Waterfall Flow mode.
The results show that scores achieved using the Waterfall Flow mode are higher than those using traditional staff notation. The Waterfall Flow feature can quickly improve a beginner's accuracy from 15.7% to 76.6%, demonstrating that intelligent pianos can help complete novices learn to play, achieving the goal of popularizing piano education.
