====== Creating an FM Synthesizer in Python ======


===== Objective: =====

This project implements an FM (Frequency Modulation) Synthesizer in Python. The synthesizer processes MIDI files and generates audio in real-time, focusing on emulating polyphonic instruments. The project also aims to explore FM synthesis techniques and create custom audio effects for MIDI playback.

===== App =====


<code python:fm_synthesizer.py>
import mido
import numpy as np
import sounddevice as sd
import time

class Note:
    def __init__(self, note_num, velocity, start_time, program_num=None):
        self.note_num = note_num
        self.velocity = velocity
        self.start_time = start_time
        self.end_time = None
        self.samples = None
        self.position = 0
        self.program_num = program_num

class FMSynthesizer:
    def __init__(self, sample_rate=44100, block_size=1024, max_polyphony=24):
        self.sample_rate = sample_rate
        self.block_size = block_size
        self.max_polyphony = max_polyphony
        self.active_notes = {}
        self.note_buffer = np.zeros(block_size)
        
    def fm_synth(self, note, velocity, duration):
        carrier_freq = 440 * 2**((note - 69) / 12)
        modulator_freq = carrier_freq * 1/2 + carrier_freq * 2/6 + carrier_freq * 4/10
        modulation_index = 0.5
        
        # Add a tiny bit of padding to avoid edge clicks
        duration += 0.01
        
        t = np.linspace(0, duration, int(self.sample_rate * duration), endpoint=False)
        modulator = np.sin(2 * np.pi * modulator_freq * t) * modulation_index
        carrier = np.sin(2 * np.pi * carrier_freq * t + modulator)
        
        # Simple envelope
        attack_time = 0.05
        release_time = 0.05
        attack_samples = int(self.sample_rate * attack_time)
        release_samples = int(self.sample_rate * release_time)
        
        envelope = np.ones_like(t)
        # Linear attack and release
        if attack_samples > 0:
            envelope[:attack_samples] = np.linspace(0, 1, attack_samples)
        if release_samples > 0:
            envelope[-release_samples:] = np.linspace(1, 0, release_samples)
        
        return carrier * envelope * (velocity / 127)

    def note_on(self, note_num, velocity, start_time, program_num):
        if len(self.active_notes) >= self.max_polyphony:
            oldest_time = float('inf')
            oldest_note = None
            for num, note in self.active_notes.items():
                if note.start_time < oldest_time:
                    oldest_time = note.start_time
                    oldest_note = num
            if oldest_note is not None:
                del self.active_notes[oldest_note]
            
        new_note = Note(note_num, velocity, start_time, program_num)
        self.active_notes[note_num] = new_note
        
    def note_off(self, note_num, end_time):
        if note_num in self.active_notes:
            self.active_notes[note_num].end_time = end_time

    def get_audio_block(self):
        self.note_buffer.fill(0)
        
        notes_to_process = list(self.active_notes.items())
        finished_notes = []
        
        for note_num, note in notes_to_process:
            if note_num not in self.active_notes:
                continue
                
            if note.samples is None:
                duration = 1
                if note.end_time is not None:
                    duration = max(1, note.end_time - note.start_time)
                try:
                    note.samples = self.fm_synth(note.note_num, note.velocity, duration)
                except Exception as e:
                    print(f"Error generating samples for note {note_num}: {e}")
                    finished_notes.append(note_num)
                    continue
                
            remaining = len(note.samples) - note.position
            if remaining <= 0:
                finished_notes.append(note_num)
                continue
                
            samples_to_add = min(remaining, self.block_size)
            try:
                self.note_buffer[:samples_to_add] += \
                    note.samples[note.position:note.position + samples_to_add]
                note.position += samples_to_add
            except Exception as e:
                print(f"Error mixing samples for note {note_num}: {e}")
                finished_notes.append(note_num)
        
        for note_num in finished_notes:
            self.active_notes.pop(note_num, None)
            
        # Gentle limiting to prevent harsh clipping
        max_val = np.max(np.abs(self.note_buffer))
        if max_val > 1.0:
            self.note_buffer = self.note_buffer / (max_val * 1.2)  # Add a tiny headroom
            
        return self.note_buffer

def play_midi_file(midi_path, selected_instruments=None):
    midi = mido.MidiFile(midi_path)
    synth = FMSynthesizer()
    program_changes = {}

    def audio_callback(outdata, frames, time, status):
        try:
            block = synth.get_audio_block()
            outdata[:, 0] = block
        except Exception as e:
            print(f"Error in audio callback: {e}")
            outdata.fill(0)
        
    with sd.OutputStream(channels=1, 
                        callback=audio_callback,
                        samplerate=synth.sample_rate,
                        blocksize=synth.block_size):
        
        start_time = time.time()
        current_time = 0
        
        for msg in midi:
            current_time += msg.time
            wait_time = start_time + current_time - time.time()
            if wait_time > 0:
                time.sleep(wait_time)
                
            try:
                if msg.type == 'program_change':
                    program_changes[msg.channel] = msg.program
                    print(f"Program change: Channel {msg.channel} -> Program {msg.program}")
                elif msg.type == 'note_on':
                    if msg.velocity > 0:
                        if selected_instruments is None or program_changes.get(msg.channel) in selected_instruments:
                            synth.note_on(msg.note, msg.velocity, current_time, program_changes.get(msg.channel))
                    else:
                        synth.note_off(msg.note, current_time)
                elif msg.type == 'note_off':
                    synth.note_off(msg.note, current_time)
            except Exception as e:
                print(f"Error processing MIDI message: {e}")
        
        time.sleep(2)

if __name__ == "__main__":
    play_midi_file("C:/temp/canyon.mid", selected_instruments=None)
    # Example: Play only piano (Program 1) and electric piano (Program 5), [1, 5]
    #play_midi_file("C:/Users/unapa/Downloads/DXdiag.mid", selected_instruments=[33, 78, 16, 18, 53, 27, 29, 30])

</code>

===== Script Overview =====


  * **Filename:** `fm_synthesizer.py`
  * **Dependencies:** `mido`, `numpy`, `sounddevice`, `time`
  * **Features:**
  - **FM Synth Engine:** Generates FM-based audio for MIDI notes.
  - **Polyphony:** Supports multiple simultaneous notes with a configurable maximum polyphony limit.
  - **MIDI Playback:** Reads and processes MIDI files to generate real-time audio.
  - **Instrument Selection:** Allows filtering specific instruments for playback.

===== Script Execution =====


==== FM Synthesis Engine: ====

  * **Carrier and Modulator Frequencies:** Derived from the MIDI note number.
  * **Modulation Index:** Affects the timbre by varying modulation depth.
  * **Envelope:** Simple attack-release envelope applied to smooth transitions.

==== Polyphonic Note Handling: ====

  * **Dynamic Management:** Tracks active notes, ensuring the maximum polyphony limit isn’t exceeded.
  * **Automatic Note Cleanup:** Frees resources of finished notes to manage performance.

==== Real-Time Audio Playback ====

  * **Callback Architecture:** Processes audio blocks in real-time using `sounddevice`.
  * **Audio Limiting:** Avoids clipping by gently limiting the output amplitude.

==== MIDI Parsing: ====

  * Processes MIDI messages (e.g., `note_on`, `note_off`, `program_change`).
  * Provides the option to select specific instruments via their MIDI program numbers.

==== Command-Line Execution: ====

  * Modify the script to accept a MIDI file as a parameter for better flexibility.
  * Example:
<code>python fm_synthesizer.py example.mid</code>


==== Error Handling: ====

     * Detects and reports errors during audio generation, MIDI parsing, or playback.

===== Usage Example =====

To play a MIDI file (e.g. `canyon.mid`), run the script as shown below. You can optionally specify a list of instruments to filter:

<code>play_midi_file("C:/temp/canyon.mid", selected_instruments=[1, 5])</code>

- **Without Filtering:**  
   All instruments in the MIDI file will be played.
   
- **Filtered Playback:**  
   Example: Play only pianos (Program 1) and electric pianos (Program 5).

===== Limitations =====

1. **Simplistic Envelope:**  
   The attack-release envelope could be extended with sustain and decay stages for greater realism.
   
2. **Basic Modulation:**  
   The FM synthesis implementation uses a fixed modulation scheme. Extending it with dynamic modulation indexes or additional oscillators could improve the tonal range.

3. **No Visualization:**  
   Currently, the synthesizer does not include any graphical or waveform visualizations.

4. **Input Handling:**  
   The script does not yet parse command-line arguments for file paths or instrument filters.

===== Final Note =====


This FM synthesizer provides a starting point for experimenting with audio synthesis and real-time MIDI playback in Python. Its modular design makes it easy to extend with additional features, such as better envelopes, more advanced synthesis techniques, or visualization tools.


====== Appendix: Monophonic Synthesizer ======  

This section covers a **simple mono synthesizer** that processes a single piano track from a MIDI file, synthesizing FM-based audio for each note and playing it in real time.  

<code python>  
import mido  # For MIDI parsing
import numpy as np  # For waveform synthesis
import sounddevice as sd  # For real-time audio playback

def read_midi(file_path):
    midi = mido.MidiFile(file_path)
    return midi

def fm_synth(note, velocity, duration, sample_rate=44100):
    # Example parameters for FM synthesis
    carrier_freq = 440 * 2**((note - 69) / 12)  # MIDI note to frequency
    modulator_freq = carrier_freq * 2 + carrier_freq * 4 + carrier_freq * 6 + carrier_freq * 8
    modulation_index = 0.33
    
    t = np.linspace(0, duration, int(sample_rate * duration), endpoint=False)
    modulator = np.sin(2 * np.pi * modulator_freq * t) * modulation_index
    carrier = np.sin(2 * np.pi * carrier_freq * t + modulator)
    
    return carrier * velocity / 127

def process_piano_track(midi):
    sample_rate = 44100
    for track in midi.tracks:
        for msg in track:
            if msg.type == 'note_on' and msg.velocity > 0:  # Note on
                duration = msg.time / midi.ticks_per_beat  # Estimate duration from MIDI timing
                note_audio = fm_synth(msg.note, msg.velocity, duration, sample_rate)
                
                # Play the synthesized audio for the note
                sd.play(note_audio, samplerate=sample_rate)
                sd.wait()  # Wait until the audio is finished playing

midi = read_midi("C:/temp/canyon.mid")
process_piano_track(midi)
</code>  

===== Overview =====  

  * **Purpose:** To demonstrate a basic FM synthesis-based mono synthesizer that processes MIDI piano tracks. Each note is synthesized and played one at a time.  
  * **Limitations:**  
    - **Mono Only:** The synthesizer processes and plays a single note at a time. Overlapping notes or chords are not supported.  
    - **No Envelope Control:** The FM synthesis does not include dynamic ADSR envelopes, which might result in abrupt starts/stops.  
    - **Simplistic Waveform Modulation:** Only sine waves are used as the carrier and modulator.  

===== Customizing the Modulation Signal =====  

To achieve different sound characteristics, you can modify the `fm_synth` function to generate various modulation waveforms. Here are examples:  

**1. Triangle Wave Modulation:**  
<code python>  
modulator = 2 * np.arcsin(np.sin(2 * np.pi * modulator_freq * t)) / np.pi * modulation_index  
</code>  

**2. Sawtooth Wave Modulation:**  
<code python>  
modulator = 2 * (t * modulator_freq - np.floor(t * modulator_freq + 0.5)) * modulation_index  
</code>  

**3. Square Wave Modulation:**  
<code python>  
modulator = np.sign(np.sin(2 * np.pi * modulator_freq * t)) * modulation_index  
</code>  

Each of these alternatives changes the timbre of the output, providing greater flexibility for sound design.  

===== Final Notes =====  

  * **Expected Behavior:** This script processes the MIDI file track by track, playing each note sequentially. It is best suited for simple piano tracks with minimal overlap between notes.  
  * **Recommendations:**
    - Use this script for experimentation or as a starting point for more advanced synthesizer projects.  
    - To handle polyphony or add envelopes, consider integrating this code with the main FM synthesizer described earlier.  

By customizing the modulation signal and adjusting parameters like `modulation_index`, `carrier_freq`, and `modulator_freq`, you can experiment with creating unique sound textures and effects.