MISCELLANEOUS TECHNICAL ARTICLES BY Dr A R COLLINS

JavaScript Spectrogram

JavaScript Spectrogram Library

This article describes spectrogram displays, both waterfall and rasterscan styles and represents the user guide for the Spectrogram JavaScript library. This library facilitates the building spectrogram displays for web pages. All the displays on this page were created using the Spectrogram-2v00.js library.

Spectrogram basics

The Spectrogram shows the behavior of a signal's spectrum as a function of time. They are commonly used in the fields of music, sonar, radar, speech processing and the like. A spectrogram is formed by plotting each successive spectrum as a line of pixels, the pixel colors representing signal amplitude in each frequency bin. The pixel data from each newly arrived spectrum is plotted adjacent to the preceding line so that a rectangular image of the signal's behavior builds up over time.

Spectrograms are instances of the more general 3D displays Rasterscan or Waterfall. A spectrogram has X,Y dimensions of frequency and time but waterfall and raster scans can display quite arbitrary units on each axis. This article provides JavaScript software for generating both Waterfall and Rasterscan displays.

The spectrogram on the right is in Waterfall format displaying the spectra of a audio track. The spectra are generated using the Web Audio API. The AudioContext instance takes the left channel of the audio track and routes it to a Web Audio Analyser node which generates the frequency spectrum of the latest audio data at each call to the node's getByteFrequencyData method. The JavaScript array holding the returned spectrum becomes the input buffer for a Waterfall display.

The code snippet below demonstrates the use of the Spectrogram module. The code shows the constructor of an object that creates a Waterfall object. It has 'begin' and 'halt' methods to start and stop the regular waterfall display updates.

function AudioSpectrogram(analyserNode, cvsID)
{
  const frqBuf = new Uint8Array(analyserNode.frequencyBinCount); // 1024
  const wfNumPts = 50*analyserNode.frequencyBinCount/128; // 400 +ve freq bins
  const wfBufAry = {buffer: frqBuf};
  const wf = new Waterfall(wfBufAry, wfNumPts, wfNumPts, "right", {});    
  const canvas = document.getElementById(cvsID);
  const ctx = canvas.getContext("2d");

  this.playing = false;
  this.begin = ()=>{
    wf.start();
    this.playing = true;
    this.drawOnScreen();
  };

  this.halt = ()=>{
    wf.stop();
    this.playing = false;
  };

  this.drawOnScreen = ()=>{
    analyserNode.getByteFrequencyData(frqBuf, 0);
    ctx.drawImage(wf.offScreenCvs, 0, 0);
    if (this.playing) requestAnimationFrame(this.drawOnScreen);
  };
}

Waterfall displays

Figure 1. Waterfall spectrogram scrolling DOWN.

The basic waterfall display is aptly named as the data flows down the screen as shown in Fig 1. The effect comes about by shifting the block of previously stacked spectra down the screen by one line and the new line of data is added at the top, the oldest line of data disappearing off the bottom of the display.

Building the waterfall display is a two step process. The raw waterfall is built off-screen by directly writing each pixels color into the memory buffer of an off-screen canvas. Each data point is represented as a single pixel and successive lines are writing as adjacent rows of pixels. This method necessarily fixes the waterfall dimension to the length of the data vectors to be plotted and to the number of data lines in the display. The second step is to use this off-screen canvas as a source for drawing an image of the Waterfall in the on-screen canvas using the canvas.writeDataImage method thus allowing on-the-fly scaling of the data for presentation in any required size.

Test Signal Specs
Type:         Linear frequency sweep
Minimum freq: 500Hz
Maximum freq: 1000Hz
Sweep time:   2sec
Background:   White noise
Signal Processing
Sample frequency:   8192 smp/sec
FFT length:         512 samples
Spectral line rate: 16/sec (real time)
Spectrum period:    62.5 msec

The waterfall display in Fig. 1 shows the first 200 frequency points plotted as a function of time. The current time and newest data is always at the top of the screen. The line rate is set at 40 line/sec and the raw waterfall is set to holds 180 lines, the oldest line, written 4.5 sec ago, is at the bottom. The signal is a simulated FFT of a frequency sweep signal in a background of white noise (set inset panel for details). To allow comparisons among the different style displays, all the following spectrograms use these same input data.

The other possible styles of waterfall display are shown in Fig 2. Fig 2a shows data scrolling upward ie. the oldest data is at the top of the screen and the newest data at the bottom. Spectrogram are often shown scrolling the data horizontally. Fig 2b shows a horizontal waterfall scrolling to the right and Fig 2c shows one scrolling to the left.

Other waterfall display styles

Figure 2a. Waterfall with direction UP and line rate 30 lines/sec.

Figure 2b. Waterfall with direction RIGHT and line rate 10 lines/sec.

Figure 2c. Waterfall with direction LEFT and line rate 20 lines/sec.

Rasterscan displays

Rasterscan displays plot each new line of data adjacent to the previous line progressing across or down (or up) the screen. The existing data is not moved. When the input point reaches the far end of the display area it jumps back to the starting edge and proceeds down or across the display again, each new line overwriting the oldest line on the screen. The rasterscan shows a fixed number of lines representing a fixed period of the most recent signal data.

Fig 3 shows several rasterscan displays all using the same data as Fig 1. Note: To make clear where the current time is on the display, a white line is drawn in front of the most recent line added.

Figure 3a. Rasterscan with direction Down.

Figure 3b. Rasterscan with direction Right.

Figure 3c. Rasterscan with direction Up.

RasterImage

Figure 4. Raster scan data from beamformer

The RasterImage object can be used to create 2D displays that don't have time as an axis. The code will accept any data generated or stored as an array of arrays of data points. Such data typically come from beamforming instruments: radars, sonars and radio telescopes. The rasterscan generator can display these signals as a color coded, rectangular image of signal intensity representing the 2D area scanned by the instrument.

To create this type of display first create the data as an array of lines of data points where each data point represents the index into a color map. Then create the RasterImage object passing a reference to the data array along with the number of lines and the length of each line of data. The color map can be loaded as as one of the optional parameters. A default color map is provided, it has 256 color entries so data indices should be in the range 0..255 when using it. The RasterImage created is an HTML canvas. This can be imported directly into a web page DOM or used as input to a graphics library drawing method.

The code snippet to create Fig 4 is shown below.

// make a 2D data array with some Gaussian blobs
for (let r=0; r<rows; r++)
{
  sigData[r] = new Float32Array(buf, r*cols*4, cols);
}
// now add a few Gaussian blobs
addGaussian(sigData, rows, cols, 20, 30, 10);
addGaussian(sigData, rows, cols, 60, 70, 1);
addGaussian(sigData, rows, cols, 50, 55, 4);
addGaussian(sigData, rows, cols, 83, 12, 5);

// make a canvas object holding the color display of the data
const rasterImg = new RasterImage(sigData, cols, rows, {dir:"up"}); 

...  // set up Cango to display the RasterImage on screen

const dataPlot = new Img(rasterImg, {imgWidth:xmax-xmin, imgHeight:ymax-ymin, lorg:7});
// now draw the image
cgo9.render(dataPlot);
  

JavaScript Spectrogram code

The Spectrogram-2v00.js library supplies JavaScript constructors for Waterfall, Rasterscan and RasterImage objects to create spectrogram displays for display in web pages.

Both Waterfall and Rasterscan objects have an 'offscreenCvs' property which is a reference to the raw spectrogram they maintain as an offscreen canvas. Application code can use spectrogram.offscreenCvs as the source for the canvas.drawImageData method to render the spectrogram to an on-screen canvas element with 'on-the-fly' rescaling to an arbitrary size and aspect ratio.

The default color map used to convert the input signal is a 256 level map with colors similar to the popular Matlab 'Jet' colormap[2], but a user defined colormap can be supplied with as few as 2 or as many as 16 million colors using the 'setColorMap' method or passing the colorMap as the 'colorMap' property of the constructor's 'options' parameter.

The raw spectrogram may be built up at a fixed line rate along using the 'setLineRate', 'start' and 'stop' methods. Alternatively, the 'newline' method is provided so the spectrogram can be built up synchronously with the arrival of a new array of signal data.

Waterfall constructor

Syntax:

var wfall = new Waterfall(ipObj, ptsPerLine, lines, direction, options);

Description:

Creates a new Waterfall object which comprises an offscreen canvas property 'offscreenCvs' into which the contents of ipObj.buffer are written at a fixed rate. This offscreen canvas is available as a source for drawing an image of the waterfall into an on-screen canvas element.

Lines of data are added at a uniform rate set by the property 'lineRate'. Each new line is always written into the same line at the top, side or bottom of the display all the previous lines are copied as a block and moved one line in the direction set by the 'direction' property creating a waterfall effect. The number of data points plotted for each line is set by the 'pxPerLine' parameter and the total number of lines in the canvas is set by the 'lines' parameter. The line rate and other user parameters may be set as key-value pairs in the 'options' object. If not been set as an optional parameter or a call to setLineRate the default lineRate of 30 line/sec is used.

Parameters:

ipObj: Object with 'buffer' property - An object whose 'buffer' property is an array holding a line of data to be plotted. The buffer property is a reference to an Array so it can be changed to point at an array of newly collected data without having to copy it element by element. A double buffered input can easily be created, to supply the data (see example code below).

Each data element in the array is assumed to be an index into a color map. The default color map has 256 entries. If an optional colorMap is not provided then the data points should have values 0..255.

ptsPerLine: Number - The number of data points per line.

lines: Number - The number of lines of data forming the display.

direction: String - The direction of apparent flow of data as seen on the screen. Valid values are "UP", "DOWN", "LEFT", "RIGHT". The string may be upper, lower or mixed case.

options: Object - The various Waterfall object properties can be set by assigning the desired value to the corresponding options property. For details of the properties see Waterfall and Rasterscan constructor 'options' properties

Returns:

Waterfall Object:

This object's 'offScreenCvs' property holds the output canvas display (see Waterfall and Rasterscan common property).

This object has the methods described below (see Waterfall and Rasterscan common methods).

Example:

Here are some code snippets showing the basic use with the input buffer array used as a double buffer for efficiency.

function getDynamicDataBuffer(dataGen)   // returns a pointer to the dynamic data buffer
{
  // create double buffer for the data can be any type but values will be assigned to Uint8
  const bufferAry = [];
  ...
  function genDynamicData()
  {
    dataGen.getLine(bufferAry[1]);  // dataGen gathers the data
    // swap the buffers
    let tmpBuf = bufferAry[0];
    bufferAry[0] = bufferAry[1];
    bufferAry[1] = tmpBuf;
    // repeat forever
    if (playing) 
    {
      setTimeout(genDynamicData, linePeriod);
    } 
  }

  bufferAry[0] = new Float32Array(dataGen.fftLen);  // values should be 0..255
  bufferAry[1] = new Float32Array(dataGen.fftLen);  // ensure length is set before returning pointer
  // start the repeating loop
  genDynamicData();

  return {buffer: bufferAry[0]};  // value of the pointer will change
}

const dataObj = getDynamicDataBuffer(); 
const wf = new Waterfall(dataObj, wfLines, wfPts, "DOWN", {startbin: fftLen/2, lineRate: 30});

Rasterscan constructor

Syntax:

var raster = new Rasterscan(ipObj, width, height, direction, options);

Description:

Creates a new Rasterscan object which comprises an offscreen canvas with input signal data from the ipObj.buffer array drawn as a line of colored pixels. This offscreen canvas is available as a source for drawing an image of the waterfall into an on-screen canvas element.

Lines of data are added at a uniform rate set by the property 'lineRate'. Each line of data is drawn adjacent to the previous line so the new data progresses across the canvas in the direction specified by the 'direction' parameter. When the input reaches the last line of pixels set by the 'lines' parameter the input point jumps back the the first line. The input point is marked by a line of white pixels being drawn in front of the new data (erasing the oldest data line).

Parameters:

ipObj: Object with 'buffer' property - An object whose 'buffer' property is an array holding a line of data to be plotted. The buffer property is a reference to an Array so it can be changed to point at an array of newly collected data without having to copy it element by element.

Each data element in the array is assumed to be an index into a color map. The default color map has 256 entries. If an optional colorMap is not provided then the data points should have values 0..255.

ptsPerLine: Number - Number of data points per line.

lines: Number - The number of lines of data forming the display.

direction: String - The direction of apparent flow of data as seen on the screen. Valid values are "UP", "DOWN", "LEFT", "RIGHT". The string may be upper, lower or mixed case.

options: Object - Several Rasterscan properties can be set by assigning the desired value to the corresponding options property. For details of the properties see Waterfall and Rasterscan constructor 'options' properties

Returns:

Rasterscan Object:

This object's 'offScreenCvs' property holds the output canvas display (see Waterfall and Rasterscan common property).

This object has the methods described below (see Waterfall and Rasterscan common methods).

Waterfall and Rasterscan constructor 'options' properties

Key stringTypeValue range / Description
direction (or dir)String"up", "down", "left", or "right". This will set the direction of apparent data flow of the waterfall.
lineRateNumber
0..50 lines/sec
Sets the number of times per second that the contents of the ipObj.buffer are written to the raw waterfall canvas. If not been set as an optional parameter or a call to setLineRate the default lineRate of 30 line/sec is used.
startBinNumberThe offset into ipObj.buffer array. The will be the starting point for drawing the rest of the array onto the waterfall canvas.
colorMapArray of ArraysEach element of this array is an array of length 4. The 4 numbers in these arrays represent the R,G,B,A values of a color. The value of each input data point is used as an index into this array to select the color of the pixel to be written into the off screen canvas. The length of the colorMap array determine the number of different signal amplitudes the display can represent. A colorMap with 2 elements will resolve only 2 levels, a 256 element will resolve 256 the maximum length is 16,777,216 (all possible 24bit colors). Each of the R, G, B and A values must be integers in the range 0..255.
onscreenParentIdStringThe ID attribute of an on-screen element. If set, the raw Waterfall canvas will be drawn on-screen as a child of that element rather than being hidden off-screen.

Waterfall and Rasterscan objects' common property

Key stringTypeDescription
offScreenCvs HTML canvas

The off-screen canvas element which holds the 2D color display of the input data.

Application code should use this property as the source for drawing the waterfall or raster scan on-screen canvas using the canvas.drawImage method or a suitable graphics library call.

Waterfall and Rasterscan objects' common methods

newLine

Syntax:

obj.newLine();

Description:

Creates a line of colored pixels by using each element of the array ipObj.buffer array as an index into the color map then writes the resulting line of RGBA color values into in the off-screen canvas. If the lineRate has been set to a value greater than zero then newLine is called at regular intervals to meet this rate independent of the rate at which new data is available in the ipObj.buffer. If lineRate has been set to 0 the application code must call the newLine method to add each line to the Waterfall or Rasterscan.

Parameters:

none.

start

Syntax:

obj.start();

Description:

If lineRate has been set to a non-zero value obj.start() will start the cycle of calling the newLine method at the lineRate value. If lineRate is 0 the call has no effect.

Parameters:

none.

stop

Syntax:

obj.stop();

Description:

Stops the update cycle that adds new data lines to the display. The Waterfall or Rasterscan will appear frozen in the state when obj.stop() was called.

Parameters:

none.

clear

Syntax:

obj.clear();

Description:

Clears the display by setting all the off-screen canvas to pixels to colorMap[0] color.

Parameters:

none.

setLineRate

Syntax:

obj.setLineRate(linesPerSec);

Description:

Sets the rate at which lines are added to the display. The current lineRate property is set to the 'linesPerSec' parameter. The data in the input buffer [0] element is converted to color and added to the raw spectrogram every 1/lineRate seconds.

Parameters:

linesPerSec: Number - The number of lines added to the display every second. If the value is greater than 0 and less then 50 the newLine method will be automatically called at intervals to achieve this line rate starting as from the time the 'start' method is called. If lineRate is set to 0 then the application code must call newLine method to advance the Waterfall or Rasterscan.

Canvas direct pixel writing

The HTML canvas element has an ImageData property which allows direct read and write access to the pixel data.

ImageData object

Syntax:

cvs = document.getElementByID(canvasID);

imgData = cvs.getImageData();

Properties:

width: Number (read only)- The width of the image in pixels.

height: Number (read only) - The height of the image in pixels.

data: Uint8ClampedArray (read only) - An array containing the raw pixel color data. Each pixel is represented by four one-byte values in RGBA format. Each color component is represented by an integer between 0 and 255. Each component is assigned a consecutive index within the array, with the top left pixel's red component being at index 0 within the array. Pixels then proceed from left to right, then downward, across each row of pixels in the canvas.

The ImageData.data array holds a repeating sequence of RGBA values where each element refers to an individual color component. That repeating sequence is as follows:

imgData.data[0] = pixel: row 0, column 0 red component value
imgData.data[1] = pixel: row 0, column 0 green component value
imgData.data[2] = pixel: row 0, column 0 blue component value
imgData.data[3] = pixel: row 0, column 0 alpha component value

imgData.data[4] = pixel: row 0, column 1 red component value
imgData.data[5] = pixel: row 0, column 1 green component value
imgData.data[6] = pixel: row 0, column 1 blue component value
imgData.data[7] = pixel: row 0, column 1 alpha component value

imgData.data[8] = pixel: row 0, column 2 red component value
imgData.data[9] = pixel: row 0, column 2 green component value
imgData.data[10] = pixel: row 0, column 2 blue component value
imgData.data[11] = pixel: row 0, column 2 alpha component value
...

The imgData.data array values are only a copy of the canvas pixel data, the copy can be manipulated ie. color of pixels changed, but to have the canvas show this changed data the imgData object must be written back into the canvas using the canvas.putImageData method.

cvs.putImageData(imgData, 0, 0);

Any rectangular area within the canvas bounds can have its pixel data read and similarly any rectangular area can have its pixel data overwritten. It is not necessary to retrieve or write the entire canvas image data

Note: Before we can plot a pixel, we must translate the pixel x, y coordinates into an index representing the offset of the red component in the imageData.data array.

var index = (y * canvasWidth + x) * 4;

Using a Color Map

To set the color of a pixel to indicate a signal level we must map the signal value to a color, this color will be represented by 4 color component vales red, green, blue and alpha (transparency) values. If the signal can be well described by say 20 levels then then each level can be assigned its own color, by an r,g,b and an a value. These RGBA values can be stored in a 4 element array and then 20 of these 4 element arrays makes up a colormap. The example below assumes that each color is completely opaque so the alpha value for each color is set to 255.

  var colorMap = [[level0.r, level0.g, level0.b, 255],   // color for level 0
                  [level1.r, level1.g, level1.b, 255],   // color for level 1
                  ... 
                  [level19.r, level19.g, level19.b, 255]];  // color for level 19

To set the color of a pixel x, y to correspond to a signal level of say 's' (a value between 0 and 19) the code would be:

  var i = = (y * canvasWidth + x) * 4;
  imgData.data[i] = colMap[s][0];    // red component
  imgData.data[i+1] = colMap[s][1];  // green component
  imgData.data[i+2] = colMap[s][2];  // blue component
  imgData.data[i+3] = colMap[s][3];  // alpha  = 255

  // modified imgData is then written back to the canvas context using putImageData(imgData, 0, 0)

Jet Colormap

A very popular colormap often used in signal processing applications is the Matlab Jet colormap which maps low level signal levels to green-blue colors and increasingly high signal levels to green, yellow and through orange to red at the highest level. The formula for generating the colormap are described here. So a color map with any number of input signal levels can be created.

The example that follow will use a 256 level Jet color map which has been saved as a JSON file JetColormap256.JSON

Canvas ImageData example

As an example of using these directly draw methods Fig 1 shows a sine wave signal with its amplitude mapped to colors and these RGBA values written directly into a canvas image data. The full source code is shown on the left of Fig 1.

function testImageData(cvsID) {
  const canvas = document.getElementById(cvsID),
        ctx = canvas.getContext("2d"),
        imgObj = ctx.getImageData(0,0, canvas.width, canvas.height),
        pxPerLine = imgObj.width,
        colMap = [[  0,  0,128,255], [  0,  0,191,255], 
                  [  0,  0,255,255], [  0, 64,255,255], 
                  [  0,128,255,255], [  0,191,255,255], 
                  [  0,255,255,255], [ 64,255,191,255], 
                  [128,255,128,255], [191,255, 64,255], 
                  [255,255,  0,255], [255,191,  0,255], 
                  [255,128,  0,255], [255, 64,  0,255], 
                  [255,  0,  0,255], [191,  0,  0,255], 
                  [168,  0,  0,255]],
        colorLevels = 16,
        dataLine = [];

  // generate a line of signal data
  for (let i=0; i < pxPerLine; i++) {
    dataLine[i] = Math.floor(Math.sin(i*Math.PI/100)**2 * colorLevels);
  }
  for (let row=0; row < 30; row++) {
    for (let px = 0; px < pxPerLine; px++) {
      let i = 4*(row * pxPerLine + px);
      let rgba = colMap[dataLine[px]];  // lookup color rgba values
      
      imgObj.data[i] = rgba[0];   // red
      imgObj.data[i+1] = rgba[1]; // green
      imgObj.data[i+2] = rgba[2]; // blue
      imgObj.data[i+3] = rgba[3]; // alpha
    }
  }
  ctx.putImageData(imgObj, 0, 0);
}