STSS Data Visualisation

In this lab, we will visualising a sound file using Processing. Processing is built on top of the Java programming language, so it will use many of the concepts you are familiar with and uses the same syntax. It is an open source environment for people who want to program images, animation and sound.

Part 1 - Getting started with Processing

Open Processing by typing Processing in the search bar and clicking "Run Processing".

Later on we will need a library called Minim. Download Minim for Processing by going to: Sketch > Import Library > Add Library > search for 'Minim' and click install. This may take a minute or two - when the installation is complete a green tick will appear next to Minim.

We are going to be creating visuals on the screen. Essentially we are treating the screen like a fancy piece of graph paper. Each pixel is at a coordinate (x (horizontal), y (vertical)). Our task is to determine what colours and shapes should appear at these coordinates.

EX1: Type the following code into Processing and press the Run button (the arrow at the top left of the window):

line(1,0,4,5);

Did you get what you expected? The coordinate system for pixels in a computer window is reversed along the y axis. (0,0) is in the top left.

Simple shapes in Processing

point(x, y) - draws a point at coordinate x,y
line(x1, y1, x2, y2) - draws a line from x1, y1 to x2, y2
rect(x, y, w, h) - draws a rectangle at x, y (coordinate for the top left corner of the rectangle) with width w and height h
ellipse(x, y, w, h) - draws an ellipse at x, y (coordinate for the center of the elipse) with width w and height h
triangle(x1, y1, x2, y2, x3, y3)

EX2: Add the statements below into your code, run the program, and see what happens. Try modifying a few of the arguments and see what effect that has

line(1,0,4,5);
point(0,2);
point(0,4);
rect(5,0,4,3);
ellipse(3,7,4,4);

Colour in Processing

In Processing every shape has a stroke() or a fill() or both. The stroke() is the colour of the shape's outline and fill() is the colour of the interior. If you forget to specify a colour Processing will use black (0) for the stroke() and white (255) for the fill(). By adding stroke() and fill() functions before the shape is drawn you can set the colour. stroke() or fill() can be eliminated with the noStroke() or noFill()

There is also a background() function which sets the background colour for the window.

Digital colours are constucted by mixing R (red), G (green, and B (blue). The amount of each colour is expressed as a range 0 (none of that colour) - 255 (as much as possible). Processing has a colour selector to aid in choosing colours. Access this via Tools > Color Selector (from the menu bar).

In addition to the red, green, and blue components of each colour, there is an optional fourth component, referred to as the colour's "alpha" (a) - alpha refers to the colours opacity and is useful when you want things to appear partially see-through. Opacity is also expressed in the range 0 (completely transparent) - 255 (completely opaque).

background(r, g, b)
stroke(r, g, b, a)
fill(r, g, b, a)
strokeWeight(weight)
noStroke()
noFill()

background(255);
noStroke();

fill(255, 0, 0);
ellipse(20, 20, 16, 16);

fill(0, 255, 0, 255);
ellipse(40, 20, 16, 16);

fill(0, 0, 255, 127);
ellipse(60, 20, 16, 16);

When you draw a shape, Processing will use the most recently specified stroke() and fill(), reading the code from top to bottom.

EX3: Type the statements above and run the program

You can also specify colours in colorMode(HSB)which takes three arguments (hue, saturation, brightness).

Hue—The colour type, ranges from 0 to 255 by default.
Saturation—The vibrancy of the colour, 0 to 255 by default.
Brightness—The, well, brightness of the colour, 0 to 255 by default

The colour selector also gives you the H, S, B values for a chosen colour

EX4: Modify your code to use colorMode(HSB), for example:

colorMode(HSB);
background(255);
noStroke();

fill(0, 255, 255);
ellipse(20, 20, 16, 16);

fill(119, 255, 255);
ellipse(40, 20, 16, 16);

fill(239, 255, 255);
ellipse(60, 20, 16, 16);

Three very important functions

So far we have been working with Processing in static mode - but we want our sketches to change over time (dynamic mode). To do this we need to use three important functions:

setup() - we do this first and only once
draw() – we do this again and again until the program quits. We want our sketches to change over time and so the most of our code goes here.
size() -specifies the size of the window you want to create. size() takes two arguments - width and height. The size() function should always be the first line of code in setup()

void setup(){
   size(100, 100);
}

void draw(){

}

Things to note

You can write comments in your code (and you should do this to remind you of what the code means). Single line comments are created with two forward slashes //.
Some of the time pressing the run button will not run the sketch because there are mistakes in the code (likely a typo). Some errors are noted in the code by a red squiggly line where Processing believes the mistake to be. An error message will also appear in the grey bar at the bottom of the input window.
All of these functions are part of the Processing library. All of the available functions can be found in the reference for Processing - which can be found here.

Now we know the basics, we're going to look at how processing deals with variables, for loops and conditionals.

Variables

We use variables to store data. There are different types of variables for example strings, integers and floating point numbers. Once we have stored some data in a variable, we can use the value of that variable later on in the program.

Processing has a number of built-in variables which might be useful, these include:

width - width (in pixels) of sketch window
height - width (in pixels) of sketch window

The following code illustrates how variables are declared and initialised in Processing. It should look pretty familiar.

EX5: Copy and paste the code below into Processing and press Run.

//declare and initialise an integer variable
int circleSize = 4;

//declare and initialise a floating point variable
float circleX = width/2;
float circleY = height/2;

void setup(){
   size(100, 100);
}

void draw(){
   ellipse(circleX, circleY, circleSize, circleSize);
   
   //add one to circleSize each time we process a frame
   circleSize = circleSize + 1;
}

For Loops

Loops are a convenient way for a program to repeat a set of steps over and opver again.

The following example illustrates the use of for loops in Processing.

EX6: Copy and paste the code below into Processing and press Run

void setup(){
   size(100, 100);
}

void draw(){
   for (int i=0; i<10; i++){
      noStroke();
      fill(i*25);
      rect(i*10, 0, 10, height);
   }
}

If-statements

Sometimes you may want to do different things depending on what the value of some variable is. For this, you can use if-statements.

The following example illustrates the use of if statements in Processing.

EX7: Copy and paste the code below into Processing and press Run

int x = 0;
int speed = 1;

void setup(){
   size(200, 200);
}

void draw(){
   background(255);

   //add current speed to x each time we process a frame
   x = x + speed; 

   //if x is at an edge
   if (x>width || x<0){
      //reverse the direction of speed
      speed = speed * -1;
   }

   //draw a circle at x
   stroke(0);
   fill(255, 0, 0);
   ellipse(x, 100, 32, 32);
}

Part 2 - Visualise a sound file

We now know everything we need to know about Processing to visualise a sound. There are no speakers in the lab machines so you'll need to plug in your headphones.

EX8 We have introduced a numbers of shapes, colours and techniques in Processing. Your task is to pull all of this together to visualise the sound file.

Download the skeleton code. The code does 4 things:

Imports a library called Minim - we use this to integrate audio into the sketch.
Uploads an audio file from the data folder - we are using a sound file from a building site in appreciation of the soundscape we have been listening to all week - you might like to use a different sound or a music track.
Performs an FFT (Fast Fourier Transform) on the file - this decomposes the signal into the frequencies that make it up - in this case sorting them into 64 bands.
Uses the value in the different bands at each clock tick to draw lines of different lengths to the screen.

Instead of drawing lines to the screen, what else could you use the data to do? Now is your chance to get creative.

You can also look at other people's sketches for inspiration. Open Processing is community of people using Processing to do some great algorithmic design. Clicking on the </>; at the top in the middle will reveal the code behind the sketch - but be warned - some of these might look pretty scary, you have already learned everything you need to know to create an amazing sketch.

An Extension

We can use a clever trick to fake a 3D effect. The trick captures an image at the end of a frame, adds some transparency and then pastes the image to the background at 99% of its original size before drawing a new frame.

Processing has a bunch of classes that are really useful - one of which is called PImage - this is a class for loading and displaying an image. Look up PImage in the processing reference and see if you can work out how to fake the 3D effect. You may find the following methods useful:

get(x, y, w, h) - grabs a section of the image starting at x, y with width w and height h
image(img, a, b, c, d) - displays img starting at a, b with width c and height d
To apply transparency to an image without affecting its color, we use white as the tint color and specify an alpha value. For instance, tint(255, 128) will make an image 50% transparent (assuming the default alpha range of 0-255).
noTint(); Disables tint