Introduction to Physical Computing

Week 1

What is Interactivity to Me?

An exchange and or act of consumption that involves two or more actors/things.  The exchange may be a request or an input.  The consumption an output or response to the preliminary action.  Things like data, food, music are exchanged and consumed.  A person interacts with food by selecting to eat something.  It is tasted, digested and it harms and or benefits the consumer.  The interactivity is between the eater and the food.  Business, human relationships and art/science are more complex systems of interactivity.  Moments are interactive.  We can be aware of our present surroundings and have a simple or elaborate interaction with our breath, ambient sounds, the pen as it leaves ink on the paper, the thoughts processed, continued , rejected as occurring in automated articulations.  The Interactivity of editing my written word.  

I am inspired by the interactivity of playing piano. I press my fingers to the keyboard in patterns that I may know or have yet to discover or learn.  The sound yielded through this exchange of kinetic energy resonates in the air.  Vibrations in the air originate from a hammer striking a string corresponding with the key I played.  The sound further sustained in the moment with my right foot on a pedal that elongates the time the string remains vibrating after I have pushed the key.  The response to the sound that my ears receive guides me to the process of sculpting music.  An interactivity occurs between my emotions, thoughts(or absence of) choice of the key played, and proceeding patterns or improvisations of sound.  I elaborate on this interactivity with composing music and visuals.  

WEEK 3 

OBSERVATION 

NYC SUBWAY TURNSTILES

The way this device works is that it scans your metro card in the reader slot and if you have enough value or time you may pass through the metal bar.  From this picture you can see the sign 'Entry' and an arrow pointing to the screen.  This screen informs the user of their metro card balance or expiration date.  If the card is valid then the green lights brighten and the lock to the bar is removed so it can cycle through allowing one individual the passage to wait for a ride on the train platform.    

I have noticed some people pull on the turnstile half way towards them and they squeeze in between and pass through without paying.  I have seen some people jump over using their hands to support them as they lift over the bar.  The most typical adverse reaction to the card reader is when someone scans their card and the screen says: "PLEASE SWIPE AGAIN."  Their reactions are always a bit alarmed or surprised as they expect to pass through but the card did not read correctly so their momentum is halted.  Some people scan their cards more cautiously and slowly.  Others breeze quickly through the turnstiles.   

It takes the least amount of time if you scan your card correctly and it has a sufficient value for the fare, approximately 3 seconds to walk through the interaction of the turnstile with fare paid and ready to board the next train.  If there are problems with the swipe, the reader or your card, then the process could take longer.  Swipe again is usually a brief interruption and takes two or three swipes prior to passage.  If there is a more serious issue like a defect with your card, you need to see the MTA Personnel at the booth.  There they may help you get a new card or tell you to use another turnstile.    

Lab: electronics

WEEK 4 

ARDUINO: Digital and analog

Musical Instrument Prototyping 

Instrument with 2 sensors

midterm project

Drum Box

-with jesse simpson & ahmad arshad

We designed and built a drum box.  It is made up of Piezo vibration sensors, 1M Ohm resistors, breadboard, wires to 4 analog inputs into Arduino.  We laser cut acrylic to enclose the sensors, breadboard, resistors, wires and Arduino.  We drilled a hole in the side of the acrylic box for the usb AB cable from the Arduino to the laptop.  

The p5 Sketch responds to serial communication from the Arduino's analog inputs and launches sound samples from a drum kick, snare, hi-hat and clap.  

Arduino code

p5 sketch.js

let serial; // variable to hold an instance of the serialport library
let portName = '/dev/cu.usbmodem1421';
let inData; // for incoming serial data
let sensor1;
let sensor2;
let sensor3;
let sensor4;
let clap;
let hat;
let snare;
let kicksub;
let xPos = 0; // x position of the graph

function preload() {
    soundFormats('wav');
    clap = loadSound('CLAP.wav');
    hat = loadSound('HAT.wav');
    snare = loadSound('SNARE.wav');
    kicksub = loadSound('KICKSUB.wav');
}

function setup() {
    createCanvas(800, 800);
    background(255, 255, 255);
    serial = new p5.SerialPort(); // make a new instance of the serialport library
    serial.on('list', printList); // set a callback function for the serialport list event
    serial.on('connected', serverConnected); // callback for connecting to the server
    serial.on('open', portOpen); // callback for the port opening
    serial.on('data', serialEvent); // callback for when new data arrives
    serial.on('error', serialError); // callback for errors
    serial.on('close', portClose); // callback for the port closing

    serial.list(); // list the serial ports
    serial.open(portName); // open a serial port
    // serial.list(); // list the serial ports
}

function draw() {
    graphData(sensor1, 255, 0, 0);
    graphData(sensor2, 255, 255, 0);
    graphData(sensor3, 10, 129, 25);
    graphData(sensor4, 10, 24, 129);

    //background(0);
    //fill(255);
    //text("sensor value: " + inData, 30, 30);
}

// get the list of ports:
function printList(portList) {
    // portList is an array of serial port names
    for (var i = 0; i < portList.length; i++) {
        // Display the list the console:
        console.log(i + " " + portList[i]);
    }
}

function serverConnected() {
    console.log('connected to server.');
}

function portOpen() {
    console.log('the serial port opened.')
}

function serialEvent() {
    // read a string from the serial port:
    var inString = serial.readStringUntil('\r\n');
    // check to see that there's actually a string there:
    if (inString.length > 0) {
        var sensors = split(inString, ',');
        sensor1 = sensors[1];
        sensor2 = sensors[0];
        sensor3 = sensors[2];
        sensor4 = sensors[3];

        if (sensor1 >= 25) {
            var m = map(sensor1, 25, 1023, 0.1, 1.0);

            if (!clap.isPlaying()) {
                clap.setVolume(m);
                clap.play();
                console.log("playing clap at volume : " + m);

            }
        }
        if (sensor2 >= 25) {
            var m = map(sensor2, 25, 1023, 0.1, 1.0);
            //            console.log(m);

            if (!hat.isPlaying()) {
                hat.setVolume(m);
                hat.play();
                console.log("playing hat at volume : " + m);

            }
        }
        if (sensor3 >= 25) {
            var m = map(sensor3, 25, 1023, 0.1, 1.0);
            //            console.log(m);

            if (!snare.isPlaying()) {
                snare.setVolume(m);
                snare.play();
                console.log("playing snare at volume : " + m);

            }
        }
        if (sensor4 >= 25) {
            var m = map(sensor4, 25, 1023, 0.1, 1.0);
            //            console.log(m);

            if (!kicksub.isPlaying()) {
                kicksub.setVolume(m);
                kicksub.play();
                console.log("playing kicksub at volume : " + m);

            }
        }
    }
}

function serialError(err) {
    console.log('Something went wrong with the serial port. ' + err);
}

function portClose() {
    console.log('The serial port closed.');
}

function graphData(newData, r, g, b) {
    // map the range of the input to the window height:
    let yPos = map(newData, 0, 255, 0, height);
    // draw the line in a pretty color:
    stroke(r, g, b);
    line(xPos, height, xPos, height - yPos);
    // at the edge of the screen, go back to the beginning:
    if (xPos >= width) {
        xPos = 0;
        // clear the screen by resetting the background:
        //background(0x08, 0x16, 0x40);
    } else {
        // increment the horizontal position for the next reading:
        xPos++;
    }
}
 

We had a few people play our drum box and they enjoyed the experience.  One person said, "why do the shapes look like iPhones?"  It was valuable to get the feedback from user testing and will consider the shapes of the drum pads in future iterations.   

Final Project

music-visual Scrolls

An Audio Visual Controller that plays my music and video/animation work as a response to my touch gesture using a paintbrush.

Scrolls suspended in the air that use scrim or muslin fabric instead of paper papyrus.  The scrolls are pressed to initialize music samples of a song composition and notes in progressions on a synthesized sound.  The initialization gesture of pressing on the fabric launches a sound and simultaneously projects visuals on the scroll fabric.  

The concept is inspired by songwriting taking shape.  The Interface shape visually resembles the head of a violin, (a wooden scroll) with the fabric making up the body (instead of strings).

I am inspired to perform music and visuals I have composed/created.  As a performer, I would like to have a musical role as a conductor in an orchestra, directing the symphony.  I have composed music with sounds from most instruments (i.e. piano, guitar, bass, synths, strings, drums/percussion) and am unable to play all instruments simultaneously for a performance (one-man band).  This music visual scroll enables me to conduct my music composition and provides a canvas for my visual artwork.   

• It uses a flex sensor at the top of the scroll where the fabric is attached.  Bending the sensor sends serial values to an arduino Analog Input.  The values are read in MAX/MSP/Jitter to initialize the sound and visuals.  The visuals are projected on the fabric.

• I will need: Arduino, Flex Sensors, Projector(s), wires, bigger breadboard, resistors, aluminum or PVC pipe (scroll piece), muslin or scrim fabric (scroll paper), Auto-pole(s) to hang the scrolls.  

final project plan

timeline and bill of materials

Week 10

Connectors for Stranded Wire, Scrolls Fabrication, Soldering

Beta Version - Playtest

Week 11 

Improve Visuals feedback response with MaxMSP/Jitter

Additional sensors to system and Arduino code

Cut and Finalize additional Fabric Sheets  

Playtest 

Week 12 

Compose Sounds, Projection Mapping

Rehearse  

Week 13

Compose Music, Choreograph Performance, Rehearse

Week 14 

Presentation 

BOM

Laptop - Arduino, MAX/MSP/Jitter

Breadboard, stranded wires, flex sensors, resistors

(Translucent) Fabric 

Projector, Tripod

PVC/Aluminum Pipes for Scroll Piece

AutoPoles 

Enclosure for Arduino + Breadboard 

system diagrams

Play testing day

Some feedback from user tests...

"radiate from a point" "brush expressivity"  "scrolls move when you push them, make more stable" "what am I creating?  I want to write my name" "particles fade?"  "difficult to know whats going on"  "make it more clear"  "correlation with sound and visuals"  "make the scrolls higher, bigger"  "simplify" "make it more clear, not subtle"  "from nothing to something"  "make it more stupid"  "pattern and sound synchronize" "fast/slow motion = sound"

painted the scrolls &

cut the fabric

FINAL DOCUMENTATION

MUSICVISUAL SCROLLS