Final Project: The Water Synth

Featured Posts, Physical Computing

Description:
A waterfall that functions as a musical instrument. When the user passes their hand under the waterfall, notes will play. The note will be sustained as long as they keep their hand in the same position. It will be possible to play chords or intervals (multiple notes at once) using both hands.

Prototype:
Our first prototype was as low-fi as possible–we created a waterfall by pouring a bucket of water into a tupperware container with a slit in it. As users put their hand under the waterfall, Jarone played notes on a scale from his iPad.

[prototype-1.mp4]

The main questions we wanted to answer in the first round of user testing were, Is this fun? and, Is the system intuitive? The answer to our first question was that people found it fun and entertaining (though there was one user tester that was displeased about getting wet). As for the second question, when we didn’t explain that it was a musical instrument, there were a few people who just sat and stared at the waterfall without putting their hands in it. This can be solved in a number of ways (e.g., a title card with instructions, or using a peg that interrupts the stream so that a note is being played in the “starting position”, or having one of us fool around with it so the audience can see how it works, etc.)

We got really nice feedback over all, with users wanting us to experiment with sensors on the y-axis, include LEDs to light the waterfall, and create individual streams so it’s more obvious where the notes occur.

Laser Testing:
After watching some YouTube videos, Jarone and I were fairly certain we could fire lasers directly through the water stream to get a nice hot spot in the water basin below (the water refracts the light of the laser along its path). However, we didn’t realize that your average laser isn’t powerful enough to travel more than about 6 inches. Since we want our waterfall to be at least 2 feet tall, this presented a problem:

[laser-test.m4v]

After testing, we realized that we aren’t going to be able to send the laser through the water’s path, so our plan is to plant the laser directly behind it.

System diagram:

       

Midi Communication:
We hooked up a midi jack and a photo sensor to an Arduino for a dry test:

[photocell-test.mp4]

We’ve succeeded in getting notes to play when the photocell is covered, but we’re still figuring out how to get one sustained note. Our next steps are to fix the code and do a wet test with a laser, water basin, and photo cell below it.

Beginning the waterfall:
We got a 400 gallon-per-hour pump that was too powerful for our initial set up, so the waterfall was shooting toward the wall instead of downward. Then when we added a valve to regulate the flow, it completely killed the water pressure so rather than a sheet we got a sad little stream.

Benedetta thinks the valve was probably rated at a much lower water pressure than the pump allows for. For the time being we’ll use individual streams for each note because producing a waterfall is harder than we expected. At this point, separate streams also makes more conceptual sense because it shows the user where each note occurs. A sheet of water gives the impression of a full range of notes you can slide between, instead of integer values that occur at arbitrary points.

Concurrency Test:
We were able to not just get one sustained note, but multiple notes at the same time! Jarone says the next thing we should do is put all the notes into an array to clean up the code.

[concurrency-test.mp4]

The Second Prototype:
When putting together our second prototype, we struggled to find the right sensors. We first tried photocells, but their range of sensitivity was too small. Then we tried both infrared and ultrasonic distance sensors. We ended up using one of each in this second prototype (the IR sensor is the finicky one on the left):

[Water-Synth-Second-Prototype.mp4]

The IR sensor was constantly giving out garbage readings, possibly because it was reflecting off the water in weird ways. The behavior of the ultrasonic sensor didn’t seem to be affected by the water. We decided that the ultrasonic sensor was the better, more reliable option.

Getting Ready for Final Class:
No luck pushing the notes into an array, but at least our wires are labeled.

labelled-wires
The third prototype:
For our final physical computing class, we really wanted to get four sensors working. Unfortunately the delays that we used to send and receive ultrasonic pings clogged up the entire system. The current state of our code only allows for two sensors. However, we made progress in terms of getting one sensor to play multiple notes, as seen in this video:

[water-synth-3-notes.mp4]

We’re still not sure the best way to “zhush” it up, other than by painting the pvc pipe. We still need to figure out the kind of enclosure we want to use for our circuit, and bring some dignity to the sad plastic basin. We also need to get a working system without using four different arduinos. But the prototype works!

[water-synth-updated.mp4]

The Enclosure:
We decided to make a wooden enclosure for the PVC pipe and water basin, and seal it with shellac in order to make it water resistant. Our friend Chester was extremely kind and made a model for us, so all Jarone and I had to do was figure out the measurements and assemble it.
  

Winter Show Highlights:
Jarone and I were fortunate enough to be selected for the ITP’s 2016 Winter Show! We got really amazing feedback. Even though many people needed coaxing to actually roll their sleeves up and get wet, they really enjoyed it once they did. The water synth was a hit with basically every demographic, from toddlers to senior citizens. We had a few professional musicians stop by and they had fun guessing what scale we used and creating quick compositions.

Source Code:

Algorithm Update

Thesis

Challenge: How do I adapt Dan Shiffman’s distance algorithm so that the starting point on the path is always “the hive”?

For the sake of both clarity and accuracy, I decided that all students should begin their paths at the same point–the point where hive is located. This meant that I had to adjust the distance algorithm, because Dan Shiffman had designed it so that any point could be used for the starting position. After a bit of fumbling about in the JavaScript file, I remembered how the algorithm works! This is the sequence in which the path order gets tested (assuming a path array with a length of eight):

[0,1,2,3,4,5,6,7]
[0,1,2,3,4,5,7,6]
[0,1,2,3,4,7,5,6]
[0,1,2,3,4,7,6,5]
[0,1,2,3,7,4,6,5]
[0,1,2,3,7,6,4,5]
[0,1,2,3,7,6,5,4] etc.

What this means is that all the possibilities for any “starting position” (the point at index 0) are tested before the starting position is swapped. All I had to do was stop looping as soon as the first element in my array was no longer was equal to zero:

if (order[0] != 0) {
___noLoop();
}

Once I figured that out, I was able to log the shortest and longest paths, seen below. I want the students to not only have access to the “optimal” path, but also be able to compare it to the “least optimal” path.

   

Serial Communication from Arduino to p5

Thesis

Finally, it was time to get my Arduino to talk to my p5 sketch. The last time I tried this I made a big mistake with my setup so the Arduino was constantly sending serial messages and overloading the serial port buffer. This created a huge lag on the p5 end. I’ve learned from my mistakes and now am checking for state change, so the only time a message will be sent is when the user is pressing a button. When a button is pressed, the Arduino will print out a string with the corresponding flower’s color.

In order to receive messages, I’m using Shawn Van Every’s p5.serialport library. In the serialEvent() function I’m using the serial.readLine() method. I then loop through my flower objects to check for the matching color, so I can draw a path to the correct flower.

As you can see in the video below, everything seems to be working! Now I need to make it work with seven giant arcade buttons, instead of three tiny push buttons.

 

p5js Prototype and Algorithm

Thesis

Before I began prototyping in p5, I was lucky to be able to follow a few of Dan Shiffman’s Coding Train tutorials on the traveling salesman problem. I’ll be using Dan’s distance algorithm for my installation. I got Dan’s code working, which you can see in the image below. 512 is the path distance measured in pixels. The purple path on top is the one that has the shortest total distance, and the white path below is the current path being tested. It’s a brute-force algorithm, so every single possible path is tested in order to find the optimal route.

In office hours I asked him if it made sense to run the algorithm concurrently with my flower visualization program, and he suggested that I hard code the flower positions and store the optimal distance beforehand to make my life easier.

Now that I didn’t have to worry about the algorithm, I was able to start the fun part: designing a p5 prototype that a user could interact with! My goal is to get data on the flower path from the arcade buttons, but in the meantime I wrote a function to test for mouse clicks on the flowers (represented by colorful dots).

Here is my wishlist for my javascript program:

  • User inputs for name (aka worker bee name), class (aka your hive), and attempt number
  • Save path order and distance to a JSON file
  • Compare Button to display all path attempts side by side
  • Restart Pollination Training Button

Push Button Test

Thesis

Before I start setting up my arcade buttons, I need to make sure that I can reliably send messages with basic push buttons on a breadboard. For testing purposes, I’m just using three buttons instead of seven—a red, green, and blue button to represent red, green, and blue flowers.

I needed to detect state change in any of the three buttons. This is crucial, because I only want to send a message when a button is moving from LOW to HIGH—otherwise I’ll clog up the serial port with unnecessary information.

In Javascript I would create a “button object” or “flower object” but I don’t think such a thing exists in C. So instead I made a series of corresponding arrays: one for pin numbers, one for the current states, one for the previous states, and one for the message to be sent (the color of the flowers). Then I used two for loops: one to declare each button pin as an input, and one to read the pins and test for state change.

The code totally works! On to the p5 side of things.

User Testing: Round 1

Thesis

Today I met with Greg Trefry and his kids (Miles and Wyatt) to user test a low-tech version of my thesis. Instead of flowers and arcade buttons, we used colorful sticks. Instead of p5 and Javascript, we mapped out the position of the flowers with magic markers and I drew out everyone’s paths by hand.

I began by saying: “Hello worker bees. Welcome to pollination training. My name is Melissa and I am the Queen Bee. Every training session we start by buzzing, so on the count of three, let’s buzz together.” I then led the four of us in a communal buzz. Miles is in 1st grade, and was very much down to buzz. Wyatt is in 3rd grade and is already too cool to buzz.

I asked them some questions, like, why do bees collect pollen? How do bees collect pollen? I was surprised that Miles already knew about the pollen baskets that some bees have on their legs.

Then I got to the heart of the matter—the order that we will choose to pollinate the flowers. I started by drawing three flowers on a whiteboard and asking them to figure out how many different paths there were. After I modeled a couple of paths, they were able to figure out that there were 6 different orders, or permutations. They were pretty engaged when using whiteboard markers to show me the different paths.

I asked them if bees would want to take the path with the longest distance or shortest distance. Again, Miles surprised me with a thoughtful answer: “They would want to take the shortest distance so they could have more energy to make honey next time.” This was a huge success—with very little prompting on my part, Miles produced a damn good definition of optimal foraging theory.

It was clear that they had the most fun when they were running around, pollinating the “flowers”, which they did by picking up the sticks (our flower stand-ins) and rubbing them on their legs (to get the pollen in the pollen baskets). Wyatt said that he wanted to take the path with the longest total distance, and Miles wanted to find a “medium” distance. I believe Wyatt was trying to be contrary, and Miles wanted to have a unique task. I’m perfectly happy with the kids experimenting with the kinds of paths they choose, but it was clear that Miles and Wyatt weren’t strategizing while they were pollinating. They just wanted to run as fast as possible.

Takeaways:

  • Greg suggested that instead of modeling paths and permutations on a whiteboard, the kids should figure it out themselves by pollinating a smaller number of flowers as many different ways as they can in a demo round
  • I still don’t know how to get the kids to realize on their own, in their own language, that the more flowers we need to pollinate, the number of permutations increases exponentially.
  • I didn’t get a chance to have Miles and Wyatt compare paths and make predictions, which is an important part of the interaction. This is mainly because after the “fun stuff” was over, it was basically impossible to get them to return to the “math stuff”
  • They wanted to do the “pollinating” again and again. This was really because of the “racing” element and not the “paths” element, but repeat interaction is still a win!

Flower Fabrication Part II

Thesis

Figuring out how to embed the arcade button into the center of the flower posed difficulties. The arcade button has a threaded plastic casing that allows you to secure it to a flat surface, like an arcade cabinet. I wanted to attach the button to the stem in a way that would be stable but not necessarily permanent (in case I would like to repurpose the buttons in the future). This was the best I could come up with:

I bent the armature wire to create a cradle for the arcade button, while still leaving the metal lugs exposed so I could add wiring after I’d finished constructing the flower. Then I wrapped the whole situation in gaffers tape. It’s not attractive, but it’ll work for now.

The next step was to add all the petals. Because I had issues with the floral tape in my 3-petal iteration I decided to mostly use gaffers tape to hold the petals in place.

   

I went to visit Ben Light to get some advice on the best way to fabricate the other flowers. He had two main concerns:

  1. Armature wire is too flexible and not strong enough for stem material. He suggested getting something with the same materiality as flexible tent poles.

2. In order for the flowers to stay upright, I need a base with a larger footprint (and the basic clay flower pot that I bought isn’t going to cut it).

Lindsey suggested using PVC pipe for the stem, and planting it in a super huge flower pot, then potentially weighing down the flower pot with a layer of concrete. Because I don’t have the time or the money to purchase 7 huge flower pots and fill them with concrete, for my proof-of-concept installation I’ve decided that the flowers will be stemless and place them directly on the floor.

Arcade Button Test

Thesis

Objective:
Wire up a 12v LED Arcade Push Button. The buttons will be embedded in the center of the giant flowers and will be the proxy for “pollination”.

Challenges:
I was able to get correct HIGH and LOW readings in the serial monitor for the push button, with the LED lighting up while the button was HIGH. However, the big challenge I had was keeping the LED turned on, even when the button was not being pressed. After searching around online for solutions, I found a blog that included a circuit diagram with a transistor. I modified my circuit to include a transistor, and voila! Here is my own diagram of the arcade button test circuit:

The reason this is important is because during the interaction I would like the kids to “pollinate” a flower by pushing on the button—and have the LED remain on to signal that the flower has already been pollinated.

Takeaways:
The button works! Now I need to figure out how to mount the button and decide how best to disguise the wires (I will probably run the wires down the stem and stick the battery in the pot).

Bonus Content:
You can see the button proudly featured in my 100 Days of Making Post below.

Flower Fabrication Part I

Thesis

The first image that struck me when conceptualizing my thesis was a room full of colorful gigantic flowers. My intention was to help transport the kids into the “honeybee mindset” by allowing them to interact with flowers that are more to scale with a bee’s POV.

Fabrication is not my strength so I consulted the Patron Saint of Crafts: Martha Stewart. Her website has an excellent tutorial on how to construct giant flowers using fabric. I modified the template a little bit based on the fabric swatches that I had purchased so I could get the most petals out of each swatch. Each petal is about 14 or 15 inches long, and made of two pieces of fabric bonded together with iron-on adhesive. In the center of each petal is a 20 inch piece of floral wire—this allows me to bend the petals into a desirable shape.

The tricky part has been securing the petals to the stem, which is made of armature wire. The blog recommended using floral tape, which adheres to itself very well, but adheres to wire very poorly. It was difficult to place the petals at equidistant points around the stem. I’ve only attached three petals so far because I need to leave room in the center for my arcade button!

DRAWING MUSIC PART II: MIDI ACTION

Tangible Interactions

Assignment: Make a musical instrument using the MIDI control protocol.

Concept: An interface where the user draws on a piece of paper to make music.

Takeaways: In a way, it’s fitting that the “instrument” sounds like a child banging on a piano, because it represents Step One in the path towards musicianship. What I’ve created doesn’t really function as an instrument, but it is Step One in the path to Instrument Actualization. Here are my final takeaways.

  • Paper is more conductive than I thought.
  • If using the Software Serial library with a Leonardo, not all pins will work (I used pins 8 and 9).
  • I used conductive fabric when making the FSRs, but copper tape might have made more sense (easier to cut, don’t need to worry about fraying)
  • What is a good amount of FSRs, and what layout makes the most sense?
  • How do I solve the crazy note fluctuation and give the user more control over the notes that get produced?