Processing/Arduino application step-by-step

Here you will learn to build your own Pachube/Arduino/Processing application step-by-step, connecting Arduino to Processing to Pachube. This tutorial consists of two basic steps.

1. Use Arduino to create an input to Pachube (i.e. provide Arduino's realtime sensor readings for others to connect to).
2. Use an output from Pachube to control Arduino (i.e. to have your Arduino respond to realtime sensor readings from remote devices).

These tutorials assume that you are using Arduino 12 and Processing 1.0.1, with an Arduino NG or Diecimila device. The tutorials also work with earlier versions of Processing (tested to v0135) and Arduino, though you will need to download Firmata manually, and check baud rate depending on which version of Firmata you choose.

Quickstart: All source files are available here. Inside this zipped folder are 5 Processing programmes. The first four are used in the explanations below; the fifth, arduinoInputOutput_Pachube_05 can be used as a quick start to create an applet that uses both inputs from and outputs to Pachube.

Please note that Pachube.com will be live and open to the public in a few months; until then, it will not be possible to add a feed to the repository or receive data from Pachube feeds unless you are a beta-tester: Sign up for Pachube here.

A. Using Arduino to create a Pachube input

This would enable your local sensor readings to be accessible to others in realtime.

Part 1. Getting Arduino, Processing and Firmata up and running

1. First, upload Firmata to your Arduino. Arduino 12 includes this by default, so simply load the sketch found under File > Sketchbook > Examples >Library-Firmata > StandardFirmata and upload to your Arduino.
2. Download and install the Firmata v2 Arduino library for Processing -- that download is currently located here: http://www.arduino.cc/playground/uploads/Interfacing/processing-arduino2... but may change. The entire "arduino" folder should be placed in Processing's "libraries" folder.
3. Make a simple light sensor circuit attached to analog pin 0 (something like this circuit would do fine).
4. Open up Processing and use the following code as your initial programme:
   import processing.serial.*;
import cc.arduino.*;

Arduino arduino;
float myValue;

void setup()
{
  println(Arduino.list());
  arduino = new Arduino(this, Arduino.list()[0], 115200);
}

void draw()
{
  myValue = arduino.analogRead(0);
  println(myValue);
}

   [Note: You can download the source files here and use arduinoInput_Firmata_01 for this exercise].

   Depending on which particular USB port is connected you'll have to change the '0' in the following line to the correct number:

   arduino = new Arduino(this, Arduino.list()[0], 115200);

   You can find out what number to use there because when you run your Processing programme, the first thing it does ("println(Arduino.list());") is display a list of serial ports in your Processing console. Look for the one that says something like "/dev/tty.usbserial-A4001cKk".

5. If this is working correctly, you should see a value printed to your console several times a second that changes as you change the light level (or cover up the light sensor with your hand).

Part 2. Adding the Arduino's realtime sensor reading to a Pachube feed

Using the code above, you just need a few extra lines to get it ready for serving to the Pachube website. The procedure is:

1. Install the EEML library for Processing.
2. Edit your previous code so that it looks like:
   import processing.serial.*;
import cc.arduino.*;

import eeml.*;

Arduino arduino;
float myValue;

DataOut dOut;

void setup()
{
  println(Arduino.list());
  arduino = new Arduino(this, Arduino.list()[0], 115200);
 
  dOut = new DataOut(this, 5210);
 
  dOut.addData(0,"light sensor, LDR, light level");
 
}

void draw()
{
myValue = arduino.analogRead(0);
println(myValue);
}

void onReceiveRequest(DataOut d){
    d.update(0, myValue);
}

   [Note: You can download the source files here and use arduinoInput_Pachube_02 for this exercise].

   Looking at this additional code line by line:

   import eeml.*;

   ... imports the EEML library. (EEML is the data format used by Pachube. To see the full range of methods available, see the complete EEML documentation here).

   DataOut dOut;

   ... initiates "dOut" which is a "DataOut" object. Basically this is an object that will handle all the information about data we want to send "outwards", i.e. that will be available for others to connect to via Pachube, and handles serving that data every time it's requested.

   dOut = new DataOut(this, 5210);

   ... creates the object dOut, specifying that it will serve on port 5210. (As an all-too-brief intro to 'ports' you should understand that, while your computer might have a single IP address, it has many different 'ports' each assigned to different types of network activity; e.g. 80 for serving web pages, 110 for POP mail activity, etc.). If for some reason 5210 is already in use by another application then you can choose any number, but try something higher in value rather than lower where they're more likely to be in use. By doing this, you computer will know that every time it gets a request on port 5210 it should forward that request to this Processing applet.

   dOut.addData(0,"light sensor, LDR, light level");

   ... adds a new data stream to the feed. Its ID number is '0', and it is tagged by a comma-delimited list of descriptions. You can use whatever you like here to describe how this value should be listed on Pachube.

   void onReceiveRequest(DataOut d){
    d.update(0, myValue);
}

   ... is the final chunk. When using the EEML library, you always need this as a callback function: basically this is the method that is called every time your Processing programme receives a request for data. (In case you have a number of different DataOut objects, operating on different ports, it also knows which particular one is being requested). The line "d.update(0,myValue);" is just saying something like "for DataOut object 'd', update stream '0' with the value of 'myValue'".

   That's all there is to it! Of course you can add other pin readings, assigned to different float values and tag those each with their own descriptive tags. (They'll also each need their own new dOut.addData() and d.update(), but all sensor readings can be handled by the single dOut object).

3. Test locally first, to make sure it's serving correctly; the simplest way to do this is open up your browser and type in http://localhost:5210/ as the address. Most browsers should show the XML returned, looking something like this:
<?xml version="1.0" encoding="UTF-8"?>
<eeml xmlns="http://www.eeml.org/xsd/005"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.eeml.org/xsd/005
http://www.eeml.org/xsd/005/005.xsd" version="5">
<environment>
  <data id="0">
    <tag>light sensor</tag>
    <tag>LDR</tag>
    <tag>light level</tag>
    <tag>light dependent resistor</tag>
    <value>238.0</value>
    <unit> </unit>
  </data>
</environment>
</eeml>

... though some (like Safari) only return the non-XML portion, which might look something like this: "light sensor LDR light level light dependent resistor 247.0", though if you "View source" it should show the complete XML.

Don't worry about what any of this XML means -- it's only important to Pachube and you don't need to be able to read it yourself. The important thing is that it's working.

If you get an error message, make sure that your Processing applet is actually running, and without errors.

4. The next step, once you're sure it's running locally is to make sure it's visible externally; i.e. outside of your computer. In order to do this, you need to know what your IP address is and to make sure that external traffic will reach your computer.

   If you are behind a firewall (or if for some reason you are having difficulty with getting Pachube to see your app), you may need to use Pachube's "manual" update feature: see here for details on how you will need to change your Processing sketch for manual updates to Pachube then come back to this page and carry on with step 5 below.

   If your computer is directly connected to the internet (e.g. it's actually attached to the DSL modem) then chances are your external IP address is easy to find (as you can determine from System Preferences > Network (Mac OS X) or using Start > Run > cmd.exe > ipconfig /all (Windows); see http://www.simplehelp.net/2006/ 07/06/ how-to-determine- your-ip-or-mac- address-in-windows/ for more info).

   If you think you now have the correct IP address, you can double check by going to http://whatismyip.com/ which should tell you automatically what your externally-visible IP address is.

   If they correspond then all is good; if not it probably means you're on a local network.

   If your IP address, as determined on your computer, looks anything like 192.168.xx.xx then that's definitely on a local network (i.e. it's not visible externally).

   If you're on a local network, then you need to ensure that external traffic that requests port 5210 (or whichever port you used) is forwarded to your computer. This might require talking to your System Administrator or if you have access to your own router then basically you need to enable "port forwarding" for 5210 to your local IP address (e.g. 192.168.0.10). This procedure will vary from router to router and is not generally complex. For example to configure a Netgear router, see http://kbserver.netgear.com/kb_web_files/N101145.asp.

   If your external IP address changes frequently (i.e. it's "dynamic") you can also use a free service like DynDNS to create static hostname.

   Once you know your IP address, and port forwarding is enabled you are ready to add your feed to the Pachube repository!

5. To add your feed to Pachube, you need to:
   • first, login to Pachube or sign up.
   • then select "Input: add a feed" or go here.
   • Find your location on the map and double-click to add a new location.
   • Enter a "Feed title".
   • Select "Feed type" automatic (or, if you had to use the manual feature earlier, select "manual" and ignore the next 3 instructions, skipping instead to "Fill in any other meta-data...").
   • For "Feed URL" type in your feed IP address and port, which might look something like 123.45.67.89:5210; or if you have a URL (e.g. you are using something like DynDNS) then use that instead.
   • You must click "before continuing click here to retrieve feed!" in order to make sure that Pachube can see your feed. If you have any problems, then check the URL to make sure it is correct; or use Validome to make sure that your EEML is serving correctly.
   • If there are no errors, then you should see the section "Datastreams" filled with your data values, including the tags you used for each datastream. Feel free to edit them now, or add to them as necessary. Verify that your light sensor reading is being read correctly by Pachube.
   • Fill in any other meta data about this feed (e.g. a description, a contact email address about it (if you wish it to be publicly available), give it a location name and select "indoor" or "outdoor" accordingly, etc. etc.).
   • click "Save feed".

   At this point you should be taken to the feed's Pachube page, which will look something like http://www.pachube.com/feeds/504 -- 504 in this case is the feed ID (and therefore its EEML URL is http://www.pachube.com/api/504.xml). If you were using the standard "automatic" feed type, then your feed page will already contain datastreams, values, tags, etc. If you used "manual", then the feed page will at this stage still be awaiting the first update -- so you'll need to return to your Processing applet, replace FEED URL with your EEML URL, and run the application, whereupon the feed page will automatically get populated with the relevant info.

   You can always find this feed page in your own user panel (e.g. http://www.pachube.com/users/hdr/feeds ) and can edit any info at any time. Feel free to give out this URL to others so that they can track the feed, or, if they are also Pachube users, they can access the realtime data. Pachube will log your sensor data whenever it is connected to the internet (so long as your URL or IP address doesn't change!). You can now doing any of a number of things including:

   • build further devices on other remote computers that connect to this feed.
   • distribute the feed URL to others so that they can make devices that remotely connect to it.
   • enable the background colour of your webpage to change with the realtime sensor reading (javascript examples to come).
   • embed a graph of historical data from the sensor readings in your blog.
   • be sure to check out the complete documentation for the EEML library to see other methods that can be applied to your DataOut object, including the ability to set geographical location data; add "units" information for sensor values; and provide maximum and minimum values expected for a particular stream.

B. Using a Pachube output to control an Arduino

This would enable your Arduino outputs to respond to remote sensors from the Pachube repository.

Part 1. Getting Arduino, Processing and Firmata up and running

1. Make sure you have Firmata and the Arduino Firmata library for Processing installed as in Steps 1 & 2, of Part 1. above.
2. Attach an LED to pin 9 on the Arduino (you should really use a resistor in series with your LED, as in this tutorial but since we're not going to be pushing it to full brightness the whole time, this is probably not absolutely necessary).
3. Use the following code as your initial Processing programme:
   import processing.serial.*;
import cc.arduino.*;

Arduino arduino;
int myValue;

void setup()
{
    size(255,255);
    println(Arduino.list());
    arduino = new Arduino(this, Arduino.list()[0], 115200);
}

void draw()
{
    myValue = mouseX;
    println(myValue);
    arduino.analogWrite(9,myValue);
}

   [Note: You can download the source files here and use arduinoOutput_Firmata_03 for this exercise].

   Depending on which particular USB port is connected you'll have to change the '0' in the following line to the correct number:

   arduino = new Arduino(this, Arduino.list()[0], 115200);

   You can find out what number to use there because when you run your Processing programme, the first thing it does ("println(Arduino.list());") is display a list of serial ports in your Processing console. Look for the one that says something like "/dev/tty.usbserial-A4001cKk".

4. If this is working correctly, you should see a value printed to your console several times a second that changes as you move your mouse from side to side over the Processing applet. You should also see your LED brightness go up and down.

Part 2. Retrieving the remote Pachube sensor reading and using this to control the Arduino LED

Using the code above, you just need a few extra lines to get it ready to receive data from the Pachube website. The procedure is:

1. Make sure you have installed the EEML library for Processing.
2. Edit your previous code so that it looks like:
   import processing.serial.*;
import cc.arduino.*;

import eeml.*;

Arduino arduino;
int myValue;

DataIn dIn;

void setup()
{
    size(255,255);
    println(Arduino.list());
    arduino = new Arduino(this, Arduino.list()[0], 115200);   
    dIn = new DataIn(this, "http://pachube.com/api/1228.xml", "ENTER_YOUR_PACHUBE_KEY_HERE", 5000);
}

void draw()
{
   println(myValue);
   arduino.analogWrite(9,myValue);
}

void onReceiveEEML(DataIn d){ 
    float remoteValue = d.getValue(0);
    myValue = int(remoteValue) / 300;
}

   [Note: You can download the source files here and use arduinoOutput_Pachube_04 for this exercise].

   Looking at this additional code line by line:

   import eeml.*;

   ... imports the EEML library. (EEML is the data format used by Pachube. To see the full range of methods available, see the complete EEML documentation here).

   DataIn dIn;

   ... initiates "dIn" which is a "DataIn" object. Basically this is an object that will handle all the information about data we want to retrieve from Pachube and handles requesting that data as needed. You will also need to register with Pachube to receive a Pachube "key". (Keep this private, particularly if you publish code, it's your unique password to access Pachube feeds). Replace the text below with the key that gets emailed to you.

       dIn = new DataIn(this, "http://pachube.com/api/1228.xml", "ENTER_YOUR_PACHUBE_KEY_HERE", 5000);

   ... creates the object dIn, which connects to a specific feed on the Pachube website and requests updated data every 5000 millisecs (i.e. 5 seconds).

   myValue = mouseX; // DELETE THIS LINE!

   ... delete this line because we want to update myValue with the remote sensor reading rather than mouseX.

   void onReceiveEEML(DataIn d){ 
    float remoteValue = d.getValue(0);
    myValue = int(remoteValue) / 300;
}

   ... is the final chunk. When using the EEML library, you always need this as a callback function: basically this is the method that is called every time your Processing programme receives back data that it has requested from Pachube (because it may not receive the data immediately; it could take a few milliseconds, or it might take several seconds, depending on network traffic). In case you have a number of different DataIn objects, pulling data in from different feeds, it also knows which particular one is returning data.

   The line "float remoteValue = d.getValue(0);" is just saying something like "set the value of 'remoteValue' to equal the value of stream ID 0 of the feed notated by 'd'" which is in this case the temperature sensor reading in a London office. Incidentally, you can also use "float remoteValue = d.getValue("LDR");" as an alternative.

   The line "myValue = int(remoteValue) / 300;" converts remoteValue to an 'int' (because Arduino analog outputs must be 'ints'); divides it by 300 (because the remote feed stream (UK's realtime electricity demand in MW) returns values from 0 to approx 60,000 and our output should be scaled from 0 to 255) and sets the result as 'myValue' which will be used in the main draw() loop to set the Arduino's output on pin 9.

The result should be that your LED changes brightness with UK's realtime electricity demand.

Of course you can add other remote sensor readings, assigned to different streams' values and controlling other Arduino output pins, but essentially that's all there is to it!

Be sure to check out the documentation for the EEML library to see all the other methods you can use for requesting data from the remote feeds. For example you can find out the "age" of a stream (i.e. how long since it's been updated); you can get an array of all streams within a feed that are tagged by a particular word; you can request data to be returned as a string (since sometimes feeds have string value streams).