Floyds 2 bit Robotics Meeting 2

The first meeting focused on getting sensor readings from a moisture sensor, turning on LEDs, and setting the position of a servo with PWM. All of it was done on the Arduino Uno. Often times we want multiple systems to work together, so for this meeting, we'll focus on integrating some existing tools along with our own modifications to communicate between a webpage we built and the Arduino Leonardo. An important takeaway I want everyone to understand is that you can use existing tools in coordination with your own. If it's already been written and is available, it's often advantageous to use existing and proven systems rather than creating your own from scratch.

This time we'll build a skittle sorter that sorts skittles using computer vision with a model we'll train on Google's "Teachable Machine". We'll use an Arduino Leonardo and communicate between JavaScript on a webpage and the Arduino code running on the Leonardo. For my example, I want to only keep red skittles and dump all of the others into a reject pile.

I designed a skittle slide that uses a vibration motor to drop one skittle into a swinging arm attached to a servo. After the vibration motor slides a skittle into the arm, the servo will move the skittle in front of the camera. Depending on what it sees ('red skittle', 'no skittle', not red skittle), we'll tell the servo to move all the way to the left to "keep" the red skittle, all the way to the right to "dump" the non-red skittle, or if there is no skittle detected, it will return to the pick-up position and run the vibration motor again.

• Level 1 Assembly - inserting the vibration motor, servo and arm onto the slider assembly and then the slider assembly onto your laptop screen where the arm can be seen by the camera.
• Level 2 Training your Teachable Machine model to recognize the three states (red skittle, non-red skittle, or no skittle)
• Level 3 Adding the code to the Leonardo
• Level 4 Build a webpage that ties it all together and allows you to control our skittlebot

You can download the 3d printable stl files here: Skittle Sorter Mount and Skittle Sorter Arm.

inserting the vibration motor, servo and arm onto the slider assembly and then the slider assembly onto your laptop screen where the arm can be seen by the camera.

Assembly Instructions for Skittle Sorter

Insert the vibration motor into the Skittle Slide/Mount:

• The slot in the Slide/Mount is slightly undersized and we heat/melt the PLA slightly to properly insert the vibration sensors
• Remove any support material that was added during the printing process
• Use a heat gun, lighter, or soldering iron with a tip for heating PLA to heat the PLA at the slot (do not overheat)
• Slide the vibration motor into the slot and keep it still while the PLA cools to securely hold the vibration motor in place.

Insert the servo into the Skittle Slide/Mount:

• Pay attention to the wires, they should slide through the slot in the Skittle Slide/Mount (gently pull the sides apart to allow the wires to fit, being careful not to break the mount)
• Slide the skittle Arm over the gear of the servo (we'll likely remove and re-adjust the position later, just make sure not to strip the hole on the arm.

Connect the wires for the vibration motor and the servo to the Arduino Leonardo:

• Note: This is a different Arduino than we had before, we are using a Leonardo for this meeting because a Leonardo can be seen by your computer as being a human interface device (like a mouse or keyboard, - this allows us to have two way communication with our code between the computer and the Arduino!)
• Attach the black wire from the servo to a Ground pin on the Arduino Leonardo
• Attach the red wire from the servo to Vin pin on the Arduino Leonardo
• Attach the white wire from the servo to Pin 3 on the Arduino Leonardo (Any PWM pin would work, but our code will be using pin 3)
• Attach the blue wire from the vibration motor to a Ground pin on the Arduino Leonardo
• Attach the red wire from the vibration motor to a pin 4 on the Arduino Leonardo (Any Digital pin would work, but our code will be using pin4)

Place the Skittle Slide/Mount onto your laptop screen to ensure it fits.

That's It! Congratulations, If you followed along you have assembled the Skittle Sorter and now we're ready to train a computer vision model!

1. Access Teachable Machine: Navigate to the Teachable Machine website (https://teachablemachine.withgoogle.com/) and click "Get Started" to begin.
2. Select Model Type: Choose an Image project, and use the standard image model (NOT the embedded image model)
3. Webcam Setup: Prepare your webcam to capture images of skittles. Ensure good lighting and a clear view of the skittles in the Skittle sorter arm.
4. Training Setup: Create 3 Classes: red skittles, no skittles, and non-red skittles in your space for training. Capture several angles and orientations of the skittles
5. Train Your Model: Initiate training by selecting "Train Model". Teachable Machine will use captured images for training the model to distinguish between the three classes.
6. Model Evaluation: After training, test your model's real-time performance using the webcam feature to identify red skittles, no skittles, and non-red skittles. Adjust and revise your classes and re "train" until you are happy with the results.
7. Export TensorFlow Lite Model: Once satisfied with the model's performance, export it as TensorFlow Lite. This lightweight format is ideal for deployment on mobile and embedded devices.
8. Integration: Move on to level 3 to start our integration!

Check out Level 3

After completing level 2, you should have an assembled skittle sorter and a trained computer vision model. Now we'll start on the Arduino Leonardo.

1. Plug one end of the Micro USB to your computer and the other end to the Arduion Leonardo.
2. Open the Arduio IDE (Integrated Development Environment), Ensure the selected board is "Arduino Leonardo" and the port matches the USB port we plugged in

Now lets add our Aruino code for this setup

// StandardFirmata sketch for Arduino Leonardo

#include <Arduino.h>
#include <Wire.h>
#include <Servo.h>
#include <Firmata.h>

Servo myservo;
bool paused = false;

void setup() {
  Serial.begin(57600);
  while (!Serial) {
    ; // Wait for serial port to connect. Needed for native USB port only
  }

  Firmata.setFirmwareVersion(FIRMATA_FIRMWARE_MAJOR_VERSION, FIRMATA_FIRMWARE_MINOR_VERSION);
  Firmata.begin(Serial);
  myservo.attach(3);  // attaches the servo on pin 3 to the servo object.
  pinMode(4, OUTPUT);
}

void loop() {
  if (!paused) {
    myservo.write(105);
    delay(2000);
    digitalWrite(4, HIGH);   // turn on the shaker
    delay(2000);            //wait for 2 seconds
    digitalWrite(4, LOW);   //turn off the shaker
    delay(100);               //let the previous commands finish
    myservo.write(50);      //position so can look with camera
    Serial.println("Look at skittle"); //tell the webpage to look
    delay(1500); //give servo time to get in position
  }

  while (Serial.available() > 0) {
    int command = Serial.read();
    if (command == 49) {
      // Example: If received command is 0x01, perform some action
      Serial.println("Skittle is Red - cool"); //Move servo to keep it'
      myservo.write(0);    //dump it to the left
      delay(2000);
    }
    else if (command == 50) {
      Serial.println("No Skittle"); //Move Servo back to start and shake
    }
    else if (command == 51) {
      Serial.println("Not Red Skittle"); //Move Servo to dump it'
      myservo.write(270);    //dump it to the right
      delay(3000);
      myservo.write(260);
      delay(200);
      myservo.write(270);
      
    }
    else if (command == 52) {
      paused = true;
    }
    else if (command == 53) {
      paused = false;
    }
    else {
      Serial.println("What!");
    }
    Serial.println("Waiting");
  }
}

Easy Right!, we'll likely need to revisit and adjust the code to get it set just right for your skittle sorter