Today's Goal: Imagine that you go grocery shopping. Or you simply need to take your robot demo and tools from your lab to your office. Instead of pushing a cart, what if the cart simply smartly followed you? Budgee, the Robot Assistant, is a commercial attempt at doing exactly that!
Create assignment repository
Example Solutions for Part a
Example Solutions for Part b
In this problem set, you will program your robot to follow you around at a safe distance, while also avoiding obstacles. The robot shouldn't run into you, shouldn't lose you too easily, and should work both inside the classroom and outside in the hallways of Pierce.
There's several parts to implement:
Part (a) Recognizing you:
To make this easier, you will wear a leg band on your leg and the robot should recognize that. This way the staff can also wear the band to test your robot. We have placed the bands in the closet in the white plastic container and there should be at least one band per robot.
(1) Your robot must show a display of its camera image and draw a bounding box around the leg band. (For debugging, it is also helpful to display the masked image). The TFs will walk around within the visual field of the (stationary) robot and see the range (angle and distance) over which you can detect the band. You should try to be able to detect within 4 feet at least. If you see multiple bands/color blobs, you should pick the best one (e.g. biggest size). Your robot should stay stationary for this test (no turning or moving)
(2) You should report the distance to the band by using the depth sensor. Note that the depth sensor image and RGB images are reasonably aligned.
Hints: It is critical to get part (a) working well in order to do well at the rest of the problem set. One hard part of part (a) is dealing with color changes due to lighting. See Lab 2 solutions repository for a openCV "trackbar" (
view_images_threshold.py) that will allow you to easily calibrate your HSV color bounds for detecting specific colors. Also you should test your code under different light conditions (not just at night).
Part (a) is due first and worth 5 points.
Part (b) Doing a good job following you:
Centering and Safe distance: The robot should maintain approximately 2 feet distance from you, while you are walking at a normal pace. This means that even if you turn, it should follow: We will mark a curved test path in Pierce 301 with masking tape. If you walk that path at a normal walking speed the robot should be able to follow you (it does not need to follow the path itself). If you stop, the robot should stop. There is one exception -- if you walk towards the robot, the robot should not back up. Backing up is dangerous, so the robot should just stay stationary if you are too close.
A good way to follow is to try and keep the leg band in the center of the view ("centering"). You can use a P-controller so that you turn at a rate that is proportional to the angle (e.g. more quickly if the band is off-center), or move forward at a rate that is proportional to the distance. This reduces the chance that the person moves too quickly for your robot to follow.
In general the robot should be able to follow you as smoothly as possible. This is the first part you should get working. You can use the velocity smoother, which the TFs have described on piazza. Maximum speed allowed is 0.3 m/s.
Getting lost: If the robot loses sight of you, it should stop and continuously beep (so we know) and scan (turn slowly in a circle) from a stationary position looking for you. Once it relocates the band, then it can follow you again as soon as it sees you.
The robot can lose sight of the leg band for many reasons. Maybe you accidentally walked too fast, or someone crosses between you and the robot, temporarily obscuring the view of the ankle brace. Temporarily losing sight should not cause a big disruption, so you should only beep if the ankle brace is not visible for several frames.
Obstacles: Our room has obstacles that are not always the same for a robot vs a person. The robot should be able to follow you around a hallway corner or around a table (we will test both), and not run into obstacles. If for some reason the robot cannot find a way around the obstacles (for example if the person walks through two closely placed chairs), the robot should beep a warning signal to indicate it can't follow. In general, keep your robot "safe".
Test 1: Staff Setup. The robot follows the Teaching Staff at a reasonable speed throughout the classroom, through a simple obstacle setup. We will make sure that the path through the setup doesn't have too tight curves (similar to the marked path for testing) and all robots will be tested on the same path. Robots will be scored for the smoothness of their following behavior, and for not bumping into obstacles or losing sight of the staff member.
Test 2: Losing you. We will do a simple test to see how your robot reacts to losing sight of the ankle brace, or to seeing two braces temporarily.
Test 3: The Pierce Hallway Race! We will use the hallway right outside Pierce 301 as our race course. You will walk your robot down the hall and back, at your best following behavior, and we will time it!
Extra Credit: If you have the above under control, then show something cool of your choice!
- Backup Plan: Take a video of your robot whenever it is working at its best and submit that to the TFs along with your code.
Grading Scheme (15 points): The teaching staff will assign a grade of up to 12 points based on the in-class performance on the 3 tests (5:2:5), and a grade of up to 3 points based on code submission. Robot behavior will be judged on how smoothly it follows, in addition to the test performance. Code will be judged on how well thought out your methods are as well as how organized and clear your code is. Your code should be well commented and you should submit a 1-2 page document explaining your strategy.
Food for thought: There are many other ways to recognize a person, e.g. could you recognize their shoes or their faces? You could also add voice commands, like telling the robot to wait or the robot calling you if it is lost. However no matter what you pick, having a system work robustly in a complex environment where there are lots of other people is not that easy, even with more sophisticated sensors and algorithms.