Confused

Sometimes when I swipe towards or away from the arrows that point inside the box, it says Im wrong. But sometimes Im right and it lets me keep going? Im confused on what Im supposed to do with them...

Mrs. Buddha about Swiperoo