Did you know that you can program Pepper Robot using Java programming language? Not just that. Using IBM Watson, you can also connect it with  speech-to-text for high quality speech recognition! Here is a tutorial to show you how you can do it.

This blog will give you a step by step procedure on how you can program a Pepper Robot using Java.

 

 

Programming Pepper Robot using Java

Follow the steps in http://doc.aldebaran.com/2-5/dev/java/index_java.html -> “Directly in the Aldebaran robot” section. Those steps are repeated here for quick reference (along with some minor modifications needed for my version of Pepper robot).

 Note: You can ssh into Pepper Robot using default user account “nao” (http://doc.aldebaran.com/2-5/dev/tools/opennao.html). Note down the ip-address of Pepper robot and add an entry into your /etc/hosts like below:

10.20.1.13 nao.local

Change ip-address above to match the ip-address assigned to your Pepper Robot.

  • Step-1 Download Java SDK 32-bit from the Oracle website.

The Pepper robot that I had access to was running a flavor of Linux 32-bit on x86 architecture. Use command “uname -a” to know details of your Pepper version. Hence I downloaded jdk-8u121-linux-i586.tar.gz . Note: I downloaded JDK instead of just JRE, as I wanted the ability to edit/compile Java programs (using javac command) on Pepper robot also.

  • Step-2 Upload and then extract it somewhere on the robot.
scp jdk-8u121-linux-i586.tar.gz nao@nao.local:
ssh nao@nao.local
tar xvfz jdk-8u121-linux-i586.tar.gz
mv jdk1.8.0_121 /home/nao/java
  • Step-3 Copy the java-naoqi-sdk-<version>-linux32 jar on the robot.

For instance, in: /home/nao/java/

scp java-naoqi-sdk-2.4.3.28-linux32.jar nao@nao.local:/home/nao/java/

Note: To download java-naoqi-sdk-2.4.3.28-linux32.jar, go to https://developer.softbankrobotics.com/us-en/downloads/pepper -> Other SDKs and utilities -> Java -> 2.4.3 -> Java SDK 2.4.3 Linux 32

  • Step-4 Create a sourceme file looking like:
nano /home/nao/java/sourceme

export JAVA_HOME=/home/nao/java/
export PATH=$PATH:/home/nao/java/bin
export CLASSPATH=.:/home/nao/java/java-naoqi-sdk-2.4.3.28-linux32.jar
  • Step-5 Compile your code and upload the .class files to the robot.
  • Step-6 Run “source sourceme” and use the command line to run your Java application.

Sample Java program to run on Pepper Robot – Demonstrates use of Pepper’s Text to Speech service

import com.aldebaran.qi.Application;
import com.aldebaran.qi.helper.proxies.ALTextToSpeech;

public class SayHello {
    public static void main(String[] args) {
        // The robotUrl is specified here
        String robotUrl = "tcp://10.20.1.13:9559";
        Application application = new Application(args, robotUrl);
        try {
            // Start the application and create a session.
            application.start();
            // A session has been created. It can be retrieved this way:
            // application.session();
            // Create an ALTextToSpeech object and link it to your current session
            ALTextToSpeech tts = new ALTextToSpeech(application.session());
            // Make your robot say something
            tts.say("I can be programmed using Java! Isn't that awesome?");
        } catch (Exception e) {
            // The application could not be started.
            e.printStackTrace();
        }
    }
}

Compile and run Java program:

$ javac SayHello.java
$ java SayHello

Connecting Pepper Robot and Watson Speech to Text using Java

Standalone Java program that streams audio from microphone to Watson Speech to Text service and displays transcribed text

scp java-sdk-3.7.0-jar-with-dependencies.jar nao@nao.local:/home/nao/java/

Add Watson Java SDK to CLASSPATH: Edit sourceme file and update CLASSPATH as below:

export CLASSPATH=.:/home/nao/java/java-naoqi-sdk-2.4.3.28-linux32.jar:java-sdk-3.7.0-jar-with-dependencies.jar
  • Step-3 Create file SpeechToTextUsingWatson.java and copy following program. Specify service credentials <username> and <password> in constructor.
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.TargetDataLine;

import com.ibm.watson.developer_cloud.http.HttpMediaType;
import com.ibm.watson.developer_cloud.speech_to_text.v1.SpeechToText;
import com.ibm.watson.developer_cloud.speech_to_text.v1.model.RecognizeOptions;
import com.ibm.watson.developer_cloud.speech_to_text.v1.model.SpeechResults;
import com.ibm.watson.developer_cloud.speech_to_text.v1.websocket.BaseRecognizeCallback;

public class SpeechToTextUsingWatson {

    SpeechToText service = new SpeechToText();
    boolean keepListeningOnMicrophone = true;
    String transcribedText = "";

    public SpeechToTextUsingWatson() {
        service = new SpeechToText();
        service.setUsernameAndPassword("<username>", "<password>");
    }

    public String recognizeTextFromMicrophone() {
        keepListeningOnMicrophone = true;
        try {
            // Signed PCM AudioFormat with 16kHz, 16 bit sample size, mono
            int sampleRate = 16000;
            AudioFormat format = new AudioFormat(sampleRate, 16, 1, true, false);
            DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);

            if (!AudioSystem.isLineSupported(info)) {
                System.err.println("Line not supported");
                return null;
            }

            TargetDataLine line = (TargetDataLine) AudioSystem.getLine(info);
            line.open(format);
            line.start();

            AudioInputStream audio = new AudioInputStream(line);

            RecognizeOptions options = new RecognizeOptions.Builder()
              .continuous(true)
              .interimResults(true)
              .inactivityTimeout(5) // use this to stop listening when the speaker pauses, i.e. for 5s
              .contentType(HttpMediaType.AUDIO_RAW + "; rate=" + sampleRate)
              .build();

            service.recognizeUsingWebSocket(audio, options, new BaseRecognizeCallback() {
                @Override
                public void onTranscription(SpeechResults speechResults) {
                    // System.out.println(speechResults);
                    String transcript = speechResults.getResults().get(0).getAlternatives().get(0).getTranscript();
                    if (speechResults.getResults().get(0).isFinal()) {
                        keepListeningOnMicrophone = false;
                        transcribedText = transcript;
                        System.out.println("Sentence " + (speechResults.getResultIndex() + 1) + ": " + transcript + "\n");
                    } else {
                        System.out.print(transcript + "\r");
                    }
                }
            });

            do {
                Thread.sleep(1000);
            } while (keepListeningOnMicrophone);

            // closing the WebSockets underlying InputStream will close the WebSocket itself.
            line.stop();
            line.close();
        } catch (LineUnavailableException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return transcribedText;
    }

    public static void main(String[] args) {
        SpeechToTextUsingWatson speechToTextUsingWatson = new SpeechToTextUsingWatson();
        String recognizedText = speechToTextUsingWatson.recognizeTextFromMicrophone();
        System.out.println("Recognized Text = " + recognizedText);
        System.exit(0);
    }
}
  • Step-4 Compile and Run Java program:
$ source sourceme
$ javac SpeechToTextUsingWatson.java
$ java SpeechToTextUsingWatson

References

3 comments on"Connecting Pepper Robot with Watson Speech to Text – Java program"

  1. KISHORE CHOWDARY April 04, 2017

    Congratulations on your article sir. Since I am a beginner in Java programming, I feel this one a bit difficult. But I have already worked with Watson. But for sure I’ll try this. But a nice one.

  2. Nice Article Shiv,

    Can you put some light on Connecting Pepper Robot and Watson Speech to Text using Javascript or Python ?

Join The Discussion

Your email address will not be published. Required fields are marked *