C:\> java TokenTester Enter a string: val = 12+8 val 12 8
You probably regard the "=" and the "+" symbols as important,
so they should be regarded as tokens.
But "+" also needs to work as a delimiter so that "12+8"
is broken into the tokens "12", "+" and "8".
This is done by using true as the third parameter
in the constructor:
Now individual delimiters (as well as tokens) are returned
by nextToken().
import java.io.*;
import java.util.*;
public class TokenTester
{
public static void main ( String[] args ) throws IOException
{
BufferedReader stdin =
new BufferedReader(new InputStreamReader(System.in));
System.out.print("Enter a string:");
String data = stdin.readLine();
StringTokenizer tok =
new StringTokenizer( data, " =+-", true ); // note the space before =
while ( tok.hasMoreTokens() )
System.out.println( tok.nextToken() );
}
}