Java String Tokens | HackerRank Solution

Java String Tokens is a HackerRank problem about splitting a sentence into words. It sounds trivial, and the core is one call to split. But this is one of those problems where most people pass the sample and then fail hidden test cases, because the input can start with punctuation, contain only spaces, or be completely empty.

What it really tests is how well you understand String.split and its corner cases. That knowledge pays off later whenever you parse CSV lines, log entries or user input.

Problem

You are given one string s. It contains only English letters, spaces and these symbols: ! , ? . _ ' @. A token is a run of one or more letters in a row, so any non-letter character separates tokens. Print how many tokens there are, then print each token on its own line in the order they appear. Duplicate tokens are counted and printed every time.

Input: a single line containing s.

Output: the number of tokens on the first line, then one token per line.

Constraints: the length of s is between 1 and 4 × 10⁵, and s uses only the characters listed above. Some hidden tests are known to contain a line of only spaces, so your code should also cope with zero tokens.

Input:
He is a very very good boy, isn't he?

Output:
10
He
is
a
very
very
good
boy
isn
t
he

Notice how isn't becomes two tokens, isn and t, because the apostrophe is not a letter.

Approach

Rather than listing every separator character, describe what a separator is: anything that is not a letter. In regex that is [^A-Za-z]+. The + means a group of several separators in a row (like ", " or "??") counts as one split point.

1. Read the whole line with nextLine(). Guard it with hasNextLine() in case the input is empty.
2. Split on [^A-Za-z]+.
3. Keep only the non-empty pieces. This step is the one most solutions miss.
4. Print the count, then each token.

Why step 3? split drops empty strings at the end of the result, but not at the start. If s begins with a separator, like " @@abc", the first piece is an empty string "", which would be counted as a token. And if s is only spaces, split returns an array with one empty string, so a naive count prints 1 instead of 0.

Dry run on part of the sample: "boy, isn't he?" splits at ", ", at ', at " ", and the final ? leaves a trailing empty piece that split removes. You get boy, isn, t, he.

Solution in Java

import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;

public class Solution {
    public static void main(String[] args) {
        Scanner scan = new Scanner(System.in);
        // The line may be blank or only spaces, so guard the read
        String s = scan.hasNextLine() ? scan.nextLine() : "";
        scan.close();

        // Anything that is not an English letter acts as a separator
        String[] parts = s.split("[^A-Za-z]+");

        // split() keeps an empty first piece when s starts with a separator,
        // so collect only the non-empty pieces
        List<String> tokens = new ArrayList<>();
        for (String p : parts) {
            if (!p.isEmpty()) {
                tokens.add(p);
            }
        }

        StringBuilder out = new StringBuilder();
        out.append(tokens.size()).append('\n');
        for (String token : tokens) {
            out.append(token).append('\n');
        }
        System.out.print(out);
    }
}

What was wrong with the old code. The previous solution on this page trimmed the string and then split on [!,?._'@\s]+. Trimming only removes spaces, not other symbols. For an input like !!hello world it printed 3 (an empty token plus two words) instead of 2. Splitting on "not a letter" and skipping empty pieces fixes it.

I also print through a StringBuilder because the input can be 400,000 characters long. Printing tens of thousands of lines one by one with System.out.println is slow, and building one string is much faster.

Tests I ran:

• The sample prints 10 and the ten tokens above.
• A line of only spaces prints 0.
• !!hello__world?? prints 2, then hello and world.
• @@abc prints 1, then abc.

Output

10
He
is
a
very
very
good
boy
isn
t
he

Complexity

Time: O(n), where n is the length of s. The split walks the string once, and the filter walks the pieces once.

Space: O(n) for the array of pieces, the token list and the output buffer.

Common mistakes

• Splitting on a single space " ". Punctuation then sticks to words ("boy,") and double spaces create empty tokens.
• Forgetting the + in the regex, which turns ", " into two split points and an empty token between them.
• Counting parts.length directly without removing the leading empty piece.
• Reading with next(), which gives you only the first word, or with nextLine() without checking hasNextLine(), which throws on empty input.

Next in the series: Pattern Syntax Checker. For more string practice, see Java Substring Comparisons.

Tip: try to solve the problem yourself before reading the solution. Use the code as a reference to check your approach, and make sure you understand why it works.

More HackerRank solutions All coding problems MCQ practice