C++ get next token using strtok()

Question

I am new to C++. I want to write a function called get_token() that will return the next token. For example str="123 456 789"; after I call get_token(str), it will return 123, then if I call get_token(str) again, it will return 456.

void get_token(char str[]) {
   char* token = strtok(str," ");
   while (token) {
      cout << "Token: "<< token << endl;
      token = strtok(NULL, " ");
   }
}

Can someone help me with this function? And write it to char* get_token(...)

This is almost precisely how `strtok` works already. The only thing different is you want to avoid specifying the delimiters. Why not `char* get_token(char *s = NULL) { return strtok(s, " "); }` -- and then you just use `for (char *tok = get_token(str); tok != NULL; tok = get_token()) cout << tok << endl;` -- in C++, you're probably better off using `std::istringstream` instead of `strtok` which is not a very safe function and not usually recommended. — paddy, Feb 19 '21 at 05:21
Please don't describe C++ questions as C. It's understood that in C++ you can use C techniques. — tadman, Feb 19 '21 at 06:01

brc-dd · Answer 1 · 2021-02-20T08:15:51.710

If there is no restriction on using std::string you can do something like this:

#include <iostream>
#include <string>

auto get_token(std::string &s, std::string delim = " ") {
    std::string token;
    if (s.back() != '\0') {
        auto parsed_till = s.find_last_of('\0') + 1,
             token_end   = s.find(delim, parsed_till);
        if (token_end != std::string::npos) {
            token = s.substr(parsed_till, token_end - parsed_till);
            s.replace(token_end, delim.size(), delim.size(), '\0');
        } else {
            token = s.substr(parsed_till);
            s += '\0';
        }
    }
    return token;
}

int main() {
    std::string str   = "1233 45 678 90";
    std::string token = get_token(str);
    while (not token.empty()) {
        std::cout << token << '\n';
        token = get_token(str);
    }
}

Note: Calls to get_token will mutate the passed string. Please copy the string beforehand if you want to reuse it in the program.

Additional reading: Parse (split) a string in C++ using string delimiter (standard C++)

Pat. ANDRIA · Answer 2 · 2021-02-19T07:59:28.933

Does this suit your need?

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_BUF_SZ 1024



char** get_token(char *str, int* len)
{

    char *copy_string;
    copy_string = (char*)malloc(MAX_BUF_SZ);

    strcpy(copy_string, str);
    // suppose we would have MAX_BUF_SZ tokens
    char** result = (char**)malloc(MAX_BUF_SZ * sizeof(char *));

    char *token = strtok(copy_string, " ");
    int i = 0;

    while(token != NULL)
    {
        result[i] = (char *)malloc(strlen(token)+1);
        result[i] = token;
        token = strtok(NULL, " ");
        //
        i++;
    }
    // to keep track of how many elements are in the result
    *len=i;
    // return the result
    return result;
}


int main()
{
    char ptr[]= "123 456 789";
    int* length =(int*)malloc(sizeof(int));
    char** tokens = get_token(ptr, length);
    int nb_elements = *length;

    // use tokens here
    printf ("the tokens are ");
    for (int i=0; i< nb_elements; i++)
    {
        printf("%s ", tokens[i]);
    }
    return 0;
}

This gives an output of :

the tokens are 123 456 789

is that possible to get the result like this: `cout<result:123. 456. 789? — ZZZZZlll, Feb 19 '21 at 08:56

C++ get next token using strtok()

2 Answers2