betoesquivel icon

tokenizer in perl

betoesquivel | PRO | 11/09/15 12:27:57 PM UTC | 0 ⭐ | 385 👁️ | Never ⏰ | []
Perl |

270 B

|

None

|

0 👍

/

0 👎

while (<BR>) {
  # Only punctuation that is not part of a word is a token on its own.
  # This only works on this text, because one flaw it has is decimal numbers
  # and currency.
  s/([,"$%\^\&*()+=\[\]!?])/ $1 /g;
  s/[^A-Z.](\.)\s+/ $1 /g;
  s/\s+/\n/g;
  print $_;
}

Comments