From text to counts टेक्स्ट से गिनती तक

1. Tokens: cutting text into words

Chatbots like ChatGPT read text as tokens — small pieces, roughly words. Our tokeniser will be simple:

1. make everything lowercase, so “Chai” and “chai” are the same word
2. split wherever there is something that is not a letter or digit (spaces, commas, !)
3. throw away empty pieces

“Chai, please! Ek aur chai.” → ["chai", "please", "ek", "aur", "chai"]

Hinglish written in English letters works fine; Devanagari would need a bigger letter range — a nice extra challenge later.

Challenge: Write a tokeniser

Write tokenise(text): lowercase the text, split it with /[^a-z0-9]+/, and remove empty strings. Return the array of words.

2. Counting words

The simplest “language model” just knows which words are common. Count them with an object used as a tally sheet:

counts[word] = (counts[word] || 0) + 1;

The || 0 handles a word we have never seen (its count is undefined at first).

To find the top word, walk through the object and keep the word with the biggest count. If two words tie, keep the one you met first — only switch when a count is strictly bigger.

                const words = ["chai", "garam", "chai", "biscuit"];
const counts = {};
for (const w of words) counts[w] = (counts[w] || 0) + 1;
console.log(counts);
              

Challenge: The chai-stall word count

Write countWords(tokens) that returns an object of word → count. Then write topWord(counts) that returns the word with the highest count (on a tie, the one that comes first in the object). Return null if counts is empty.

3. Bigrams: what comes next?

A bigram is a pair of neighbouring words. In “chai is hot chai is sweet” the pairs are: chai→is, is→hot, hot→chai, chai→is, is→sweet.

Count them in a table of tables:

{ chai: { is: 2 }, is: { hot: 1, sweet: 1 }, hot: { chai: 1 } }

Now the model can answer “after chai, what usually comes?” This is a baby version of what your phone keyboard does — and, scaled up enormously with neural networks instead of counts, of how large language models guess the next token.

Try it above: type a few training sentences, then a word, and see the top next words with their counts. Press “generate” to let it write.

Challenge: Build the bigram table

Write bigramCounts(tokens): for every neighbouring pair tokens[i] → tokens[i + 1], add 1 to table[tokens[i]][tokens[i + 1]]. Return the table (an object of objects). The last word has no next word, so it adds nothing.