Skip to content
Hallway
0:00 / 0:00

#lexer#robpike#text/template

Lexical Scanning in Go - Rob Pike

GoogleFOSSSydney

1K subscribers • 1 video on Hallway

YouTube
  • Recommended by 14 people: 13 on Hacker News and 1 on Lobsters.
  • A classic. People kept recommending it from 2012 to 2025.
  • Presented by Rob Pike, co-creator of Go
  • Linked in the descriptionSlides on rspace.googlecode.com

Intermediate. Assumes working knowledge.

161K views15 years ago14 recommendations

Programming languagesSystems programmingGoCompilers and interpretersConcurrency and parallelism

Rob Pike demonstrates how to build an elegant, concurrent lexer in Go using state functions, goroutines, and channels.

Lexical Scanning in Go, a talk by Rob Pike at Google Technology User Group given on Tuesday, 30 August 2011. The slides: rspace.googlecode.com/hg/slide/lex.html

Why it's here

14 people recommend this, 1 disputes it

What people say. Highly recommended as a classic, elegant demonstration of using Go's concurrency primitives to simplify compiler frontend design.

Moments people linked to

willvarfaron Hacker News • Recommends

On “Parsing huge XML files with Go”

Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise. The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: link to this video
View on Hacker News

TomNomNomon Hacker News • Recommends

On “Templating in Go”

Rob Pike did a really good talk entitled 'Lexical Scanning In Go'0, which covers how the lexing for text/template works. It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. 0 link to this video
View on Hacker News

groseon Hacker News • Recommends

On “Ask HN: What's your favorite tech talk?”

Lexical Scanning in Go by Rob Pike link to this video I love everything about this talk. It walks you through building a lexer from scratch in a simple and elegant way, through a very interesting use of coroutines. I appreciate the bits of humor in the talk as well.
View on Hacker News

hu3on Hacker News • Recommends

On “Rust for tokenising and parsing”

Related, I love Rob Pike's talk about lexical Scanning in Go (2011). Educational and elegant approach. link to this video
View on Hacker News

drbawbon Hacker News • Recommends

On “Go is amazing, period.”

As an aside: watching Pike's talk on lexing and parsing in Go really hits this point home. It shows how language support for cheap threads (goroutines) and channels really change the way you come up with an algorithm. link to this video
View on Hacker News

jbarhamon Hacker News • Recommends

On “Function types in Go”

> Building a lexer out of a recursive function type actually sounds like a really cool thing to try Rob Pike did just that for Go's text/template package ( link ). See the source lexer source code at link and note that stateFn is defined recursively. There's a video by Pike describing how he built the parser at link to this video .
View on Hacker News

mseepgoodon Hacker News • Recommends

On “Concurrency is not Parallelism. Rob Pike at Waza 2012 [video]”

> What most people don't realize is that for concurrency to be useful, there must be parallelism. No, as far as I understand from this and other talks, Rob Pike argues that concurrency is useful even without parallelism: it's a better way to structure software. A nice example for this is shown in his talk about lexical scanning in Go: link to this video
View on Hacker News

thisishugoon Hacker News • Recommends

On “Building a toy template engine in Python”

I'm not sure I like the idea of parsing templates with regular expressions. There is a better way, as demonstrated in Robe Pike's video1 detailing how the Go template language is implemented. Edit: which isn't meant to detract from the fact that this is a great article. 1 link to this video
View on Hacker News

canthonytuccion Hacker News • Recommends

On “Less is exponentially more (2012)”

For those who don't know who Rob Pike is, he's pretty awesome, helped create UTF-8 among other things, and now works on Go. link He has given some really great Go talks which are available on youtube, about an hour each: Origins of Go Concurrency Style - link Lexical Scanning in Go - link to this video Go Concurrency Patterns (One of my favorites) - link
View on Hacker News

landr0idon Hacker News • Recommends

On “Implementing a programming language in D: Lexical Analysis”

Wow, perfect timing! I'm currently writing my first lexer/parser in D as well ( link ) for a Scheme-like language used in Halo 3. I found Rob Pike's talk on lexical scanning to be pretty useful ( link to this video ) and I've modeled mine pretty heavily after the text/template lexer: link .
View on Hacker News

etatobyon Hacker News • Recommends

On “Writing a Compiler in Go”

…and is cleanly split across abstraction boundaries (packages "scanner", "parser", "ast", and so on.) There's also a smaller, simpler parser in the "text/template/parse" package, that is used to parse the templating languages in "text/template" and "html/template". As an introduction to it, I recommend the talk "Lexical Scanning in Go" by Rob Pike: link to this video…
View on Hacker News

gorgoileron Hacker News • Recommends

On “Ask HN: What piece of code/codebase blew your mind when you saw it?”

Rob Pike’s lexer was very nicely done and struck a chord with me. It’s not nearly as magical as some of the other things here. It’s just great engineering: link to this video
View on Hacker News

Xeoncrosson Hacker News • Recommends

On “Datalog in Rust”

If you enjoyed the state machine + parsing part and also I recommend the older "Lexical Scanning in Go" talk that Rob Pike gave: link to this video It's in Go, but you can apply most of it easily in other languages. I'm so happy that modern languages like Rust, Zig, and Go support unicode/runes/graphemes natively. So many problem just disappear compared to Java, .net, C++, or scripting languages.
View on Hacker News

Thaion Lobsters • Recommends

Lexical Scanning in Go - Rob Pike
3View on Lobsters

sapiogramon Hacker News • Disputes

On “Go: What we got right, what we got wrong”

…g model is now identical to threads. Using them as coroutines only leads to misery (read: data races) in my experience. They're faster than OS threads, sure, but that's an implementation detail at this point. When do you think they eventually realized they were re-inventing the same green threads Java had created and scrapped 10 years earlier? 1 link to this video…
View on Hacker News

u/MiloThatchCapitalin r/C_Programming • Mentions

Why returning the address of a local struct seems to be valid here? I just saw this video: [video and thought this should be doable in pure C as well. I scribbled down a few lines and at the second glance I noticed that I return the address of a local object: #include <stdio.h> typedef struct StateFunction { struct StateFunction* (*state_fn)(void); } StateFunction; StateFunction* exclamation(void) { printf("!!\n"); return (void*)0; } StateFunction* world(void) { printf("world"); return &(StateFunction) { .state_fn = exclamation, }; } StateFunction* hello(void) { printf…
21View on Reddit