Skip to content
Hallway
0:00 / 0:00

#zig#webcrawler#asyncio

Building a Web Crawler from scratch in Zig

nubskr

3.5K subscribers • 6 videos on Hallway

YouTube
  • Recommended by 1 person: 1 in engineering communities.

Intermediate. Assumes working knowledge.

3.2K views4 months ago1 recommendation

Programming languagesSystems programmingZigConcurrency and parallelismNetworking and protocols

nubskr builds a web crawler from scratch in Zig using asynchronous I/O to fetch pages, extract links, and manage a URL queue.

reference code: gist.github.com/nubskr/26cd0cb7b6507b16a4f77b156995c366 zig async IO video: youtu.be/-30RvjXTxvc?si=Dl2s6pfmWf5o5iOh In this video, I build a web crawler in Zig using async I/O. It starts with one URL, fetches the page, extracts links, resolves relative URLs, filters out junk, and keeps crawling. Along the way, we build the core pieces of a crawler: request handling, URL queues, metadata tracking, link extractions, etc. Basically: Googlebot crawls billions of pages. This one is Zig, so we’ll see what happens. Chapters: - Teaser - Intro - Building the CLI - Fetching Webpages - Indexing Webpages - Resolving URLs - One Thread Is Too Slow - Multithreaded Web Scraping - Building the Scraping Worker - Hunting Bugs and Testing #zig #systemsprogramming #asynchronousprogramming

Why it's here

1 person recommends this

What people say. Recommended as a practical guide to building a concurrent web crawler in Zig.

u/Ok_Marionberry8922in r/Zig • Recommends

Building a Web Crawler from scratch in Zig
31View on Reddit