Feeds and fetching
What counts as a source, how often it is read, how duplicates are avoided, and what happens with a new source's backlog
What counts as a source
A source is anything that publishes a feed. Owly Post knows four kinds: rss (newsletters, blogs,
news sites), youtube, podcast and reddit. You never pick the kind yourself: you paste a URL
and the feed is detected from it.
Detection works from the URL you give it:
- YouTube: a channel URL (
youtube.com/channel/UC…) or a handle (youtube.com/@name) becomes the channel's video feed. Playlist and video URLs are not supported. - Apple Podcasts: a show page (
podcasts.apple.com/…/id123456) is resolved to the show's real RSS feed through the iTunes lookup. - Reddit:
reddit.com/r/subredditbecomes the subreddit feed. - Everything else: first the URL itself is checked for being a feed, then the page is read for
a
<link rel="alternate">feed announcement, and finally the well-known paths are probed:/feed,/rss,/rss.xml,/atom.xml,/feed.xmland/index.xml. This covers every Substack and most blogs.
A feed whose items mostly carry audio files is treated as a podcast, whatever the URL looked like. Pages that have no feed at all cannot be followed.
Fetching goes through a guard that refuses private and local network addresses. A feed on your LAN
or on localhost is rejected by design.
How often sources are read
Every source is fetched on a schedule, every six hours by default. Each fetch is a conditional request: the feed is only downloaded when the publisher reports something new (ETag or Last-Modified), so an unchanged feed costs nothing. A single feed gets 15 seconds to answer.
When a fetch fails, the failure is counted. After five consecutive failures the source is marked
error and shown as needing attention. An errored source is not abandoned: it is retried once a
day, and a successful fetch clears the error and sets it back to active.
Deduplication
Two layers keep the same article from appearing twice:
- Within a source, an item is identified by its feed id (
guid) or, failing that, by its normalised URL. The same item in a later fetch is skipped. - Across sources, the normalised URL may exist only once. A post that two feeds both carry is stored under the first feed that delivered it.
Normalising a URL means lowercasing the host, dropping the fragment (#…) and removing all
utm_* tracking parameters.
Backlog of a new source
A feed usually carries its recent history, so the first fetch of a new source stores everything the feed contains. That backlog is bounded downstream rather than at fetch time:
- Summaries are written in batches of 100 per fetch run; a large backlog is worked through over a few runs.
- Only items published in the last 30 days can enter a digest, so a source with years of archives cannot flood it.
- The very first digest deliberately covers the whole 30-day window, so a fresh setup gets a real digest rather than an empty one.
In the cloud, the trial uses a shorter window so a new account's backlog cannot burn through its AI budget in one go. See Topics and sources for the current limits.
Full text
Many feeds only carry an excerpt. When a feed item has fewer than 500 characters of content, the
article page is fetched and its text extracted, for rss and reddit sources only: YouTube and
podcast items keep their description and show notes, since watch and episode pages extract junk.
Stored content is capped at 20,000 characters.
The full text is only ever used as input for the summary. Once an item is summarised, nothing reads it again, so the text is cleared after 30 days to keep the database small. Titles, links, summaries and topics are kept forever.