Skip to content

Reading Pages

When your package needs the content of a web page (for example, to draft a post about it), don’t fetch the page with an HttpClient of your own. The address usually comes from a user, so a plain fetch can be pointed at the server’s own network. Use ILinkPageReader (UmbSocial.Core.Links) instead. It ships in the free core and reads through the same guarded fetch as UmbSocial’s link cards.

public interface ILinkPageReader
{
Task<LinkPage?> ReadAsync(Uri url, IReadOnlyCollection<string> allowedHosts, CancellationToken ct);
}
public sealed record LinkPage(Uri FinalUrl, string? Title, string? Description, string Html);

ReadAsync returns the page, or null when the page is refused or can’t be read:

  • FinalUrl is where the page ended up after any redirects.
  • Title and Description are read from the page’s Open Graph tags, or from its <title> and description tags if it has no Open Graph tags.
  • Html is the page’s HTML, cut to its first 512 KB. A longer page is still returned, just shortened.

allowedHosts lists the hosts the reader may reach. Each entry matches that host and any subdomain of it, so example.com allows example.com and blog.example.com, but not notexample.com. An empty list refuses everything.

The list is checked on the first address and again on every redirect, before the host is looked up or contacted. A redirect to a host that isn’t on the list is refused.

On top of the allow-list, every read is held to the same rules as a link card:

  • https only, on the default port, with no user name or password in the address;
  • every address the host resolves to must be public. Private, loopback, link-local (cloud metadata) and other special ranges are refused. The check runs again when connecting, so a DNS answer that changes in between can’t get round it;
  • at most 3 redirects, each one checked in the same way;
  • HTML pages only, the first 512 KB, and 5 seconds for the whole read.

Nothing is cached. If you call the reader often, cache the results yourself.

Inject ILinkPageReader and pass the hosts that make sense for your feature, such as the site’s own domains:

using UmbSocial.Core.Links;
public sealed class PageSummaries(ILinkPageReader pages)
{
public async Task<string?> TitleOfAsync(Uri url, CancellationToken ct)
{
var page = await pages.ReadAsync(url, ["example.com"], ct);
return page?.Title;
}
}

Treat what comes back as untrusted: the page’s author chose every word of it. Quote it as data. Never follow instructions found in it.

Register your own ILinkPageReader (with AddSingleton, AddTransient or AddHttpClient) to read pages from somewhere else, such as a cache or a crawler. Yours wins whichever composer runs first. If your implementation fetches pages itself, keep the same fences and the allow-list, because the address is user input.