Ohi, author here! Thanks for posting Hister. Feel free to A.M.A.
My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach.
Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations.
>Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites.
Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better.
One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag) and it's a great default, but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-handle-able enough if needed) that I don't feel any desire to switch to WARCs or similar.
Though specifically I'm probably looking for something like "content-extract everything" + "key combo to save singlefile version too" + "upload singlefile archives to backfill / recover". Like 99% of the time content extraction is preferred, and I'm glad to see it... it's just not always enough, and having to go elsewhere for exceptions breaks a lot of the utility.
Exactly! I had the very same issues before Hister.
> One question if ya don't mind, while I explore: any chance of singlefile support?
Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
I assume it content-extracts that on upload? I'd really like to move storage into hister too, if possible. That way you could also switch from an extracted view to a "full" view in the UI. Though I assume that'd be fairly simple to build later.
Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
Safari support is currently stalled in this PR: https://github.com/asciimoo/hister/pull/46. I do not have access to macOS, so help with development and testing would be very welcome.
You should integrate with Karakeep.app, it's only natural that you'd want both a search engine, and a nice "archive" and "article pretty view" features :)
What do you mean by integration? Hister already supports importing data from Karakeep, Linkding, Linkwarden, Readeck, Shaarli, and wallabag: https://hister.org/docs/import
Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first.
> would've been great with a more liberal license
It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful research tool for one of my hobbies (award travel), that revolves around being in the know around various concepts and quirks.
I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP server and used that corpus for research on those topics. So I can ask things like "has anyone ever mentioned running into this problem before?" [1]
If you have a hobby or working situation that requires you to regularly reference a core set of websites or reference materials, Hister provides almost all the tools out of the box to start a search engine against it. The default datasets they promote include the Python Stlib, MDN and RFC corpus, as an example. [2]
I tried this out this week and liked it but really wish this project had some form of auth. Opening the contents of every page you’ve ever visited, even to the local network, is not the best idea.
Hister supports token based, password based, and OIDC/OAuth authentications with optional multi-user handling. Details about user handling can be found here: https://hister.org/docs/user-handling
It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.
It seemed like the public mode was the default when I set it up. If so, that’s a fairly dangerous default as keeping a “clean” history with no secrets leaked seems neigh impossible.
The default configuration binds only to localhost, and a fresh installation starts with an empty database/index. Could you clarify which specific attack surface you are concerned about in that scenario?
My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach.
Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations.
Project page: https://github.com/asciimoo/hister
Tiny read-only demo: https://demo.hister.org/
Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better.
One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag) and it's a great default, but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-handle-able enough if needed) that I don't feel any desire to switch to WARCs or similar.
Though specifically I'm probably looking for something like "content-extract everything" + "key combo to save singlefile version too" + "upload singlefile archives to backfill / recover". Like 99% of the time content extraction is preferred, and I'm glad to see it... it's just not always enough, and having to go elsewhere for exceptions breaks a lot of the utility.
> One question if ya don't mind, while I explore: any chance of singlefile support?
Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
would've been great with a more liberal license
Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first.
> would've been great with a more liberal license
It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP server and used that corpus for research on those topics. So I can ask things like "has anyone ever mentioned running into this problem before?" [1]
If you have a hobby or working situation that requires you to regularly reference a core set of websites or reference materials, Hister provides almost all the tools out of the box to start a search engine against it. The default datasets they promote include the Python Stlib, MDN and RFC corpus, as an example. [2]
[1]: https://wmchen.com/blog/revisiting-hister/
[2]: https://hister.org/datasets
What tools or features would Hister need to support your complete search workflow?
https://github.com/rumca-js/Internet-Places-Database
Also I maintain android app that can be used to search places.
https://f-droid.org/en/packages/io.github.rumcajs.offlineweb...
I believe hister you have to fill in with your data, right?
It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.