517

Building A Lightweight Internet Research Setup With, And Caching Layer For, SearxNG With Jina

I realized, in my usual “doh!” shock and horror, that I neglected to make a note in the last Drop about a newsletter hiatus this week (hey, if John Oliver can take arbitrary weeks off…). There’s just alot going on at the hrbrcompound and $WORK this week.

So, I feel like I owe y’all at least one Drop!

Let’s take today’s mea culpa Drop to talk a bit about the programmatic side of SearXNG and how to wire results from it up to Jina to create a mini-markdown content database which also serves as a nice caching layer to Jina calls, and bundle it all up into a lightweight Internet research utility.

SearXNG

Photo by Skitterphoto on Pexels.com

The mind boggles at the fact that I have not yet covered SearXNG (GH). A link for it was casually Dropped in September of 2022, but never received the full Drop treatment.

SearXNG is a free, self-hostable (in your workstation’s Docker — or other container engine — setup) internet metasearch engine that aggregates results from over 70 search services without tracking or profiling. By aggregating results from numerous search services, it gives us a broader range of information from a single query. It’s flexible enough to let us choose which search engines to include and adjust various settings to tailor the overall search experience/output. The conainter setup instructions are pretty ace, so I’ll leave that to you. After you get it working, this is what an example query result from an interactive session might look like:

NOTE: to enable the “Download results”, you can add the following to the settings.yml:

formats: - html - json - rss - csv

(You’ll need to restart the container after that.)

I still use Kagi for my regular interactive searches, but rely on SearXNG’s API for programmatically pulling content. That sounds fancier than it is, since all I have is a Bash script (searx) with a single curl call:

#!/usr/bin/env bashcurl \ --url "http://your-instance-ip-port/search" \ --data-urlencode "q='${1}'" \ --data "format=json" \ --fail \ --silent \ --insecure \ --location 2>/dev/null || echo '{ "message": "Error fetching Searxng results" }'

An given search incantation with that returns JSON, and each result looks like this:

$ searx 'national public data breach' | jq -sr '.[0].results[]'

{ "url": "https://krebsonsecurity.com/2024/08/national-public-data-published-its-own-passwords/", "title": "National Public Data Published Its Own Passwords – Krebs on Security", "content": "2 days ago - New details are emerging about a breach at National Public Data (NPD), a consumer data broker that recently spilled hundreds of millions of Americans' Social Security Numbers, addresses, and phone numbers online. KrebsOnSecurity has learned that another NPD data broker…", "publishedDate": null, "thumbnail": "", "engine": "brave", "parsed_url": [ "https", "krebsonsecurity.com", "/2024/08/national-public-data-published-its-own-passwords/", "", "", "" ], "template": "default.html", "engines": [ "qwant", "duckduckgo", "brave" ], "positions": [ 1, 1, 1 ], "score": 9.0, "category": "general"}

That score value is super handy. It represents a relevance score that indicates how relevant or important the result is deemed to be in relation to the search query.

SearxNG calculates this score based on various factors, including:

the position of the result across different search engines
the number of search engines that returned this result
other relevance metrics provided by the individual search engines

The value can range from 0 to 100, with higher values indicating greater relevance; so, that 9.0 means that result is likely pretty important/good.

SearxNG uses this score to help rank and order the results it presents via the web interface, and it does a bit of work behind the scenes to normalize and compare relevance across different sources.

Rather than leave the cozy CLI, I can get a feel for the top search results right in the termina:

$ searx 'national public data breach' | jq -sr '.[0].results[] | select(.score >= 1) | "\(.score) \(.title)"'9.0 National Public Data Published Its Own Passwords – Krebs on Security2.4 Was your Social Security number compromised in a massive data breach?2.1666666666666665 The Slow-Burn Nightmare of the National Public Data Breach | WIRED1.5227272727272727 National Public Data breach update: class action lawsuits pile up1.5119047619047619 National Public Data Cyber Attack: Massive Data Breach Exposed Countless Social Security Numbers and Personal Info - CNET Money1.2857142857142856 Data broker blunders as millions are exposed with public passwords | Fox News1.5 2.9 billion records stolen in Social Security data hack, USDoD claims1.0666666666666667 Hackers may have stolen your Social Security number in a massive breach ...1.0666666666666667 Mozilla Monitor | National Public Data

Headlines are great, but I want to see the content. While there are many ways to do that, let’s talk about getting the content with Jina.

Jina

Photo by Boris Hamer on Pexels.com

We only mentioned the embeddings side of Jina last October. They have quite a few free API endpoints with extremely generous API call allowances and stupid affordable extended allowances (they’re likely slurping up all URLs/content you send to it, so keep that in mind as to why the free tier is so generous).

The one we’re talking about today is the Reader endpoint. It turns the content at any URL into super clean Markdown. You can see that right now by hitting up https://dailydrop.hrbrmstr.dev/2024/08/18/bonus-drop-58-2024-08-18-one-thing-well/ — which is a Jina AI link to the Markdown version of the last Bonus Drop. It also converts PDF files to Markdown super well.

Despite having a cool free tier, I don’t want to hit it up each time for SearXNG results I’ve already seen, so I threw together a small Go CLI jinac that adds a caching layer to Jina’s Reader API results using SQLite as the cache store.

An API result will have the following structure:

$ jinac 'https://dailydrop.hrbrmstr.dev/2024/08/15/drop-517-2024-08-15-thursdataday/' | jq

{ "code": 200, "status": 20000, "data": { "title": "Drop #517 (2024-08-15): Thursdataday", "url": "https://dailydrop.hrbrmstr.dev/2024/08/15/drop-517-2024-08-15-thursdataday/", "content": "_committed; sq; CSVs Are Kinda Bad_…", "publishedTime": "2024-08-15T13:11:00+00:00", "usage": { "tokens": 2483 } }}

We can see all the URLs we have cached via:

Let’s wire this up with the SearXNG CLI and glow to give me a super-fast, TUI for reading the content at the top search results.

CLI [Re]Search

Photo by Pixabay on Pexels.com

Here’s an expanded view of what might go into another script (researx) to let us preview search results right in the terminal:

#!/usr/bin/env bashsearx "${*}" | jq -sr '.[0].results[] | select(.score >= 0.5) | .url' | while IFS= read -r url; do echo "${url}" jinac "${url}" | jq -r '.data.content' | glow -p done

So, if I do:

$ researx national public data breach

I’ll be able to read through the content without much cruft:

And, I now have all that content saved locally. Which means I have programmatic access to it:

$ jinac --show-cache-dir # the CLI uses XDG cross-platform appdir standardsCache directory: /Users/hrbrmstr/Library/Application Support/jinac

$ duckdb \ -list -s "FROM cache SELECT url LIMIT 3" \ "/Users/hrbrmstr/Library/Application Support/jinac"/cache.db 2>/dev/nullurlhttps://krebsonsecurity.com/2024/08/national-public-data-published-its-own-passwords/https://www.fastcompany.com/91176676/national-public-data-breach-update-class-action-lawsuits-npdhttps://www.wgal.com/article/national-public-data-breach-find-out-if-your-social-security-number-was-involved/61928667

including the JSON column:

$ duckdb \ -list \ -s "FROM cache SELECT (response::JSON).data.title AS title LIMIT 3" \ "/Users/hrbrmstr/Library/Application Support/jinac"/cache.db 2>/dev/null 2>/dev/nulltitle"National Public Data Published Its Own Passwords""National Public Data breach update: Lawsuits pile up against Florida-based background check company after 'security incident'""National Public Data breach: How to find out if your Social Security number was involved"

The content is also in decent shape for piping into your fav AI overlord.

The jinac code is up on Codeberg.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev ☮️

https://dailydrop.hrbrmstr.dev/2024/08/22/drop-519-2024-08-22-cache-me-if-you-can/

#517

0 0 0 View Post & Replies See Original

sfeed; gen-license; gitnr

We’ve got three projects that all do one thing, and do said thing well (truth-be-told, there are a handful of “one-thing well” CLIs in the first section).

TL;DR

(This is an AI-generated summary of today’s Drop using Sonnet via Perplexity.)

sfeed: A collection of command-line tools for parsing, updating, and formatting RSS and Atom feeds, designed to be simple, fast, and flexible. It includes various utilities for processing feed data into different formats and a curses interface for browsing. (https://codemadness.org/sfeed.html)
gen-license: A command-line tool for generating software licenses, allowing users to quickly create LICENSE files for their projects. An alternative to this is the ‘license’ tool from sourcehut, which provides similar functionality without making API calls to GitHub. (https://github.com/nexxeln/license-generator)
gitnr: A cross-platform CLI and TUI for generating .gitignore files using templates from various sources, including GitHub’s official repository, TopTal’s collection, local files, and remote URLs. It automatically filters out duplicate ignore lines for efficient .gitignore creation. (https://github.com/reemus-dev/gitnr/tree/main)

sfeed

sfeed (GH) is a collection of command-line tools for parsing, updating, and formatting RSS and Atom feeds. It’s designed to be simple, fast, and flexible, following the Unix philosophy of doing one thing well and allowing easy integration with other tools.

The main component of the sfeed cinematic universe (SCU) processes XML feeds into a tab-separated format, since XML is all kinds of horribad. This conversion makes it very straightforward to manipulate the data with any set of modern data tools. The SCU includes tools for updating feeds, filtering content, merging items, and converting feed data into formats like JSON!!, HTML, plain text, and more. There’s even a well-executed curses interface for RSS feed browsing. And, it makes extensive use of configuration files, letting us implement custom filters or override default behaviors, such as download methods (by default, sfeed relies on curl).

Speaking of the SCU, here’s what’s inside the git tin:

sfeed: Read XML RSS or Atom feed data from stdin. Write feed data in TAB-separated format to stdout.
sfeed_atom: Format feed data (TSV) to an Atom feed.
sfeed_content: View item content, for use with sfeed_curses.
sfeed_curses: Format feed data (TSV) to a curses interface.
sfeed_frames: Format feed data (TSV) to HTML file(s) with frames.
sfeed_gopher: Format feed data (TSV) to Gopher files.
sfeed_html: Format feed data (TSV) to HTML.
sfeed_json: Format feed data (TSV) to JSON Feed.
sfeed_opml_export: Generate an OPML XML file from a sfeedrc config file.
sfeed_opml_import: Generate a sfeedrc config file from an OPML XML file.
sfeed_markread: Mark items as read/unread, for use with sfeed_curses.
sfeed_mbox: Format feed data (TSV) to mbox.
sfeed_plain: Format feed data (TSV) to a plain-text list.
sfeed_twtxt: Format feed data (TSV) to a twtxt feed.
sfeed_update: Update feeds and merge items.
sfeed_web: Find URLs to RSS/Atom feed from a webpage.
sfeed_xmlenc: Detect character-set encoding from a XML stream.
sfeedrc.example: Example config file. Can be copied to $HOME/.sfeed/sfeedrc.
style.css: Example stylesheet to use with sfeed_html and sfeed_frames.

If you’re “just” going to use it as a feed reader, all you have to do is add feed URLs in a sfeedrc file, run sfeed_update to fetch and parse feeds, and choose a formatting tool to display the results.

But, it can do way more than just that!

For example, you can discover if a site has an feed:

$ curl -s 'https://rud.is/b/' | sfeed_webhttps://rud.is/b/feed/ application/rss+xmlhttps://rud.is/b/comments/feed/ application/rss+xml

Use sfeed_json to work with feed elements in, say, jq:

$ curl -s "https://dailydrop.hrbrmstr.dev/feed/" | sfeed | sfeed_json | jq '.items[] | .title'"Drop #518 (2024-08-16): Clean As A Whistle""Drop #517 (2024-08-15): Thursdataday""Drop #516 (2024-08-13): Typography Tuesday""Drop #515 (2024-08-12): Starting The Week In Style[s]""Bonus Drop #57 (2024-08-11): Vector Similarly Search And RAG-Lite""Drop #514 (2024-08-09): DuckDB Vector Search""Drop #513 (2024-08-08): TUI Thursday""Drop #512 (2024-08-07): The One With All The JSON Resources""Drop #511 (2024-08-06): Typography Tuesday""Bonus Drop #56 (2024-08-03): Knowledge Drop"

Use sfeed_content to do a quick, paginated read:

$ curl -s "https://dailydrop.hrbrmstr.dev/feed/" | sfeed | sfeed_content

You’ll need to do some work, since neither a brew info sfeed nor a apt search sfeed came up with any hits.

A git clone git://git.codemadness.org/sfeed then make initially failed for me on macOS, but the quick fix was to put:

CFLAGS += -D_DARWIN_C_SOURCE

right before the

# use system flags.

line.

The README is extensive, and has plenty of usage examples to help get you started.

gen-license

Photo by Pixabay on Pexels.com

I switch between many editors in a given day for many reasons. Lately, I’ve been trying to use Zed as the main editor for Go projects. A sign that I had become far to reliant on VS Cod[e|ium] extensions was when I was looking to add a LICENSE file to a recent project, and brought up Zed’s command palette to use the license generator that did not exist (I have a VS Codium extension installed for this).

Yes I could keep a copy of the tiny handful of licenses I use and make a Zsh function to copy one to a current directory, but why do that when I can cargo install gen-license and have them all in a pick list with one command:

$ gen-license✔ Choose a license · MIT License✔ Enter your name · Bob Rudis✔ Enter year · 2024✔ License created successfully Please take a look at it and make changes if necessary

But, I should have read the code before using it as my setup notified me that an unknown app was making a call to GitHub’s API (I try hard to not give any telemetry to the one of the most evil corproations on the planet).

If that bugs you as much as it did me, then you, too, can clone https://sr.ht/~zethra/license/, follow the installation guidance and then run:

$ license list | fzf

to get a similar experience.

gitnr

After the revelation in the middle section, I set out to replace the editor plugin I use to maintain .gitignore files in project with a CLI utility.

Gitnr is a cross-platform CLI and TUI that generates .gitignore files using templates from various sources, including:

GitHub’s official gitignore repository
TopTal‘s gitignore collection
Local files on your system
Remote URLs

When using multiple templates, Gitnr automatically filters out duplicate ignore lines, ensuring a clean and efficient .gitignore file. For example:

$ gitnr create gh:Go tt:go

will filter out all the duplicates.

The TUI (in the section header) is also nicely done.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev ☮️

https://dailydrop.hrbrmstr.dev/2024/08/18/bonus-drop-58-2024-08-18-one-thing-well/

#511 #512 #513 #514 #515 #516 #517 #518 #56 #57

0 0 0 View Post & Replies See Original