Shameless self promo: I was upset by this as well so I’m working now on a curated search engine just for anything related to webdev. It focuses on blogs and docs. No BS, just high quality sources.

https://kukei.eu

Also it’s hosted on a PC in my living room ;)

What the fucking fuck. In Europe this would cause French and Germans combined beheading entire governments. One thing I want to keep to myself is my porn choices. Giving my ID to a porn site sounds like a big fucking NO

Can someone explain it to Europeans?

I keep my drives encrypted with a key currently hosted in my router hoping they wouldn’t steal that. I’m thinking of actually putting it to cloud so I can disable it remotely.

It was quite a ride to make everything work and I made a blog post explaining it so I remember what I did.

https://nowicki.io/self-hosting-lvm-raid1-with-key-over-ftp/

So it was DNS?

ISP can’t see pages. They can see domains or IPS but that’s it.

Am no native English speaker so I have to ask. Excuse me WHAT?

No, I’m working on it since a week from scratch. Had a #showerThought idea and since also had some time I immediately started to realise it and here I am now :D Waiting for 300K pages to be crawled so I can add more sources there :D

I’m making one right now. It’s still kinda rough and MVP but soon should be more mature and useful. Try it out: https://kukei.eu

Thanks for the kind words!

Thanks but don’t expect too much yet. Many sources are still missing. If you notice something should be there but it’s not even being crawled feel free to reach me one Mastodon or add it directly via PR here: https://github.com/Kukei-eu/spider/blob/main/index-sources.js

Another person in real life told the same. Adding to the backlog!

https://github.com/Kukei-eu/kukei-web/issues/3

I’m on iPhone 12 mini. I love that small design and I strongly believe phones should be small.

Thanks for the good words! Highly appreciate it!

1. SO and Reddit are on the TODO list. It even had SO (in the bottom indeed) once but not via crawling, via SO Search API. It has very poor quality results and was super slow so I had to remove it while thinking of a better solution. Crawling entire SO might be little too much of this project at this state tho but if I have enough courage and hours at night I might parse that 20GB stack overflow archive dump and try doing something useful with it.

Same for Reddit but here I have mixed feelings about it in general and hope it’s going to die soon being replaced by amazing Lemmy communities.

I also used to type some question and end with “reddit” in Google to get good quality content, but here with kukei the experiment is whether blogosphere can replace it properly when index is promoting it.

2. Why blogs?

This is my main thing. To promote good quality blogs that I tried to follow via RSS but somehow never did. Having them all indexed (and more, some Mastodon community gave me amazing links to index) makes me actually visit them often.

For the “SEO cancer” that where curation comes into play. Before crawling I check unknown blogs to me and decide whether something goes in or not.

Great ideas. For the source code I’m not sure but I’ll put it to the backlog of cool things I get from Lemmy and work on them one by one. Thanks!

The crawler takes only the sources that are defined in the crawler repo (it’s open source, check the github org or kukei-spider).

So in this way it’s “curated” in a sense that it would not add anything else to the index.

Thx for the comments. I’ll fix the mobile view and will definitely redesign it all a bit over weekend. I see a lot of room for improvements.

Also will check how to submit it to Lenses. Highly appreciate it!

EDIT: mobile view is fixed, also did some small adjustments in the whitespaces between result items.

Good idea. I had this thought once to do some narrow indexing of websites, e.g. stack overflow is a big issue, indexing all of this is crazy, picking up some specific tags on the other hand feels like tons of work. In the end I adjust the whole project as it grows with hope that after every tuning it gets better.

As long as I have fun with it I’ll continue :D

For ?? I guess it already has a decent results. I’ll periodically check those kind of cases once the index gets more languages.

https://kukei.eu/?q=js+%3F%3F+operator

Thanks! If you have some suggestions in the future I’m always open to hear