YaCy-Blog

Today we released YaCy 1.2. This is a major change in the architecture of YaCy since Search functions now use primary Solr as indexing engine instead our Peer-to-Peer optimized distributed indexing algorithms. This step also means that YaCy gives up the attempt to create it’s own indexing technology in favor of the much more advanced Solr/Lucene library.

While we will still do Peer-to-Peer search, we will now consider YaCy also as a Crawling and Search Framework for Solr. Today we presented „YaCy as Search Appliance“ at the ApacheCon 2012 with great success:

Notable new features in YaCy 1.2 are:

• Embedded Solr 4.0.0 with standard Solr XML search interface integrated
• Enhanced crawling with live link structure visualization
• A Host Browser to explore the file structure of crawled hosts: this shows loaded pages, pages with errors and pending files in the same way as a file browser would show the content of a host.

We believe that the new features are also valuable for Web Administration and Search Engine Optimization (i.e. to find dead links etc). Please have a look to the screen shots at the YaCy ApacheCon 2012 Talk Slides.

YaCy is available as Windows, Mac and debian package and also as tarball. To download YaCy, visit the YaCy Home Page.

tl;dr

Use Solr, but don‘t home-brew your own code around it if you do web-, file- or intranet-search, it‘s all inside YaCy. And don‘t buy a commercial appliance, this is free and better!

Today we release a first bugfix release after the 1.0 release last week: download YaCy 1.01. In the context of the main release event we asked for help and new developers: this was a great success! Our new Git repository was cloned 11 times and we got bug fixes and translations: a greek and a chinese translation is meanwhile beeing produced. More help is very welcome, please see our tutorial movie about starting development with YaCy and if you like to make a interface translation, please see this manual.

As response to the great feed-back to our release we had to check if our network was able to scale to up to 4000 peers which our network consisted of in the meantime. We found out that the biggest problem was just a display of the network graphic which is fixed now. During the time of the strongest growth of the search network the distributed hash table failed because it takes time to distribute the index elements to the new peers. Existing peers became invisible since the new peers became ‚in front‘ of old one. This caused bad search results. Another problem that was often discussed was: search time and ranking quality. The search time is connected to fraud detection which we showed in a short speech about that topic and we worked on that problem further in the 1.01 release. On public demand we also added indexing to the integrated bookmark system. (see: German Article about YaCy and the integrated Bookmarking).

Please try the new release since it should also be a bit more stable: if you have not discovered the update function yet then please give it a try (it’s very easy to update YaCy!): click on Administration -> System Update (in the top right) -> Select the latest Release from the drop-down box -> Download -> Install Release.

This project can only live if people help. If you help! Please join in!